Best Practices
Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes Gao ID: GAO-02-701 July 15, 2002This report examines how best practices offer improvements to the way the Department of Defense (DOD) develops new weapons systems, primarily the design and manufacturing aspects of the acquisition process. Knowledge about a product's design and producibility facilitates informed decisions about whether to significantly increase investments and reduces the risk of costly design changes later in the program. Leading commercial companies employ practices to capture design and manufacturing knowledge in time to make key decisions during product development. First, the companies kept the degree of the design challenge manageable before starting a new product development program by using an evolutionary approach. Second, the companies captured design and manufacturing knowledge before the two critical decision points in product development: when the design was demonstrated to be stable--the second knowledge point--and when the product was demonstrated to be producible at an affordable cost--the third knowledge point. DOD has made changes to its acquisition policy in an attempt to improve its framework for developing weapons systems, but the policy does not require the capture of design or manufacturing knowledge or sufficient criteria to enter the system demonstration and production phases. In addition, it does not require a decision review to enter the demonstration phase of product development.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-02-701, Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes
This is the accessible text file for GAO report number GAO-02-701
entitled 'Best Practices: Capturing Design and Manufacturing Knowledge
Early Improves Acquisition Outcomes' which was released on July 15,
2002.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products‘ accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
Report to the Subcommittee on Readiness and Management Support,
Committee on Armed Services,
U.S. Senate:
July 2002:
BEST PRACTICES:
Capturing Design and Manufacturing Knowledge Early Improves Acquisition
Outcomes:
GAO-02-701:
Letter:
Executive Summary:
Purpose:
Background:
Results in Brief:
Principal Findings:
Recommendations for Executive Action:
Agency Comments:
Chapter 1:
Best Practices of Leading Commercial Companies:
DOD‘s Traditional Approach to Product Development:
DOD‘s Adoption of Best Practices:
Objectives, Scope, and Methodology:
Chapter 2:
DOD Programs Had Better Outcomes When Design and Manufacturing
Knowledge Was Captured at Key Program Junctures:
Chapter 3:
Leading Commercial Companies Use Evolutionary Product Development
Framework to Reduce Development Risks:
Leading Commercial Companies Use a Product Development Process to
Capture Design and Manufacturing Knowledge for Decision Making:
When DOD Programs More Closely Approximated Best Practices, Outcomes
Were Better:
Chapter 4:
Acquisition Policy Lacks Specific Implementation Criteria:
Incentives in the DOD Acquisition Environment Do Not Favor Capture of
Design and Manufacturing Knowledge Early
Enough:
Chapter 5:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendixes:
Appendix I: Comments from the Department of Defense:
Appendix II: GAO Staff Acknowledgements:
Acknowledgments:
Related GAO Products:
Tables:
Table 1: Activities That Enable the Capture of Design and Manufacturing
Knowledge:
Table 2: Attainment of Design and Manufacturing Knowledge in DOD
Programs and the Program Outcomes:
Table 3: Activities to Capture Design Knowledge and Make Decisions:
Table 4: Examples of Prototypes Used by Cummins Inc. at Various Stages
of Product Development:
Table 5: Activities to Capture Manufacturing Knowledge and Make
Decisions:
Table 6: Cpk Index and Probability of a Defective Part:
Table 7: Analysis of DOD Acquisition Policy for Inclusion of Best
Practices for Knowledge-based Design and Manufacturing Decisions:
Figures:
Figure 1: Research, Development, Test and Evaluation, and Procurement
Funding for Fiscal Years 1995 to 2007:
Figure 2: Knowledge-based Process for Applying Best Practices to the
Development of New Products:
Figure 3: Notional Illustration Showing the Different Paths That a
Product‘s Development Can Take:
Figure 4: DOD‘s Concurrent Approach to Weapon System Development:
Figure 5: Notional Single-Step and Evolutionary Approaches to
Developing New Products:
Figure 6: Achieving Stability on AIM-9X Missile Program by Knowledge
Point 2:
Figure 7: History of Drawing Completion for the F-22 Program:
Figure 8: PAC-3 Design Knowledge at Critical Design Review:
Figure 9: Illustration to Show How the Best Practice Model Would Apply
to DOD‘s Acquisition Process:
Letter July 15, 2002:
The Honorable Daniel Akaka
Chairman
The Honorable James Inhofe
Ranking Minority Member
Subcommittee on Readiness and Management Support
Committee on Armed Services
United States Senate:
As you requested, this report examines how best practices offer
improvements to the way the Department of Defense develops new weapon
systems, primarily the design and manufacturing aspects of the
acquisition process. It examines the attainment of design and
manufacturing knowledge and its use at critical junctures to make
decisions about weapon systems‘ readiness to move forward in the
acquisition process. We make recommendations to the Secretary of
Defense for improvements to weapon system acquisition policy to better
align design and manufacturing activities with best practices that have
shown that the capture and use of key knowledge can result in better
cost, schedule, and performance outcomes.
We are sending copies of this report to the Secretary of Defense; the
Secretary of the Army; the Secretary of the Navy; the Secretary of the
Air Force; the Director of the Office of Management and Budget; the
Director, Missile Defense Agency; and interested congressional
committees. We will also make copies available to others upon request.
In addition, the report will be available at no charge on the GAO Web
site at http://www.gao.gov.
If you have any questions regarding this report, please call me at
(202) 512-4841. Other contacts are listed in appendix II.
Katherine V. Schinasi
Director
Acquisition and Sourcing Management:
Signed by Katherine V. Schinasi:
[End of section]
Executive Summary:
Purpose:
Historically, the Department of Defense (DOD) has taken much longer and
spent much more than originally planned to develop and acquire its
weapon systems, significantly reducing the department‘s buying power
over the years. Clearly, it is critical to find better ways of doing
business and, in particular, to make sure that weapon systems are
delivered on time and cost-effectively. This is especially true given
the vast sums DOD is spending and is expected to spend on weapons
acquisition--$100 billion alone in 2002 and an anticipated $700 billion
over the next 5 years. DOD has recognized the nature of this problem
and has taken steps to address it, including advocating the use of best
practices for product development from commercial companies. Leading
commercial companies have achieved more predictable outcomes from their
product development processes because they identify and control design
and manufacturing risks early and manage them effectively. While DOD
has made some progress in recent years, GAO‘s recent weapon system
reviews show that persistent problems continue to hinder acquisition
cost, schedule, and performance outcomes. For this reason, GAO has
continued a body of work to identify the lessons learned by best
commercial companies to see if they apply to weapon system
acquisitions.
This report addresses how DOD can manage its weapon system acquisition
process to ensure important knowledge about a system‘s design, critical
manufacturing processes, and reliability is captured and used to make
informed and timely decisions before committing to substantial
development and production investments. It identifies best practices to
facilitate this decision making at two critical junctures--transition
from system integration to system demonstration during product
development and then transition into production. Ultimately, this
should improve cost, schedule, and quality outcomes of DOD major weapon
system acquisitions. In response to a request from the Chairman and the
Ranking Minority Member, Subcommittee on Readiness and Management
Support, Senate Committee on Armed Services, GAO (1) assessed the
impact of design and manufacturing knowledge on DOD program outcomes,
(2) compared best practices to those used in DOD programs, and (3)
analyzed current weapon system acquisition guidance for applicability
of best practices to obtain better program outcomes.
Background:
In any new product development program there are three critical points
that require the capture of specific knowledge to achieve successful
outcomes. The first knowledge point occurs when the customer‘s
requirements are clearly defined and resources--proven technology,
design, time, and money--exist to satisfy them. Commercial companies
insist that technology be mature at the outset of a product development
program and, therefore, separate technology development from product
development. The second knowledge point is achieved when the product‘s
design is determined to be capable of meeting product requirements--the
design is stable and ready to begin initial manufacturing of
prototypes. The third knowledge point is achieved when a reliable
product can be produced repeatedly within established cost, schedule,
and quality targets. GAO‘s prior work on best practices covers
achieving the first knowledge point.[Footnote 1] This report examines
best practices for achieving the second and third knowledge points.
Commercial companies understand the importance of capturing design and
manufacturing knowledge early in product development, when costs to
identify problems and make design changes to the product are
significantly cheaper. In a knowledge-based process, the achievement of
each successive knowledge point builds on the preceding one, giving
decision makers the knowledge they need--when they need it--to make
decisions about whether to invest significant additional funds to move
forward with product development. Programs that follow a knowledge-
based approach typically have a higher probability of successful cost
and schedule outcomes. Problems occur in programs when knowledge builds
more slowly than commitments to enter product development or
production. The effects of this delay in capturing knowledge can be
debilitating. If a decision is made to commit to develop and produce a
design before the critical technology, design, or manufacturing
knowledge is captured, problems will cascade and become magnified
through the product development and production phases. Outcomes from
these problems include increases in cost and schedule and degradations
in performance and quality.
Results in Brief:
The success of any effort to develop a new product hinges on having the
right knowledge at the right time. Knowledge about a product‘s design
and producibility facilitates informed decisions about whether to
significantly increase investments and reduces the risk of costly
design changes later in the program. Every program eventually achieves
this knowledge; however, leading commercial companies GAO visited have
found that there is a much better opportunity to meet predicted cost,
schedule, and quality targets when it is captured early, in preparation
for critical investment decisions. A product development process
includes two phases followed by production--integration phase and
demonstration phase. The commercial companies GAO visited achieved
success in product development by first achieving a mature, stable
design supported by completed engineering drawings during an
integration phase and then by demonstrating that the product‘s design
was reliable and critical manufacturing processes required to build it
were in control before committing to full production. The more
successful DOD programs GAO reviewed--the AIM-9X and the FA-18-E/F
programs--had achieved similar knowledge as the commercial companies,
resulting in good cost and schedule outcomes. In contrast, the DOD
programs, which had completed about one-quarter of their drawings when
they transitioned to the demonstration phase and had less than half of
their manufacturing processes in control when entering production,
experienced poor cost and schedule outcomes.
Leading commercial companies employed practices to capture design and
manufacturing knowledge in time for making key decisions during product
development. Two were most prominent. First, the companies kept the
degree of the design challenge manageable before starting a new product
development program by using an evolutionary approach to develop a
product. This minimized the amount of new content and technologies on a
product, making it easier to capture the requisite knowledge about a
product‘s design before investing in manufacturing processes, tooling,
and facilities. Second, the companies captured design and manufacturing
knowledge before the two critical decision points in product
development: when the design was demonstrated to be stable--the second
knowledge point--and when the product was demonstrated to be producible
at an affordable cost--the third knowledge point. A key measure of
design stability was stakeholders‘ agreements that engineering drawings
were complete and supported by testing and prototyping when necessary.
A key measure of producibility was whether the companies‘ critical
manufacturing processes were in control and product reliability was
demonstrated. Most DOD programs GAO reviewed did not complete
engineering drawings prior to entering the demonstration phase, nor did
they bring critical manufacturing processes in control or demonstrate
reliability prior to making a production decision.
DOD has made changes to its acquisition policy[Footnote 2] in an
attempt to improve its framework for developing weapon systems, but the
policy does not require the capture of design or manufacturing
knowledge or sufficient criteria to enter the system demonstration and
production phases. In addition, it does not require a decision review
to enter the demonstration phase of product development. Further, there
is little incentive for DOD program managers to capture knowledge early
in the development process. Instead, the acquisition environment
emphasizes delaying knowledge capture and problem identification since
these events can have a negative influence on obtaining annual program
funding--a key to success for DOD managers. In contrast, commercial
companies encourage their managers to capture product design and
manufacturing knowledge to identify and resolve problems early in
development, before making significant increases in their investment.
GAO is making recommendations to the Secretary of Defense on ways to
improve DOD‘s acquisition process to achieve better outcomes by
incorporating best practices to capture design and manufacturing
knowledge and then use this knowledge as a basis for decisions to
commit significant additional time and money as an acquisition program
progresses through system demonstration and into production.
Principal Findings:
Timely Design and Manufacturing Knowledge Is Critical to Program
Success:
Knowledge that a product‘s design is stable early in the program
facilitates informed decisions about whether to significantly increase
investments and reduces the risk of costly design changes that can
result from unknowns after initial manufacturing begins. Likewise,
later knowledge that the design can be manufactured affordably and with
consistent high quality prior to making a production decision ensures
that targets for cost and schedule during production will be met.
Leading commercial companies do not make significant investments to
continue a product development or its production until they have
knowledge that the product‘s design works and it can be manufactured
efficiently within cost and schedule expectations.
DOD programs that captured knowledge similar to commercial companies
had more successful outcomes. For example, the AIM-9X and the F/A-18E/
F captured design and manufacturing knowledge by key decision points
and limited cost increases to 4 percent or less and schedule growth to
3 months or less. In fact, the AIM-9X had 95 percent of its drawings
completed at its critical design review. The F/A-18E/F had 56 percent
of its drawings completed and also had over 90 percent of its higher
level interface drawings completed, adding confidence in the system
design. Both took steps to ensure that manufacturing processes were
capable of producing an affordable product by the time the programs
made production decisions.
On the other hand, the F-22, PAC-3, and Advanced Threat Infrared
Countermeasures/Common Missile Warning System (ATIRCM/CMWS) programs
did not capture sufficient knowledge before significant investments to
continue the programs and experienced cost growth that ranged from 23
to 182 percent and schedule delays that ranged from
18 months to over 3 years. None of these programs had completed more
than 26 percent of their engineering drawings for their critical design
reviews, and only the F-22 and PAC-3 programs attempted to track the
capability of their critical manufacturing processes prior to
production.
Best Practices Enable Timely Capture of Design and Manufacturing
Knowledge:
Leading commercial companies developed practices that enabled the
timely capture of design and manufacturing knowledge. First, they used
an evolutionary approach to product development by establishing time-
phased plans to develop a new product in increments based on
technologies and resources achievable now and later. This approach
reduced the amount of risk in the development of each increment,
facilitating greater success in meeting cost, schedule, and performance
requirements. The commercial companies GAO visited used the
evolutionary approach as their method for product development. Each
company had a plan for eventually achieving a quantum leap in the
performance of its products and had established an orderly, phased
process for getting there, by undertaking continuous product
improvements as resources became available. For the most part, DOD
programs try to achieve the same leap in performance but in just one
step, contributing to development times that can take over 15 years to
deliver a new capability to the military user.
Second, each leading commercial company had a product development
process that was prominent and central to its success. The process was
championed by executive leadership and embraced by product managers and
development teams as an effective way to do business. Critical to the
product development process were activities that enabled the capture of
specific design and manufacturing knowledge and decision reviews to
determine if the knowledge captured would support the increased
investment necessary to move to the next development phase or into
production. These activities provided knowledge that the product design
was stable at the decision point to start initial manufacturing
(exiting the integration phase) as demonstrated by the completion of 90
percent of the engineering drawings. They also captured knowledge that
a product was ready to begin production (exiting the demonstration
phase) as demonstrated by proof that critical processes were in control
and product reliability was achievable. The activities that enabled the
capture and use of this knowledge to make decisions are listed in table
1.
Table 1: Activities That Enable the Capture of Design and Manufacturing
Knowledge:
[See PDF for Image]
[End of table]
DOD programs that had more successful outcomes used key best practices
to a greater degree than others. For example, the AIM-9X missile
program completed 95 percent of its engineering drawings at the
critical design review because it made extensive use of prototype
testing to demonstrate the design met requirements coupled with design
reviews that included program stakeholders. The F/A-18-E/F program
eliminated over 40 percent of the parts used to build predecessor
aircraft to make the design more robust for manufacturing and
identified critical manufacturing processes, bringing them under
control before the start of production. Both programs developed
products that evolved from existing versions, making the design
challenge more manageable.
On the other hand, DOD programs with less successful outcomes did not
apply best practices to a great extent. At their initial manufacturing
decision reviews, the F-22, PAC-3, and ATIRCM/CMWS had less than one-
third of their engineering drawings, in part, because they did not use
prototypes to demonstrate the design met requirements before starting
initial manufacturing. On the F-22 program, it was almost 3 years after
this review before 90 percent of the drawings needed to build the F-22
were completed. Likewise, at their production decision reviews, these
programs did not capture manufacturing and product reliability
knowledge consistent with best practices. For example, the PAC-3
missile program had less than 40 percent of its processes in control
and, as a result, the missile seekers had to be built, tested, and
reworked on average 4 times before they were acceptable. The F-22
entered production despite being substantially behind its plan to
achieve reliability goals. As a result, the F-22 is requiring
significantly more maintenance actions than planned.
A Better Match of Policy and Incentives Is Needed to Ensure Capture of
Design and Manufacturing Knowledge:
DOD‘s acquisition policy establishes a good framework for developing
weapon systems; however, more specific criteria, disciplined adherence,
and stronger acquisition incentives are needed to ensure the timely
capture and use of knowledge and decision making. DOD recently changed
its acquisition policy to emphasize evolutionary acquisition and
establish separate integration and demonstration phases in the product
development process. Its goal was to develop higher quality systems in
less time and for less cost. While similar to the leading commercial
companies‘ approach, the policy lacks detailed criteria for capturing
and using design and manufacturing knowledge to facilitate better
decisions and more successful acquisition program outcomes. It also
lacks a decision review to proceed from the integration phase to the
demonstration phase of product development.
While the right policy and criteria are necessary to ensure a
disciplined, knowledge-based product development process, the
incentives that influence the key players in the acquisition process
will ultimately determine whether they will be used effectively. In
DOD, current incentives are geared toward delaying knowledge so as not
to jeopardize program funding. This undermines a knowledge-based
process for making product development decisions. Instead, program
managers and contractors push the capture of design and manufacturing
knowledge to later in the development program to avoid the
identification of problems that might stop or limit funding. They focus
more on meeting schedules than capturing knowledge. On the other hand,
commercial companies must develop high-quality products quickly or they
may not survive in the marketplace. Because of this, they encourage
their managers to capture product design and manufacturing knowledge to
identify and resolve problems early in development, before making
significant increases in their investment. Instead of a schedule-driven
process, their process is driven by events that bring them knowledge:
critical design reviews that are supported by completed engineering
drawings and production decisions supported by reliability testing and
statistical process control data. They do not move forward without the
design and manufacturing knowledge needed to make informed decisions.
Recommendations for Executive Action:
GAO recommends that the Secretary of Defense revise policy and guidance
on the operation of the defense acquisition system to include (1) a
requirement to capture specific design knowledge to be used as exit
criteria for transitioning from system integration to system
demonstration and (2) a requirement that the current optional interim
progress review between system integration and demonstration be a
mandatory decision review requiring the program manager to verify that
design is stable and that this be reported in the program‘s Defense
Acquisition Executive Summary and Selected Acquisition Report. The
policy and guidance should also be revised to include (1) a requirement
to capture and use specific manufacturing knowledge at the production
commitment point as exit criteria to transition from system
demonstration into production and (2) a requirement to structure major
weapon system contracts to ensure the capture and use of knowledge for
DOD to make investment decisions at critical junctures when
transitioning from system integration to system demonstration and then
into production.
Agency Comments:
DOD generally agreed with the report and its recommendations. A
detailed discussion of DOD‘s comments appears in appendix I.
[End of section]
Chapter 1: Introduction:
The Department of Defense (DOD) spends close to $100 billion annually
to research, develop, and acquire weapon systems, and this investment
is expected to grow substantially. Over the next 5 years, starting in
fiscal year 2003, DOD‘s request for weapon system development and
acquisition funds is estimated to be $700 billion (see fig. 1).
How effectively DOD manages these funds will determine whether it
receives a good return on its investment. Our reviews over the past 20
years have consistently found that DOD‘s weapon system acquisitions
take much longer and cost much more than originally anticipated,
causing disruptions to the department‘s overall investment strategy and
significantly reducing its buying power. Because such disruptions can
limit DOD‘s ability to effectively execute war-fighting operations, it
is critical to find better ways of doing business.
In view of the importance of DOD‘s investment in weapon systems, we
have undertaken an extensive body of work that examines DOD‘s
acquisition issues from a different, more cross-cutting perspective--
one that draws lessons learned from the best commercial product
development efforts to see if they apply to weapon system acquisitions.
This report looks at the core of the acquisition process, specifically
product development and ways to successfully design and manufacture the
product. Our previous reports looked at such issues as how companies
matched customer needs and resources, tested products, assured quality,
and managed suppliers and are listed in related GAO products at the end
of the report.
Figure 1: Research, Development, Test and Evaluation, and Procurement
Funding for Fiscal Years 1995 to 2007:
[See PDF for image]
Source: DOD.
[End of figure]
Best Practices of Leading Commercial Companies:
Leading commercial companies expect their program managers to deliver
high-quality products on time and within budget. Doing otherwise could
result in the customer walking away. Thus, the companies have created
an environment and adopted practices that put their program managers in
a good position to succeed in meeting these expectations. Collectively,
these practices ensure that a high level of knowledge exists about
critical facets of the product at key junctures during development.
Such a knowledge-based process enables decision makers to be reasonably
certain about critical facets of the product under development when
they need this knowledge.
To ensure the right level of knowledge at each key decision point in
product development, leading commercial companies separate technology
from product development and take steps to ensure the product design is
stabilized early so product performance and producibility can be
demonstrated before production. The process followed by leading
companies, illustrated in figure 2, can be broken down into the
following three knowledge points.
* Knowledge point 1 occurs when a match is made between the customer‘s
needs and the available resources--technology, design, time, and
funding. To achieve this match, technologies needed to meet essential
product requirements must be demonstrated to work in their intended
environment. In addition, the product developer must complete a
preliminary product design using systems engineering to balance
customer desires with available resources.
* Knowledge point 2 occurs when the product‘s design demonstrates its
ability to meet performance requirements. Program officials are
confident that the design is stable and will perform acceptably when at
least 90 percent of engineering drawings are complete. Engineering
drawings reflect the results of testing and simulation and describe how
the product should be built.
* Knowledge point 3 occurs when the product can be manufactured within
cost, schedule, and quality targets and is reliable. An important
indicator of this is when critical manufacturing processes are in
control and consistently producing items within quality standards and
tolerances. Another indicator is when a product‘s reliability is
demonstrated through iterative testing that identifies and corrects
design problems.
Figure 2: Knowledge-based Process for Applying Best Practices to the
Development of New Products:
[See PDF for image]
Source: GAO‘s analysis.
[End of figure]
This report focuses on best practices for achieving knowledge points 2
and 3, particularly at how successful companies design and manufacture
a product within established cost, schedule, and quality targets. The
concepts discussed build on our previous reports, which looked at the
earlier phases of an acquisition, including matching customer needs and
available resources.
A key success factor evident in all our work is the ability to obtain
the right knowledge at the right time and to build knowledge to the
point that decision makers can make informed decisions about moving
ahead to the next phase. Programs that do this typically have
successful cost and schedule outcomes. Programs that do not typically
encounter problems that eventually cascade and become magnified through
the product development and production phases. As shown in figure 3,
the effects of not following a knowledge-based process can be
debilitating.
Figure 3: Notional Illustration Showing the Different Paths That a
Product‘s Development Can Take:
[See PDF for image]
Source: GAO‘s analysis.
[End of figure]
DOD‘s Traditional Approach to Product Development:
DOD has historically developed new weapon systems in a highly
concurrent environment that usually forces acquisition programs to
manage technology, design, and manufacturing risk at the same time.
This environment has made it difficult for either DOD or congressional
decision makers to make informed decisions because appropriate
knowledge has not been available at key decision points in product
development. DOD‘s common practice for managing this environment has
been to create aggressive risk reduction efforts in its programs. Cost
reduction initiatives that typically arise after a program is
experiencing problems are common tools used to manage these risks.
Figure 4 shows the overlapping and concurrent approach that DOD uses to
develop its weapon systems. This figure shows that DOD continues to
capture technology, design, and manufacturing knowledge long after a
program passes through each of the three knowledge points when this
knowledge should have been available for program decisions.
Figure 4: DOD‘s Concurrent Approach to Weapon System Development:
[See PDF for image]
Source: GAO‘s analysis.
[End of figure]
More important, the problems created by this concurrent approach on
individual programs can profoundly affect DOD‘s overall modernization
plans. It is difficult to prioritize and allocate limited budgets among
needed requirements when acquisition programs‘ cost and schedule are
always in question. Programs that are managed without the knowledge-
based process are more likely to have surprises in the form of cost and
schedule increases that are accommodated by disrupting the funding of
other programs. Because of these disruptions, decision makers are not
able to focus on a balanced investment strategy.
DOD‘s Adoption of Best Practices:
DOD is taking steps to change the culture of the acquisition community
with actions aimed at reducing product development cycle times and
improving the predictability of cost and schedule outcomes. DOD
recently made constructive changes to its acquisition policy that
embrace best practices. These changes focused primarily on (1) ensuring
technologies are demonstrated to a high level of maturity before
beginning a weapon system program and (2) taking an evolutionary, or
phased, approach to developing new weapon systems. Because these
changes occurred in 2000 and 2001, it is too early to determine how
effectively they will be put into practice. While these are good first
steps, further use of best practices in product development would
provide a greater opportunity to improve weapon system cost and
schedule outcomes.
Objectives, Scope, and Methodology:
Our overall objective was to determine whether best practices offer
methods to improve the way DOD ensures that the design is stable early
in the development process and whether having manufacturing processes
in control before production results in better cost, schedule, and
quality outcomes in DOD major acquisition programs. Specifically, we
identified best practices that have led to more successful product
development and production outcomes, compared the best practices to
those used in DOD programs, and analyzed current weapon system
acquisition guidance for applicability of best practices.
To determine the best practices for ensuring product design and
manufacturing maturity from the commercial sector, we conducted general
literature searches. On the basis of our literature searches and
discussions with experts, we identified a number of commercial
companies as having innovative development processes and practices that
resulted in successful product development. We visited the following
commercial companies:
* Caterpillar designs and manufactures construction and mining
equipment, diesel and natural gas engines, and industrial gas turbines.
In 2001, it reported sales and revenues totaling $20.45 billion. We
visited its offices in Peoria, Illinois.
* Cummins Inc. (Engine Business group) designs and manufactures diesel
and natural gas engines ranging in size from 60 to 3,500 horsepower for
mining, construction, agriculture, rail, oil and gas, heavy and medium-
duty trucks, buses, and motor homes. In 2001, the Engine Business Group
reported sales of $3.1 billion. We visited its offices in Columbus,
Indiana.
* General Electric Aircraft Engines designs and manufactures jet
engines for civil and military aircraft and gas turbines, derived from
its successful jet engine programs, for marine and industrial
applications. In 2001, it reported earnings totaling $11.4 billion. We
visited its offices in Evendale, Ohio.
* Hewlett Packard designs and manufactures computing systems and
imaging and printing systems for individual and business use. In 2001,
it reported revenues totaling $45.2 billion. We visited its offices
involved in the design and manufacturing of complex ink jet imaging
equipment in Corvallis, Oregon.
* Xerox Corporation designs and manufactures office equipment,
including color and black and white printers, digital presses,
multifunction devices, and digital copiers designed for offices and
production-printing environments. In 2001, it reported revenues
totaling $16.5 billion. We visited its offices in Rochester, New York.
At each of the five companies, we conducted structured interviews with
representatives to gather uniform and consistent information about each
company‘s new product development processes and best practices. During
meetings with these representatives, we obtained a detailed description
of the processes and practices they believed necessary and vital to
mature a product design and get manufacturing processes under control.
We met with design engineers, program managers, manufacturing and
quality engineers, and developers of the knowledge-based processes and
policies.
During the past 5 years, we have gathered information on product
development practices from such companies as 3M, Boeing Commercial
Airplane Group, Chrysler Corporation, Bombardier Aerospace, Ford Motor
Company, Hughes Space and Communications, and Motorola Corporation.
This information enabled us to develop an overall model to describe the
general approach leading commercial companies take to develop new
products.
Our report highlights several best practices in product development
based on our fieldwork. As such, they are not intended to describe all
practices or suggest that commercial companies are without flaws.
Representatives from the commercial companies visited told us that the
development of their best practices has evolved over many years and
that the practices continue to be improved based on lessons learned and
new ideas and information. They admit that the application and use of
these have not always been consistent or without error. However, they
strongly suggested that the probability of success in developing new
products is greatly enhanced by the use of these practices. Further,
because of the sensitivity to how data that would show the actual
outcomes of new product development efforts might affect their
competitive standing, we did not obtain specific cost, schedule, and
performance data. Most examples provided by these companies were
anecdotal. However, the continued success of these companies over time
in a competitive marketplace indicated that their practices were
important and key to their operations. Furthermore, based on our
observations during meetings at these companies, it was apparent that
because of the level of detailed process tools developed for their
managers and executive leadership these best practices were a
centerpiece of their operations.
Next, we compared and contrasted the best practices with product
development practices used in five DOD major acquisition programs.
Below is a brief description of each program we examined:
* The F-22 fighter aircraft program. This aircraft is designed with
advanced features to allow it to be less detectable to adversaries,
capable of high speeds for long ranges, and able to provide the pilot
with improved awareness of the surrounding situation through the use of
integrated avionics. The F-22 program began in 1986 and entered limited
production in 2001. The Air Force expects to buy 341 at a total
acquisition cost (development and procurement) estimated at $69.7
billion.
* The Patriot Advanced Capability (PAC-3) missile program. This program
is intended to enhance the Patriot system, an air-defense, guided
missile system. PAC-3 is designed to enhance the Patriot radar‘s
ability to detect and identify targets, increase system computer
capabilities, improve communications, increase the number of missiles
in each launcher, and incorporate a new ’hit-to-kill“ missile. The
’hit-to-kill“ missile capabilities represent a major part of the
development program, as these are not capabilities included in prior
versions of the Patriot system. The missile program began in 1994 and
entered limited production in 1999. The Army plans to buy 1,159
missiles at a total acquisition cost estimated at $8.5 billion.
* The Advanced Threat Infrared Countermeasures/Common Missile Warning
System (ATIRCM/CMWS) program. ATIRCM/CMWS is a defensive countermeasure
system for protection against infrared guided missiles. The common
missile warning system detects missiles in flight, and the advanced
threat infrared countermeasure defeats the missile with the use of a
laser. The combined system is designed for helicopter aircraft. The
common missile warning system is also designed for tactical aircraft
such as fighters. The program began in 1995 and is expected to start
limited production in 2002. The Army and the Special Operations Command
plan to buy 1,078 systems at a total acquisition cost estimated at $2.9
billion.
* The AIM-9X missile program. AIM-9X is an infrared, short range, air-
to-air missile carried by Navy and Air Force fighter aircraft. The AIM-
9X is an extensive upgrade of the AIM-9M. The AIM-9X is planned to have
increased resistance to countermeasures and improved target acquisition
capability. A key feature is that it will have the ability to acquire,
track, and fire on targets over a wider area than the AIM-9M. The AIM-
9X program began in 1994 and entered limited production in 2000. DOD
plans to buy 10,142 missiles at a total acquisition cost estimated at
$3 billion.
* The F/A-18 E/F fighter aircraft program. This aircraft is intended to
complement and eventually replace the current F/A-18 C/D aircraft and
perform Navy fighter escort, strike, fleet air defense, and close air
support missions. It is the second major model upgrade since the F/A-18
inception. The development program began in 1992. The program entered
limited production in 1997 and full rate production in 2000. The Navy
plans to buy 548 aircraft at a total acquisition cost estimated at
$48.8 billion.
We selected these programs for review based on cost, schedule, and
performance data presented in the Selected Acquisition Reports[Footnote
3] for each program. We also selected these programs because we
considered them to be in two basic categories--successful and
unsuccessful cost and schedule performance outcomes. This basis for
selection was to compare and contrast the development practices used on
each with best practices used by the commercial companies. For each
program, we interviewed key managers and design and manufacturing
engineering representatives. In some cases, we discussed design and
manufacturing issues with representatives of the primary contractor for
the specific program to obtain information on the practices and
procedures used by the program to ready the product design for initial
manufacturing and testing as well as for production. We also discussed
the use and potential application of best practices that we identified.
In addition to discussions, we analyzed significant amounts of data on
engineering drawings, design changes, labor efficiencies,
manufacturing processes, quality indicators, testing, and schedules. We
did not verify the accuracy of the data but did correlate it to other
program indicators for reasonableness. Our analysis of the data was
used as a basis to develop indicators of each program‘s development
efficiencies and detailed questions to discuss product design and
manufacturing practices.
We conducted our review between May 2001 and April 2002 in accordance
with generally accepted government auditing standards.
[End of section]
Chapter 2 Timely Design and Manufacturing Knowledge Is Critical to
Program Success:
The success of any effort to develop a new product hinges on having the
right knowledge at the right time. Every program eventually achieves
this knowledge; however, leading commercial companies we visited have
found that there is a much better opportunity to meet predicted cost,
schedule, and quality targets when it is captured early, in preparation
for critical decisions. Specifically, knowledge that a product‘s design
is stable early in the program facilitates informed decisions about
whether to significantly increase investments and reduces the risk of
costly design changes that can result from unknowns after initial
manufacturing begins. This knowledge comes in the form of completed
engineering drawings before transitioning from the system integration
phase to the system demonstration phase of product development. Best
practices suggest that at least 90 percent of the drawings for a
product‘s design be completed before a decision to commit additional
resources is made. Likewise, later knowledge that the design can be
manufactured affordably and with consistent high quality prior to
making a production decision ensures that cost and schedule targets
will be met. This knowledge comes in the form of evidence from data
that shows manufacturing processes are in control and system
reliability is achievable. Leading commercial companies rely on
knowledge obtained about critical manufacturing processes and product
reliability to make their production decisions.
The Department of Defense (DOD) programs we reviewed captured varying
amounts of design and manufacturing knowledge in the form of completed
engineering drawings and statistical process control data. We found a
correlation between the amount of knowledge each captured and their
cost and schedule outcomes. Programs that were able to complete more
engineering drawings and control their critical manufacturing processes
had more success in meeting cost and schedule targets established when
they began.
DOD Programs Had Better Outcomes When Design and Manufacturing
Knowledge Was Captured at Key Program Junctures:
Conceptually, the product development process has two phases: a system
integration phase to stabilize the product‘s design and a system
demonstration phase to demonstrate the product can be manufactured
affordably and work reliably. The system integration phase is used to
stabilize the overall system design by integrating components and
subsystems into a product and by showing that the design can meet
product requirements. When this knowledge is captured, knowledge point
2 has been achieved. It should be demonstrated by the completion of at
least 90 percent of engineering drawings, which both DOD and leading
commercial companies consider to be the point when a product‘s design
is essentially complete. In the DOD process, this knowledge point
should happen by the critical design review, before system
demonstration and the initial manufacturing of production
representative products begins. The system demonstration phase is then
used to demonstrate that the product will work as required and can be
manufactured within targets. When this knowledge is captured, knowledge
point 3 has been achieved. Critical manufacturing processes are in
control and consistently producing items within quality standards and
tolerances for the overall product. Also, product reliability has been
demonstrated. In the DOD process, like with the commercial process,
this knowledge point should happen by the production commitment
milestone. Bypassing critical knowledge at either knowledge point will
usually result in cost, schedule, and performance problems later in
product development and production.
We found that the most successful programs had taken steps to gather
knowledge that confirmed the product‘s design was stable before the
design was released to manufacturing organizations to build products
for demonstration. They had most of the detailed design complete,
supported by the completion of a large percentage of engineering
drawings to manufacturing. Again, engineering drawings are critical
because they include details on the parts and work instructions needed
to make the product and reflect the results of testing. These drawings
allowed manufacturing personnel to effectively plan the fabrication
process and efficiently build production representative prototypes in
the factory so manufacturing processes and the product‘s performance
could be validated before committing to production. The most successful
DOD programs also captured the knowledge that manufacturing processes
needed to build the product would consistently produce a reliable
product by the end of system demonstration, before making a production
decision. On these programs, the initial phase of production--sometimes
known as low-rate initial production--was able to focus on building
operational test articles and improving the production processes,
instead of continuing the product‘s design and development.
Problematic programs moved forward into system demonstration without
the same knowledge from engineering drawings that successful cases had
captured. They increased investments in tooling, people, and materials
before the design was stable. In these programs, only a small
percentage of the drawings needed to make the products had been
completed at the time the designs were released to manufacturing
organizations for building production representative prototypes. In
doing so, these programs undertook the difficult challenge of
stabilizing the designs at the same time they were trying to build and
test the products. This design immaturity caused costly design changes
and parts shortages that, in turn, caused labor inefficiencies,
schedule delays, and quality problems. Consequently, these programs
required significant increases in resources--time and money--over what
was estimated at the point each program began the system demonstration
phase.
The most problematic programs also started production before design and
manufacturing development work was concluded. In these cases, programs
were producing items for the customers while making major product
design and tooling changes, still establishing manufacturing processes,
and conducting development testing. These programs encountered
significant cost increases, schedule delays, and performance problems
during production.
Table 2 shows the relationship between design stability and
manufacturing knowledge at key junctures and the outcomes for the DOD
programs we reviewed. To measure design stability at the start of the
system demonstration phase, knowledge point 2, we determined the
percentage of the product‘s engineering drawings that had been
completed by the critical design review. In DOD programs, after the
critical design review, the system design is released to manufacturing
to begin building the production representative prototypes for the
system demonstration phase. To measure producibility at the production
decision, knowledge point 3, we determined whether the critical
manufacturing processes were in statistical control at that time. We
compared this information with best practices. The cost and schedule
experiences of the program since the start of system demonstration are
also shown.
Table 2: Attainment of Design and Manufacturing Knowledge in DOD
Programs and the Program Outcomes:
Weapon system: Best practice; Percentage of drawings completed prior to
manufacturing: At least 90 percent of drawings completed; Percentage of
critical manufacturing processes in control at production: All critical
processes in statistical control; Program experience since system
demonstration started: Meet cost and schedule targets.
Weapon system: AIM-9X (air to air missile); Percentage of drawings
completed prior to manufacturing: 95 percent; Percentage of critical
manufacturing processes in control at production: Unknown[A]; Program
experience since system demonstration started: 4 percent unit cost
increase,; 1-month production delay.
Weapon system: FA-18 E/F fighter; Percentage of drawings completed
prior to manufacturing: 56 percent[B]; Percentage of critical
manufacturing processes in control at production: 78 percent; Program
experience since system demonstration started: No unit cost increase,;
3-month production delay.
Weapon system: F-22 fighter; Percentage of drawings completed prior to
manufacturing: 26 percent; Percentage of critical manufacturing
processes in control at production: 44 percent; Program experience
since system demonstration started: 23 percent unit cost increase,; 18-
month production delay.
Weapon system: Patriot Advanced Capability (PAC-3) missile; Percentage
of drawings completed prior to manufacturing: 21 percent; Percentage of
critical manufacturing processes in control at production: 35 percent;
Program experience since system demonstration started: 159 percent unit
cost increase,; 39-month production delay.
Weapon system: Advanced Threat Infrared Countermeasures/Common Missile
Warning System (ATIRCM/CMWS); Percentage of drawings completed prior to
manufacturing: 21 percent; Percentage of critical manufacturing
processes in control at production: 0; Program experience since system
demonstration started: 182 percent unit cost increase,; 34-month
production delay.
[A] While AIM-9X used statistical process control on a limited basis,
we believe other factors contributed to a successful production outcome
to date. Other factors included early achievement of design stability,
early identification of key characteristics and critical manufacturing
processes, use of established manufacturing processes for components
common to other weapon systems, design trade-offs to enhance
manufacturing capability, and a product design less vulnerable to
variations in manufacturing processes.
[B] F/A-18 E/F had 56 percent of drawings completed but also had
completed most of the higher-level assembly drawings. The combination
of these drawings with the fact the aircraft was a variant of
previously fielded F-18 aircraft models provided the program a
significant amount of knowledge that the design was stable at the start
of system demonstration.
Source: DOD program offices and Selected Acquisition Reports.
[End of table]
As shown in the table, the AIM-9X and FA-18 E/F programs had captured a
significant amount of design knowledge at the start of system
demonstration and manufacturing knowledge by the start of production.
In each of those programs, product developers had the advantage of
prior versions of the systems. These programs came very close to
meeting their original cost and schedule estimates for product
development. The other three programs, F-22, PAC-3, and ATIRCM/CMWS,
had less knowledge at each key junctures. Their development cost and
schedule results significantly exceeded estimates. Specific details on
the AIM-9X, F-22, and ATIRCM/CMWS program experiences follow.
AIM-9X Program Experience:
The AIM-9X program began in 1994, continuing the long-term evolution of
the AIM-9 series of short-range air-to-air missiles. In 1999, after
developing and testing a number of engineering prototype missiles, the
program held a critical design review to determine if the program was
ready to begin initial manufacturing of a production representative
prototype for system demonstration. At this review, about 95 percent of
the eventual engineering drawings were completed--a stable design by
best practices. Because
AIM-9X was the next generation in this family of missiles, the program
had significant knowledge on how to produce the missile. At the 1999
critical design review, the estimated development and production costs
totaled $2.82 billion. As of December 2001, the estimate was $2.96
billion, less than a 5 percent increase.
F-22 Program Experience:
The F-22 program began detailed design efforts in 1991 when it entered
a planned 8-year product development phase. In 1995, about the expected
midpoint of the phase, the program held its critical design review to
determine if the design was stable and complete. Despite having only
about a quarter of the eventual design drawings completed for the
system, the program declared the design to be stable and ready to begin
initial manufacturing. At that time, the program office had estimated
the cost to complete the development program at $19.5 billion. However,
the program did not complete 90 percent of its drawings for the
aircraft until 1998,
3 years into the system demonstration phase. During the building of the
initial aircraft, several design and manufacturing problems surfaced
that affected the deliveries of major sections of the aircraft. Large
sections were delivered incomplete to final assembly and had to be
built out of the planned assembly sequence.
In 1997, an independent review team examined the program and determined
the product development effort was underestimated. The team found that
building the first three aircraft was taking substantially more labor
hours than planned. Between 1995 and 1998, the development estimate for
the F-22 increased by over $3.3 billion and the schedule slipped by a
year. Achieving design stability late has contributed to further cost
increases. As of December 2001, the estimated development cost was
$26.1 billion, a 34 percent increase since the critical design review
was held in 1995.
While the program attributes some production cost increases to a
reduction in F-22 quantities, it has been significantly affected by
design and manufacturing problems that started during development. The
independent review team evaluated the cost impact on the production
aircraft that would likely occur because of cost and schedule problems
in development and found that production aircraft would have to begin
later, at a slower pace, and cost more than expected. The team
estimated that production costs could increase by as much as $13
billion if savings were not found. The Air Force subsequently increased
the estimate to more than $19 billion in cost savings required to avoid
cost increases. In 2001, when the F-22 limited production decision was
made, the program had less knowledge about the aircraft‘s reliability
and manufacturing processes than more successful cases. For example, at
its limited production decision, it had only 44 percent of its critical
manufacturing processes in control. In September 2001, the program
reported that overall production cost would likely increase by more
than $5.4 billion. This estimate was based on the effort needed so far
to build the aircraft during product development.
ATIRCM/CMWS Program Experience:
Since it began in 1995, the ATIRCM/CMWS program has had significant
cost growth and schedule delays during product development. The product
developer held a major design review in 1997. Like the F-22, the review
demanded less proof about the product‘s design in the form of
engineering drawings before deciding to begin initial manufacturing. At
that time, only 21 percent of the engineering drawings had been
completed, and it was still unknown whether the design would meet the
requirements. In fact, the program knew that a major redesign of a
critical component was needed. Despite this, the program office deemed
the risk acceptable for moving the program forward to begin
manufacturing prototypes. Over the next 2 years, the program
encountered numerous design and manufacturing problems. It was not
until 1999, about 2 years after the critical design review, that
program officials felt that the design had stabilized; however, by this
time, the product development cost had increased 160 percent and
production had been delayed by almost 3 years.
ATIRCM/CMWS is scheduled to begin limited production in early 2002, but
without the same degree of assurance as the more successful programs
that the product can be manufactured within cost, schedule, and quality
targets. The program has not yet determined if manufacturing processes
needed to build the product are in control. Many of the development
units were built by hand, in different facilities, and with different
processes and personnel. Program officials stated that because they did
not stabilize the design until late in development, manufacturing
issues were not adequately addressed. Since 1997, the estimated unit
cost for the system has increased by 182 percent.
[End of section]
Chapter 3 Best Practices Enable Timely Capture of Design and
Manufacturing Knowledge:
Leading commercial companies have been successful in achieving product
development goals because they have found ways to enable the capture of
design and manufacturing knowledge about the products they are
developing in a timely way. We found two practices that allowed leading
commercial companies to capture necessary knowledge for product
development. First, they established a framework of evolutionary
product development that limited the amount of design and manufacturing
knowledge that had to be captured. This framework limited the design
challenge for any one new product development by requiring risky
technology, design, or manufacturing requirements to be deferred until
a future generation of the product. Second, each company (1) employed a
disciplined product development process that brought together and
integrated all of the technologies, components, and subsystems required
for the product to ensure the design was stable before entering product
demonstration and (2) demonstrated the product was reliable and
producible using proven manufacturing processes before entering
production.
The product development process includes tools that both capture
knowledge and tie this knowledge to decisions about the product‘s
design and manufacturing processes before making commitments that would
significantly affect company resources. For example, during system
integration, each leading commercial company used various forms of
prototypes and information from predecessor products to stabilize the
product‘s design and identify critical processes, then used a decision
review that required agreements from key stakeholders that the
requisite design knowledge was captured in making a decision to move
into system demonstration. During system demonstration, each company
used statistical process control and reliability testing to ensure the
product could be produced affordably and would be reliable, then used a
similar decision review that required agreements from key stakeholders
that the requisite knowledge was captured when deciding to move into
production.
The Department of Defense (DOD) programs that we reviewed used some of
these practices to varying degrees and experienced predictable
outcomes. For example, the AIM-9X and F/A-18 E/F programs were
evolutionary in nature, modifications of existing products with a
manageable amount of new technological or design challenges. They also
gathered design and manufacturing knowledge, although not to the extent
we found at commercial companies. Finally, they held program reviews
and ensured that the design and manufacturing knowledge was captured
before moving forward. They had relatively successful outcomes. The
other DOD programs--the F-22, ATIRCMS, and PAC-3--did not closely
approximate best practices in capturing design or manufacturing
knowledge during product development. They took on greater design
challenges, had program reviews that were not supported by critical
design and manufacturing knowledge, and made decisions to advance to
the next phases of development without sufficient design and
manufacturing knowledge.
Leading Commercial Companies Use Evolutionary Product Development
Framework to Reduce Development Risks:
A key to the success of commercial companies was using an evolutionary
approach to develop a product. This approach permitted companies to
focus more on design and development with a limited array of new
content and technologies in a program. It also ensured that each
company had the requisite knowledge for a product‘s design before
investing in the development of manufacturing processes and facilities.
Companies have found that trying to capture the knowledge required to
stabilize the design of a product that requires significant amounts of
new content is an unmanageable task, especially if the goal is to
reduce cycle times and get the product into the marketplace as quickly
as possible. Design elements not achievable in the initial development
were planned for subsequent development efforts in future generations
of the product, but only when technologies were proven to be mature and
other resources were available.
Commercial companies have implemented the evolutionary approach by
establishing time-phased plans to develop new products in increments
based on technologies and resources achievable now and later. This
approach reduces the amount of risk in the development of each
increment, facilitating greater success in meeting cost, schedule, and
performance requirements. In effect, these companies evolve products,
continuously improving their performance as new technologies and
methods allow. These evolutionary improvements to products eventually
result in the full desired capability, but in multiple steps,
delivering a series of enhanced interim capabilities to the customer
more quickly.
Historically, DOD‘s approach has been to develop new weapon systems
that often attempt to satisfy the full requirement in a single step,
regardless of the design challenge or the maturity of technologies
necessary to achieve the full capability. Under this single-step
approach, a war fighter can wait over 15 years to receive any improved
capability. Figure 5 shows a notional comparison between the single-
step and evolutionary approaches.
Figure 5: Notional Single-Step and Evolutionary Approaches to
Developing New Products:
[See PDF for image]
Source: GAO‘s analysis and DOD acquisition guidance.
[End of figure]
Each commercial company we visited used the evolutionary approach as
the primary method of product development. General Electric builds on
the basic capability of a fielded product by introducing proven
improvements in capability from its advanced engineering development
team. General Electric considers the introduction of immature
technologies into fielded products or new engine development programs
as a significant cost and schedule risk. Its new product development
process is primarily focused on reducing and managing risk for design
changes and product introductions. Cummins and Hewlett Packard managers
indicated that, in the past, their companies learned the hard way by
trying to make quantum leaps in product performance and by including
immature technologies. Now, both companies have new product development
processes that actively manage the amount of new content that can be
placed on a new product development effort. Caterpillar also limits new
content on its new products as a way to more successfully and cost-
effectively develop new, but evolutionary, products. Even during the
development of its 797 mining truck, which it considered a major design
challenge, it did not require the truck to achieve capabilities--such
as prognostics for better maintenance--that it could not demonstrate or
validate in the design in a timely manner.
Of the five DOD programs we reviewed, two--the F/A-18-E/F and the
AIM-9X--were variations of existing products--the F/A-18-C/D and the
AIM-9M--and the programs made a commitment to use existing technologies
and processes as much as possible. These two programs had relatively
successful cost and schedule outcomes. They represented an exception to
the usual practice in DOD. The overwhelming majority of DOD‘s major
acquisitions today require major leaps in capability over their
predecessors or any other competing weapon systems, with little
knowledge about the resources that will be required to design and
manufacture the systems. Decisions are continually made throughout
product development without knowing the cost and schedule
ramifications.
Leading Commercial Companies Use a Product Development Process to
Capture Design and Manufacturing Knowledge for Decision Making:
Leading commercial companies we visited had spent significant amounts
of time and resources to develop and evolve new product development
processes that ensured design and manufacturing knowledge was captured
at the two critical decision points in product development: when the
product‘s design was demonstrated to be stable--knowledge point 2--and
when the product was demonstrated to be producible at an affordable
cost--knowledge point 3. The process established a disciplined
framework to capture specific design and manufacturing knowledge about
new products. Companies then used that knowledge to make informed
decisions about moving forward in a new product development program.
Commercial companies tied this knowledge to decisions about the
products‘ design and manufacturing processes before making commitments
that would significantly impact company resources. Each commercial firm
we visited had a new product development process that was prominent and
central to the firm‘s successes. It included three aspects: (1)
activities that led to the capture of specific design knowledge, (2)
activities that led to the capture of specific manufacturing and
product reliability knowledge, and (3) decision reviews to determine if
the appropriate knowledge was captured to move to the next phase.
Design Knowledge Should Be Captured before Entering Product
Demonstration:
To ensure that the product‘s design was stable before deciding to
commit additional resources to product demonstration, commercial
companies demanded knowledge, either from existing product information
or by building engineering prototypes. They also used a disciplined
design review process to examine and verify the knowledge that had
culminated at the end of product integration, This design review
process required agreement from stakeholders that the product design
could be produced and would satisfy the customer‘s requirements.
Stakeholders included design engineers, manufacturing or production
personnel, and key supplier representatives who used engineering
drawings, supported by test results and engineering data, as a key
indicator of the design‘s stability. Once the program achieved a stable
design, the certainty of their cost and schedule estimates was
substantially increased, allowing them to plan the balance of the
product development program with high confidence. Table 3 shows the
activities required to capture design knowledge that leads to executive
decisions about whether to transition to the next phase of development.
Table 3: Activities to Capture Design Knowledge and Make Decisions:
Knowledge: Design is stable and performs as expected (knowledge point
2): Activities to Achieve Stable Design Knowledge; * Limit design
challenge - The initial design challenge is limited to a product that
can be developed and delivered quickly and provide the user with an
improved capability. A time-phased plan is used to develop improved
products--future generations--in increments as technologies and other
resources become available.; * Demonstrate design meets requirements -
The product‘s design is demonstrated to meet the user‘s requirements.
For a new product that is not based on an existing product, prototypes
are built and tested. If the product is a variant of an existing
product, companies often used modeling and simulation or prototypes at
the component or subsystem level to demonstrate the new product‘s
design.; * Complete critical design reviews - Critical design reviews
are used to assess whether a product‘s design meets requirements and is
ready to start initial manufacturing. They are conducted for the
system, subsystems, and components to assess design maturity and
technical risk.; * Stakeholders agree drawings complete and producible
- The agreement by stakeholders (engineers, manufacturers, and other
organizations) is used to signify confidence that the design will work
and the product can be built.; * Executive level review to begin
initial manufacturing - Corporate stakeholders meet and review relevant
product knowledge, including design stability, to determine whether a
product is ready to initiate manufacturing of production representative
prototypes used during system demonstrations. The decision is tied to
the capture of knowledge..
[End of table]
Demonstrating the Design Helped Achieve Stability:
A key tool used by each company to ensure that a product‘s design was
stable by the end of the product integration phase was a demonstration
that the design would meet requirements. The companies visited
indicated that prototypes at various system levels were the best way to
demonstrate that the product‘s design would work. If the product under
development was an incremental improvement to existing products, such
as the next generation of a printer or engine, these companies used
virtual prototypes for any components that were being used for the
first time. If the product included more new content or invention,
fully integrated prototypes were frequently used to demonstrate that
the design met requirements. Prototypes at this stage in development
were typically not built in a manufacturing facility. This allowed
demonstrations of the design before the companies made more costly
investments in manufacturing equipment and tooling to build production
representative prototypes for the demonstration phase. Table 4 shows an
example of the types and purposes for various kinds of prototypes used
by Cummins Inc. depending on the amount of knowledge it needed to
capture and the point it was in the development process. Prototypes
were used by commercial companies throughout the product development
process and not just during product integration.
Table 4: Examples of Prototypes Used by Cummins Inc. at Various Stages
of Product Development:
Prototype; Product integration: Engineering prototypes (virtual or
physical); Product demonstration: Production representative
prototypes; Production: Initial products.
Purpose; Product integration: Demonstrate form, fit and function, and a
stable design; Product demonstration: Demonstrate the product is
capable, reliable, and manufacturing processes in statistical control;
Production: Demonstrate ready for full rate production.
Build environment; Product integration: Engineering; Product
demonstration: Manufacturing; (1st set of production tooling);
Production: Production (all rate tooling).
[End of table]
Cummins, the world sales leader in diesel engines over 200 horsepower,
effectively uses prototypes to ensure that a design is stable and
believes in the value of prototyping throughout product development. A
Cummins representative stated that not using prototypes becomes a
matter of ’pay me now or pay me later,“ meaning that it is far less
costly to demonstrate a product‘s design early in development with
prototypes, concepts, and analyses than to incur the cost of
significant design changes after a product has entered production--a
much more costly environment to make changes. Cummins built and tested
12 engineering concept prototype engines for its Signature 600 engine,
a new concept, 600 horsepower, overhead cam diesel engine that
represented a quantum leap in performance beyond Cummins‘ existing
products. These prototypes were built using production-like tooling and
methods using production workers. In addition to using engineering
prototypes during the product integration phase of product development,
Cummins and other companies we visited used other prototypes--such as
production representative prototypes--in the remaining product
development phases before production, as shown in table 4, to
demonstrate product reliability and process control. Prior to reaching
production for its Signature 600 engine, Cummins used many prototypes
to complete hundreds of thousands of test hours, accumulating millions
of test miles.
Caterpillar, a major manufacturer of heavy equipment, has a continuous
product improvement philosophy. That is, it tries to develop new
products that increase the capabilities of existing product lines, but
it limits the amount of new content on any one product development
because new content inherently increases design risk. In evolving its
products this way, Caterpillar is able to use modeling and simulation
prior to initial manufacturing because it has existing products to
provide a baseline of knowledge and a good benchmark for assessing the
simulated performance. In addition, with knowledge of existing
components, it can focus attention on maturing the new content, the
higher risk element of the new product. When Caterpillar developed the
797 mining truck, a new 360-ton payload truck design, it demonstrated
design stability by identifying the critical components and building
engineering prototypes of them for reliability testing and
demonstration of the design before beginning initial manufacturing.
This knowledge, coupled with vast experience in manufacturing trucks,
ensured the stability of the 797-truck design before initial
manufacturing started. Caterpillar was able to deliver this design in
18 months after the product development was started.
Disciplined Reviews and Stakeholder Agreements Supported the Capture of
Design Knowledge:
The commercial companies we visited understood the importance of having
disciplined design reviews and getting agreement from the stakeholders
that the product‘s design had been demonstrated to meet requirements
before beginning initial manufacturing. Each company had a design
review process that began at the component level, continued through the
subsystem level, and culminated with a critical design review of the
integrated system to determine if the product was ready to progress to
the next phase of development. In addition to design engineers, a
cross-functional team of stakeholders in the process included key
suppliers, manufacturing representatives, and service and maintenance
representatives. From past experience, commercial companies have
discovered that cross-functional teams provide a complete perspective
of the product. While design engineers bring important skills and
experience to creating a product design, they may not be aware of
manufacturing issues, available technologies, or manufacturing
processes, and they may design a product that the company cannot afford
to produce or maintain.
The product‘s design is stable when all stakeholders agree that
engineering drawings are complete and that the design will work and can
be built. A commercial company considers engineering drawings[Footnote
4] to be a good measure of the demonstrated stability of the product‘s
design because they represent the language used by engineers to
communicate to the manufacturers the details of a new product design--
what it looks like, how its components interface, how it functions, how
to build it, and what critical materials and processes are required to
fabricate and test it. The engineering drawing package released to
manufacturing includes items such as the schematic of the product‘s
components, interface control documents, a listing of materials,
notations of critical manufacturing processes, and testing
requirements. It is this package that allows a manufacturer to build
the product in the manufacturing facility.
In developing the Signature 600, Cummins used cross-functional design
teams that included stakeholders from suppliers, machine tool
manufacturers, foundry and pattern makers, purchasing, finance,
manufacturing engineering, design engineering, and other technical
disciplines. Signature 600 components were designed with the key
suppliers co-located at the Cummins design facility. Likewise,
Caterpillar said that early supplier and manufacturing involvement was
critical to success and that engineering drawings were signed by design
and manufacturing stakeholders. Caterpillar representatives said that
signing the drawings was a certification that the design could be
manufactured the next day, if necessary.
Executive Level Reviews Were Required to Begin Initial Manufacturing:
Each commercial company, after capturing specific design knowledge, had
an executive level review at the decision point to determine if the
product design had sufficiently progressed to permit a transition from
product integration to product demonstration. This decision point used
the knowledge captured as exit criteria for moving to the next phase of
development. For example, to demonstrate the product design was stable
and ready to move from integration to demonstration, the design had to
be demonstrated, at least 90 percent of the engineering drawings had to
be completed, design reviews had to be completed, and stakeholders had
to agree the design was complete and producible. If the design team
could not satisfy the exit criteria, then other options had to be
considered. Options included canceling the development program,
delaying the decision until all criteria were met, or moving ahead with
a detailed plan to achieve criteria not met by a specific time when
leadership would revisit the other options. One company emphasized that
if a major milestone is delayed, an appropriate adjustment should be
made to the end date of the program, thereby avoiding compressing the
time allotted for the rest of product development and managing the
risks that subsequent milestones will be missed.
This decision point coincides with the companies‘ need to increase
investments in the product development and continue to the next phase.
For this reason, the decision point was considered critical to
achieving success in product development and could not be taken
lightly. For example, transitioning from the integration to the
demonstration phase requires a significant investment to start building
and testing production representative prototypes in a manufacturing
environment. This requires establishing a supplier base and purchasing
materials. In addition, establishing tooling and manufacturing
capability is also required. After a product passes this decision point
and added investments are made, the cost of making changes to the
product design also increases significantly. Therefore, commercial
companies strive to firm the design as early in the process as possible
when it is significantly cheaper to make changes.
Manufacturing and Product Reliability Knowledge Should Be Captured
before Starting Production:
We found that leading commercial companies used two tools to capture
knowledge that a product‘s design was reliable and producible within
cost, schedule, and quality targets before making a production
decision. These tools are (1) a quality concept that uses statistical
process control to bring critical manufacturing processes under control
so they are repeatable, sustainable, and consistently producing parts
within the quality tolerances and standards of the product and (2)
product tests in operational conditions that ensure the system would
meet reliability goals-the ability to work without failure or need of
maintenance for predictable intervals. Company officials told us that
these two tools enabled a smooth transition from product development to
production, resulting in better program outcomes. Companies employed
these tools on production representative prototypes, making the
prototypes a key ingredient to successful outcomes. Table 5 shows the
activities required to capture manufacturing knowledge that leads to
executive decisions about whether to transition from product
development into production.
Table 5: Activities to Capture Manufacturing Knowledge and Make
Decisions:
Knowledge: Product can be produced within cost, schedule, and quality
targets (knowledge point 3): Activities to Achieve Manufacturing
Knowledge; * Identify key system characteristics and critical
manufacturing processes - Key product characteristics and critical
manufacturing processes are identified. Because there can be thousands
of manufacturing processes required to build a product, companies focus
on the critical processes--those that build parts that influence the
product‘s key characteristics such as performance, service life, or
manufacturability.; * Determine processes in control and capable -
Statistical process control is used to determine if the processes are
consistently producing parts. Once control is established, an
assessment is made to measure the process‘s ability to build a part
within specification limits as well as how close the part is to that
specification. A process is considered capable when it has a defect
rate of less than 1 out of every 15,152 parts produced.; * Conduct
failure modes and effects analysis - Bottom-up analysis is done to
identify potential failures for product reliability. It begins at the
lowest level of the product design and continues to each higher tier of
the product until the entire product has been analyzed. It allows early
design changes to correct potential problems before fabricating
hardware.; * Set reliability growth plan and goals - A product‘s
reliability is its ability to perform over an expected period of time
without failure, degradation, or need of repair. A growth plan is
developed to mature the product‘s reliability over time through
reliability growth testing so that it has been demonstrated by the time
production begins.; * Conduct reliability growth testing -Reliability
growth is the result of an iterative design, build, test, analyze, and
fix process for a product‘s design with the aim of improving the
product‘s reliability over time. Design flaws are uncovered and the
design of the product is matured.; * Conduct executive level review to
begin production - Corporate stakeholders meet and review relevant
product knowledge, including manufacturing and reliability knowledge,
to determine whether a product is ready to begin production. The
decision is tied to the capture of knowledge..
[End of table]
Statistical Process Control Is Important to Controlling Critical
Manufacturing Processes:
Commercial companies rely on statistical process control data to track,
control, and improve critical manufacturing processes before production
begins. Bringing processes under statistical control reduces variations
in parts manufactured, thus reducing the potential for defects. Product
variation has been called the ’silent killer“ on the manufacturing
floor because it can result in defects that require additional
resources to either rework or scrap the product. Products fielded with
defects may have degraded performance, lower reliability, or increased
support costs. Experience has taught commercial companies that it is
less costly--in terms of time and money--to eliminate product variation
by controlling manufacturing processes than to perform extensive
inspection after a product is built. Because thousands of manufacturing
processes can be required to build a product, companies focus on the
critical processes--those that build parts that influence the product‘s
performance, service life, or manufacturability. Therefore, when design
engineers are designing the new product, they must identify its key
characteristics so that manufacturing engineers can identify and
control critical manufacturing processes. Key product characteristics
and critical manufacturing processes are noted on the engineering
drawings and work instructions that are released to manufacturing.
Once critical processes are identified, companies perform capability
studies to ensure that a process will produce parts that meet
specifications. These studies yield a process capability index (Cpk), a
measure of the process‘s ability to build a part within specified
limits. The index can be translated into an expected product defect
rate. The industry standard is to have a Cpk of 1.33 or higher, which
equates to a probability that 99.99 percent of the parts built on that
process will be within the specified limits. Four of the five[Footnote
5] companies we visited wanted their critical processes at a minimum of
a 1.33 Cpk and many had goals of achieving higher Cpks. Table 6 shows
various Cpk values and the defect rate associated with each value. The
table also shows the higher the Cpk, the lower the defect rate.
Table 6: Cpk Index and Probability of a Defective Part:
Manufacturing process capability (Cpk): Cpk - .67 (not capable);
Associated defect rate: 1 in 22 parts produced.
Manufacturing process capability (Cpk): Cpk - 1.0 (marginally capable);
Associated defect rate: 1 in 370 parts produced.
Manufacturing process capability (Cpk): Cpk - 1.33 (industry standard);
Associated defect rate: 1 in 15,152 parts produced.
Manufacturing process capability (Cpk): Cpk - 2.0 (industry growth
goal); Associated defect rate: 1 in 500,000,000 parts produced.
[End of table]
Cpk values also have an additive effect on various individual parts
when each part is integrated into the final product. For example, a
product composed of 25 parts, where each part is produced on a
manufacturing process with a Cpk of 0.67, has a 95.5 percent
probability that each part will be defect free. However, when all 25
parts are assembled into the final product, the probability that the
final product will be defect free is only
32 percent. In comparison, if the same parts are produced with
manufacturing processes at a Cpk of 1.33, the probability of each part
being defect free is 99.99 percent. When these same 25 parts are
assembled into the final product, the probability that the final
product will be defect free is
99.8 percent. This comparison illustrates the impact that having
manufacturing processes in control has on the amount of rework and
repair that would be needed to correct defects and make the product
meet its specifications.
Cummins uses statistical process control data to measure a product‘s
readiness for production. In developing the new Signature 600 diesel
engine, Cummins included manufacturing engineers and machine tool and
fixture suppliers in the design decision process as the engine concept
was first being defined. Cummins built production representative
prototypes of its engines to demonstrate that the design and the engine
hardware would perform to requirements. These prototypes represented
the first attempt to build the product solely using manufacturing
personnel, production tooling, and production processes. Cummins used
the knowledge captured from these and subsequent prototypes to refine
and eventually validate the manufacturing processes for the engine.
This process of employing statistical process control techniques on
prototype engines verified that the manufacturing processes were
capable of manufacturing the product to high quality standards within
established cost and schedule targets.
Other companies we visited emphasized the importance of controlling
manufacturing processes before committing to production. For example,
Xerox captures knowledge about the producibility of its product early
in the design phase. By production, it strives to have all critical
manufacturing processes for the product--including key suppliers‘
processes--in control with a Cpk index of at least 1.33. Xerox achieves
this by building production representative prototypes and by requiring
suppliers of key components and subassemblies to produce an adequate
sample of parts to demonstrate the suppliers‘ processes can be
controlled, usually before the parts are incorporated into the
prototypes. General Electric Aircraft Engines has digitally captured,
and made available to design engineers, Cpk data on almost all of its
manufacturing processes and it strives to have critical processes in
control to a point where they will yield no more than
1 defect in 500 million parts, a Cpk of 2.0. Other companies, such as
Caterpillar and Hewlett Packard, told us that getting manufacturing
processes in control prior to production is key to meeting cost,
schedule, and quality targets. Each of the companies visited used this
as an indicator of the product‘s readiness for production and
emphasized the importance of having critical manufacturing processes
under control by the start of production.
Demonstrating Product Reliability Indicates the Product Is Ready for
Production:
A product is reliable when it can perform over a specified period of
time without failure, degradation, or need of repair. Reliability is a
function of the specific elements of a product‘s design. Making design
changes to achieve reliability requirements after production begins is
inefficient and costly. Reliability growth testing provides visibility
over how reliability is improving and uncovers design problems so fixes
can be incorporated before production begins.
In general, reliability growth is the result of an iterative design,
build, test, analyze, and fix process. Prototype hardware is key to
testing for reliability growth. Initial prototypes for a complex
product with major technological advances have inherent deficiencies.
As the prototypes are tested, failures occur and, in fact, are desired
so that the product‘s design can be made more reliable. Reliability
improves over time with design changes or manufacturing process
improvements. The earlier this takes place, the less impact it will
have on the development and production program. Companies we visited
matured a product‘s reliability through these tests and demanded proof
that the product would meet the customer‘s reliability expectations
prior to making a production decision.
Improvements in the reliability of a product‘s design can be measured
by tracking a key reliability metric over time. This metric compares
the product‘s actual reliability to a growth plan and ultimately to the
overall reliability goal. Several commercial companies we visited began
gathering this data very early in development and tracked it throughout
development. The goal was to demonstrate the product would meet
reliability requirements before starting full rate production.
Caterpillar establishes a plan to grow and demonstrate the product‘s
reliability before fabrication of a production representative prototype
begins. Before Caterpillar starts making parts, it estimates the
product‘s reliability in its current stage of development based on
knowledge captured from failure modes and effects analysis,[Footnote 6]
component prototype testing, and past product experience. This
information marks the starting point for the product‘s reliability
growth plan and is the basis for assessing whether the plan is
achievable by production. If Caterpillar believes the risks are too
high and the goal cannot be achieved on time, decision makers assess
trade-offs between new and existing components to reduce the risks to a
more manageable level. Trade-offs might be made if the product‘s
performance still fails to meet requirements. If trade-offs are not
possible, decision makers may decide not to go forward with the
development. Once Caterpillar has established this plan, it tracks
demonstrated reliability against it as a management tool to measure
progress. It sets an interim reliability milestone and expects to be at
least halfway toward the expected goal by the time it begins to build
production units. Caterpillar has learned from experience that it will
achieve the full reliability goal by full production if it meets the
interim goal by the time it produces pilot production units. If the
reliability is not growing as expected, then decisions about changing
or improving the design must be addressed.
Caterpillar improves the product‘s reliability during development by
testing prototypes, uncovering failures, and incorporating design
changes. According to Caterpillar officials, the production decision
will be delayed if they are not on track to meeting their reliability
goal. These officials told us that Caterpillar maintains the philosophy
of first getting the design right, then producing it as quickly and
efficiently as possible. They emphasized that demonstrating reliability
before production minimized the potential for costly design changes
once the product is fielded.
Executive Level Reviews Are Conducted to Begin Production:
The commercial companies, after capturing specific manufacturing
knowledge, had executive level reviews to determine if the product
development had sufficiently progressed to permit a transition into
production. Executives used the knowledge captured as exit criteria for
the transition. For example, to demonstrate the product was ready for
production, critical processes had to be in control and testing should
have demonstrated the product reliability. If the design team could not
satisfy the exit criteria, then other options had to be considered. The
production decision led to increased investments for materials and
resources such as additional tooling to build the product at a planned
rate, facilities, people, training and support.
When DOD Programs More Closely Approximated Best Practices, Outcomes
Were Better:
Our analysis of DOD programs showed that those more closely
approximating best practices had better outcomes. The F/A-18 E/F
fighter and the AIM-9X missile were based extensively on predecessor
programs and employed similar tools to capture design and manufacturing
knowledge at critical program junctures. These programs had
demonstrated a significantly higher degree of design stability prior to
entering system demonstration and committing to initial manufacturing
when compared to other DOD weapon programs in our review. They also
gained control of most of their manufacturing processes and
demonstrated that the products were reliable before entering
production. The success of these programs is best demonstrated by the
fact that they have been close to meeting cost, schedule, and
performance objectives. On the other hand, the PAC-3 missile, F-22
fighter, and ATIRCM/CMWS programs did not use these best practices.
These programs were not based on predecessor products or evolutionary
in nature, and each product‘s full capability was expected in one step,
with the first product off the production line. With this daunting
task, these programs failed to demonstrate a stable design before
committing to initial manufacturing, causing quality and labor
problems. These programs also had much less knowledge about the
manufacturability of their design when they entered production. As a
result, they experienced significant increases in development costs and
production delays usually at the expense of other DOD programs. Details
on the five DOD programs follow.
AIM-9X Missile Program:
The AIM-9X development practices closely paralleled best practices used
by the commercial companies we visited. The program achieved design
stability before moving into system demonstration by incorporating
mature technologies and components from other missiles and munitions,
using engineering prototypes to demonstrate the design, holding a
series of design reviews prior to the system level critical design
review, and completing and releasing 95 percent of the engineering
drawings at that time. Figure 6 shows the building of knowledge
required to achieve a stable design on the AIM-9X.
Figure 6: Achieving Stability on AIM-9X Missile Program by Knowledge
Point 2:
[See PDF for image]
Source: GAO‘s analysis.
[End of figure]
The AIM-9X program made extensive use of engineering prototypes to
stabilize the missile‘s design before building production
representative prototypes. Program officials stated that testing of
engineering prototypes uncovered problems with missile design and
manufacturing tooling early in the development, during system
integration, allowing time to re-design and re-test in follow-on
configurations. According to program officials, this not only helped
stabilize the design before entering initial manufacturing but grew
system reliability and reduced total ownership costs. The program also
held design reviews for each of the major subsystems, allowing the
program to achieve and demonstrate a stable design in July 1999, before
beginning initial manufacturing of production representative
prototypes.
While the AIM-9X used statistical process control only to a limited
extent, other factors have allowed it to have a more successful
production outcome to date. Program officials took steps to ensure that
manufacturing aspects of the product were included in the design,
including empowering a product leader with a manufacturing background,
identifying the key characteristics and critical manufacturing
processes early, making design trade-offs to enhance manufacturing
capability, and demonstrating a robust design to make the product less
vulnerable to variations in manufacturing process. In addition, the
ability to achieve design stability at the critical design review
allowed program officials to focus the system demonstration phase on
maturing the manufacturing processes. Prior to committing to
production, the program demonstrated that the product could be
efficiently built using production processes, people, tools, and
facilities to build prototypes. According to the former program
manager, these steps gave the officials knowledge that a reliable
product could be produced within cost and schedule targets prior to
entering production. To date, the AIM-9X program has largely met its
production targets.
F/A-18 E/F Program:
The F/A-18 E/F aircraft development program was able to take advantage
of knowledge captured in developing and manufacturing prior versions of
the aircraft. This evolutionary approach significantly contributed to
the cost and schedule successes of this program. Because the F/A-18 E/
F was a variant of the older F/A-18 aircraft, the developer had prior
knowledge of design and manufacturing problems. This knowledge, coupled
with the use of modeling and computer-aided design software, helped
create a design that was easier to manufacture. While the program did
not fully use each of the best practices, it did embrace the concepts
of capturing design and manufacturing knowledge early in the program.
During the program‘s critical design review, about 56 percent of the
drawings were completed and, while the program did not meet the best
practice of 90 percent complete, it did have additional drawing data of
the F/A-18 E/F assemblies available for review at the critical design
review. The Navy used early versions of the F/A-18 aircraft to
demonstrate new component designs and new materials. In addition, the
aircraft was designed to have 42 percent fewer parts than its
predecessor, making its design more robust. The program also identified
the critical manufacturing processes and collected statistical process
control data early in product development. At the start of production,
78 percent of these critical processes were in control. Unit costs for
the F/A-18 E/F program have not grown since the critical design review
and its schedule has been delayed by only 3 months.
F-22 Fighter Program:
The F-22 program is structured to provide the product‘s full capability
with the first product off the production line--an extreme design
challenge. This required the product design to include many new and
unproven technologies, designs, and manufacturing processes. It did not
demonstrate design stability until about 3 years after it held its
critical design review. The program completed 3,070 initial engineering
drawings at its critical design review in 1995, about 26 percent of the
eventual drawings needed. It did not complete 90 percent of the
necessary engineering drawings until 1998, after the first two
development aircraft were delivered. Figure 7 shows the drawing
completion history for the program.
Figure 7: History of Drawing Completion for the F-22 Program:
[See PDF for image]
Source: GAO‘s analysis.
[End of figure]
After its critical design review, the F-22 program encountered several
design and manufacturing problems that resulted in design changes,
labor inefficiencies, cost increases, and schedule delays. For example,
delivery of the aft fuselage--the rear aircraft body section--was late
for several of the test aircraft and two ground test articles because
of late parts and difficulties with the welding process. According to
the F-22 program office, design maturity and manufacturing problems
caused a ’rolling wave“ effect throughout system integration and final
assembly. Late engineering drawing releases to the factory floor
resulted in parts shortages and work performed out of sequence. These
events contributed to significant cost overruns and delays to aircraft
deliveries to the flight test program.
The F-22 program initially had taken steps to use statistical process
control data during development and gain control of critical
manufacturing processes by the full rate production decision. In
1998,[Footnote 7] we reported that the program had identified 926
critical manufacturing processes and had almost 40 percent in control 2
years before production was scheduled to begin. Although this did not
match the standard set by commercial companies, it offered major
improvements over what other DOD programs had attempted or achieved.
Unfortunately, citing budgetary constraints and specific hardware
quality problems that demanded attention, the program abandoned this
best practices approach in 2000 with less than 50 percent of it
critical manufacturing processes in control. Currently, the program is
using post-assembly inspection to identify and fix defects rather than
statistical process control techniques to prevent them. In March
2002,[Footnote 8] we recommended that the F-22 program office monitor
the status of critical manufacturing processes as the program proceeds
toward high rate production. The program stated that it would assess
the processes status as the program moves forward.
The program entered limited production despite being substantially
behind its plan to achieve reliability goals. A key reliability
requirement for the
F-22 is mean time between maintenance, defined as the number of
operating hours for the aircraft divided by the number of maintenance
actions. The reliability goal for the F-22 is a 3-hour mean time
between maintenance. The Air Force estimated that in late 2001, when
the F-22 entered limited production, it should have been able to
demonstrate almost 2 flying hours between maintenance actions. However,
when it actually began limited production it could only fly an average
of 0.44 hours between maintenance actions. In other words, the F-22 is
requiring significantly more maintenance actions than planned.
Additionally, the program has been slow to fix and correct problems
that have affected reliability. To date, the program has identified
about 260 different types of failures, such as main landing gear tires
wearing out more quickly than planned, fasteners being damaged, and
canopy delaminating. It has identified fixes for less than 50 percent
of these failures. Ideally, the design fixes for the failures should be
corrected prior to manufacturing production units.
PAC-3 Missile Program:
The PAC-3 missile did not achieve design stability until after the
building of production representative prototypes for system
demonstration began. At the program‘s critical design review, the PAC-
3 program had completed 980 engineering drawings--21 percent of the
eventual drawings needed for the missile. Since then, almost 3,700 more
drawings have been completed. The total number of drawings expected to
represent the completed design grew from about 2,900 at the critical
design review to almost 4,700 as of July 2001. This uncertainty in the
expected drawings not only indicates that the design was not stable
when initial manufacturing began but also shows that there was a
significant lack of knowledge about the design. Figure 8 shows the
design knowledge at the critical design review, when the decision was
made to commit to initial manufacturing of the missile.
Figure 8: PAC-3 Design Knowledge at Critical Design Review:
[See PDF for image]
Source: GAO‘s analysis.
[End of figure]
Prototypes of the product design were not built before the critical
design review or before initial manufacturing started to show that the
design would work. Therefore, because of the immature design, initially
manufactured development missiles were hand-made, took longer to build
than planned, and suffered from poor quality. As a result, many design
and manufacturing problems surfaced during system demonstration.
Subsystems did not fit together properly, and many failed ground and
environmental tests the first time. The contractor attributed $100
million of additional cost to first time manufacturing problems.
Prior to entering limited production in 1999, the program had less than
40 percent of the critical manufacturing processes in control for
assembling the missile and the seeker. According to program officials,
there was little emphasis during development or initial production on
using statistical control on critical manufacturing processes. Most of
the development missiles were built in specialty shops rather than in a
manufacturing environment. The result was a lack of knowledge about
whether the critical manufacturing processes could produce the product
to established cost, schedule, and quality targets. This uncertainty is
reflected in contractor estimates that more than 50 percent of the time
charged to build the initial production missiles will be for
engineering activities. Actual production labor is expected to account
for about 30 percent of the charged time.
To further understand the problems on the PAC-3 program, we focused on
its seeker subsystem, which is key to acquiring and tracking targets
and represents a large percentage of the missile‘s cost. Currently,
despite being in production, it is unclear whether the supplier of the
seeker can produce it within cost, schedule, and quality targets.
During development, the supplier had difficulty in designing and
manufacturing this subsystem. It was not uncommon for seekers to be
built, tested, and reworked seven or eight times before they were
acceptable. The program entered production, despite these producibility
issues. Now, even with 2 years of production experience, the supplier
continues to have difficulty producing the seeker with acceptable
quality. Data provided by the supplier in October 2001 showed that less
than 25 percent of the seekers were being manufactured properly the
first time and the rest had to be reworked, on average, four times.
ATIRCM/CMWS Program:
According to program officials, ATIRCM/CMWS did not have a stable
design until about 2 years after the critical design review. A
contributing factor to this was a lack of understanding about the full
requirements for the new system at the critical design review in 1997.
This led to a major redesign of the common missile warning system‘s
sensor. At the critical design review, only 21 percent of a product‘s
engineering drawings had been completed. It did not complete 90 percent
drawings--the best practice--until 1999. The immature design caused
inefficiencies in manufacturing, rework, and delayed deliveries. In
addition, between 1995 and 1999, the development contract target price
increased by 165 percent.
The ATIRCM/CMWS program did not begin reliability growth testing until
4 years after its critical design review, leaving only 1 year to test
the system prior to scheduled production. Program officials said that
an immature design limited their ability to begin reliability testing
earlier in development. About one-third of the way through the
reliability growth test program, testing was halted because too many
failures occurred in components such as the power supply, the high
voltage electrical system, and the cooling system. According to a
program official, the inability to demonstrate system reliability
contributed to a production delay of about
1 year. The program plans to build, develop, and test six additional
development units during 2002 and 2003 that will incorporate design
changes to fix the system failures. ATIRCM/CMWS plans to enter limited
production in the early part of 2002 with significantly less knowledge
about the design‘s producibility than commercial companies. The
contractor does not use statistical process control and has not
identified critical manufacturing processes. A production readiness
review identified the lack of statistical process control as a major
weakness that needs to be corrected.
[End of section]
Chapter 4 A Better Match of Policy and Incentives Is Needed to Ensure
Capture of Design and Manufacturing Knowledge:
The Department of Defense‘s (DOD) acquisition policy[Footnote 9]
establishes a good framework for developing weapon systems; however,
disciplined adherence, more specific criteria, and stronger acquisition
incentives are needed to ensure the timely capture and use of knowledge
in decision making. DOD changed its acquisition policy to emphasize
evolutionary acquisition and establish separate integration and
demonstration phases in the product development process. Its goal was
to develop higher quality systems in less time and for less cost.
However, DOD‘s acquisition policy lacks detailed criteria for capturing
and using design and manufacturing knowledge to facilitate better
decisions and more successful acquisition program outcomes. As
demonstrated by successful companies, using these criteria can help
ensure that the right knowledge is collected at the right time and that
it will provide the basis for key decisions to commit to significant
increases in investment as product development moves forward.
While the right policy and criteria are necessary to ensure a
disciplined, knowledge-based product development process, the
incentives that influence the key players in the acquisition process
will ultimately determine whether they will be used effectively. In
DOD, current incentives are geared toward delaying knowledge so as not
to jeopardize program funding. These incentives undermine a knowledge-
based process for making product development decisions. Instead,
program managers and contractors push the capture of design and
manufacturing knowledge to later in the development program to avoid
the identification of problems that might stop or limit its funding.
They focus more on meeting schedules than capturing and having the
knowledge necessary to make the right decisions at those milestones.
Such an approach invariably leads to added costs because programs are
forced to fix problems late in development.
By contrast, commercial companies must develop high-quality products
quickly or they may not survive in the marketplace. Because of this,
they encourage their managers to capture product design and
manufacturing knowledge to identify and resolve problems early in
development, before making significant increases in their investment.
Instead of a schedule-driven process, their process is driven by events
that bring them knowledge: critical design reviews that are supported
by completed engineering drawings and production decisions that are
supported by reliability testing and statistical process control data.
They do not move forward without the design and manufacturing knowledge
needed to make informed decisions.
Acquisition Policy Lacks Specific Implementation Criteria:
Greater emphasis on evolutionary acquisitions and structuring the
product development process into two phases--system integration and
system demonstration--were good first steps for DOD to achieve its
goals of buying higher quality systems in less time and for lower
costs. However, DOD policy still lacks criteria to be used to capture
specific design and manufacturing knowledge and does not require the
use of that knowledge as exit criteria at key decision points to
transition from system integration to system demonstration and then
into production. In three of the five DOD program examples in chapter
3, managers decided to move forward in development, even when
developers had failed to capture design and manufacturing knowledge to
support increased investments. As a result, these programs encountered
significant increases in acquisition costs as well as delays in
delivering capabilities to the war fighter.
Table 7 illustrates key criteria used by commercial companies that are
currently lacking in DOD‘s policy. The table shows the design and
manufacturing knowledge needed to make more informed decisions. The
capture of some of the important manufacturing and reliability
knowledge should begin in the integration phase in order to have the
full knowledge needed to make decisions at the end of the demonstration
phase for transitioning into production.
Table 7: Analysis of DOD Acquisition Policy for Inclusion of Best
Practices for Knowledge-based Design and Manufacturing Decisions:
Commercial criteria: X; Best practices to capture design knowledge by
decision point to enter system demonstration phase: Use of key
indicator to show design stability (90 percent of drawings completed);
DOD criteria: [Empty].
Commercial criteria: X; Best practices to capture design knowledge by
decision point to enter system demonstration phase: Limit design
challenge prior to entering system integration; DOD criteria: X.
Commercial criteria: X; Best practices to capture design knowledge by
decision point to enter system demonstration phase: Demonstrate the
design meets requirements; DOD criteria: X.
Commercial criteria: X; Best practices to capture design knowledge by
decision point to enter system demonstration phase: Complete critical
design reviews; DOD criteria: [Empty].
Commercial criteria: X; Best practices to capture design knowledge by
decision point to enter system demonstration phase: Obtain stakeholder
agreements that drawings complete and producible; DOD criteria:
[Empty].
Commercial criteria: X; Best practices to capture design knowledge by
decision point to enter system demonstration phase: Hold decision
review to begin initial manufacturing; DOD criteria: [Empty].
Best practices to capture design knowledge by decision point to enter
system demonstration phase: Commercial criteria : Best practices to
capture product knowledge by decision point to enter production; DOD
criteria: Commercial criteria : [Empty].
Commercial criteria: X; Best practices to capture design knowledge by
decision point to enter system demonstration phase: Use of key
indicators to show the product is ready for production (processes in
statistical control and product reliability demonstrated); DOD
criteria: [Empty].
Commercial criteria: X; Best practices to capture design knowledge by
decision point to enter system demonstration phase: Identify key system
characteristics and manufacturing processes; DOD criteria: [Empty].
Commercial criteria: X; Best practices to capture design knowledge by
decision point to enter system demonstration phase: Determine critical
processes are in control and capable; DOD criteria: [Empty].
Commercial criteria: X; Best practices to capture design knowledge by
decision point to enter system demonstration phase: Conduct failure
modes and effects analysis; DOD criteria: [Empty].
Commercial criteria: X; Best practices to capture design knowledge by
decision point to enter system demonstration phase: Set reliability
growth goals; DOD criteria: [Empty].
Commercial criteria: X; Best practices to capture design knowledge by
decision point to enter system demonstration phase: Conduct reliability
growth testing; DOD criteria: [Empty].
Commercial criteria: X; Best practices to capture design knowledge by
decision point to enter system demonstration phase: Hold decision
review to begin production; DOD criteria: X.
[End of table]
According to DOD‘s current acquisition policy, the system integration
phase of an acquisition normally begins with the decision to launch a
program. The policy states that, during this phase, a system‘s
configuration should be documented and the system should be
demonstrated using prototypes in a relevant environment. While these
are noteworthy activities and resemble best practices, the policy does
not provide criteria for what constitutes the level of knowledge
required for completing this stage, nor does it require a decision--
based on those criteria--as to whether a significant, additional
investment should be made. Commercial companies demand knowledge from
virtual or engineering prototypes, 90 percent of required engineering
drawings for the product supported by test results, demonstration that
the product meets customer requirements, a series of disciplined design
reviews, and stakeholder agreement that the design is stable and ready
for product demonstration before a commitment is made to move forward
and invest in product demonstration. Under DOD‘s revised policy, it is
still difficult to determine if a product should enter product
demonstration with a stable design.
DOD‘s current acquisition policy also states that the system
demonstration phase begins after prototypes have been built and
demonstrated in a relevant environment during system integration.
According to the policy, a system must be demonstrated before the
department will commit to production. The low-rate initial production
decision occurs after this phase of product development. Like the end
of system integration, the policy fails to provide specific criteria
for what constitutes the knowledge required to support the decision to
move into production. For example, the policy states there should be
’no significant manufacturing risks“ but does not define what this
means or how it is measured. Without criteria for building knowledge
during the demonstration phase, the production decision is often based
on insufficient knowledge, creating a higher probability of
inconsistent results and cost and schedule problems. On the other hand,
commercial companies demand proof that manufacturing processes are in
control and product reliability goals are attained before committing to
production. With more specific knowledge in hand at the end of
development, decision makers can make a more informed decision to move
into production with assurances that the product will achieve its cost,
schedule, and quality outcomes.
Finally, while DOD‘s policy separates product development into a two-
stage process--integration and demonstration--it does not require a
decision milestone to move from one stage to the next. The policy
states that an interim progress review should be held between the two
stages, but the review has no established agenda and no required
outputs of information unless specifically requested by the decision
maker. Its purpose is to confirm that the program is progressing as
planned. On the other hand, commercial companies consider this review a
critical decision point in their product development process because it
precedes a commitment to significantly increase their investment.
Therefore, they use specific, knowledge-based standards and criteria to
determine if the product is ready to enter the next phase and they hold
decision makers accountable for their actions. These decision reviews
are mandatory and are typically held at the executive level of the
commercial firm.
Figure 9 illustrates the commercial model for knowledge to be captured
and delivered during product integration and product demonstration and
the possible application of that model to DOD‘s acquisition process.
Without a similar decision review to bring accountability to the DOD
process, acquisition programs can--and do--continue to advance into
system demonstration without a stable design. As shown in our case
studies, this provides for a high probability of cost growth and
schedule delays to occur.
Figure 9: Illustration to Show How the Best Practice Model Would Apply
to DOD‘s Acquisition Process:
[See PDF for image]
Source: GAO‘s analysis.
[End of figure]
Incentives in the DOD Acquisition Environment Do Not Favor Capture of
Design and Manufacturing Knowledge Early Enough:
The incentives for program managers and product developers to gather
knowledge and reduce risk are also critical to DOD‘s ability to adopt
best practices for product development. In DOD, incentives are centered
on obtaining scarce funding on an annual basis in a competitive
environment to meet predetermined and typically optimistic program
schedules. These incentives actually work against the timely capture of
knowledge, pushing it off until late in the process to avoid problems
that might keep a program from being funded. Because design and
manufacturing knowledge is not captured, key decision points intended
to measure and ensure that a weapon system has sufficiently matured to
move forward in the process risk becoming unsupported by critical
knowledge. In leading commercial companies, the opposite is true.
Because companies know they have to deliver high-quality products
quickly and affordably, they limit the challenge for their program
managers and provide strong incentives to capture design and
manufacturing knowledge early in the process. Program managers are
empowered to make informed decisions before big investments in
manufacturing capability are required.
DOD‘s current acquisition environment is driven by incentives to make
decisions while significant unknowns about the system‘s design and
manufacturability persist. This environment results in higher risks and
a greater reliance on cost-reimbursement[Footnote 10] contracts for
longer periods of time during product development. Because events that
should drive key decisions, such as critical design reviews, interim
progress reviews, and production decision reviews, are based on
inadequate design and manufacturing knowledge, they do not support
decisions to invest more and move to the next phase of the acquisition
process. Nevertheless, this approach has proven effective in securing
funds year to year. For example, the F-22, PAC-3, and ATIRCMS/CMWS
programs had less than one-third of their engineering drawings
completed at their critical design review, but each obtained the
funding necessary to move onto the initial manufacturing of production
representative prototypes. That funding allowed a significant increase
in investment to develop a manufacturing capability before critical
knowledge had been captured. The incentive to capture funding for the
program was greater than the incentive to wait, capture knowledge, and
reduce the risk of moving forward. Each of these programs encountered
significant cost increases and schedule delays.
The incentives are quite different for leading commercial companies.
For them, the business case centers on the ability to produce a product
that the customer will buy and that will provide an acceptable return
on investment. If the firm has not made a sound business case, or has
been unable to deliver on one or more of the business case factors, it
faces a very real prospect of failure--the customer may walk away.
Also, if one product development takes more time and money to complete
than expected, it denies the firm opportunities to invest those
resources in other products. For these reasons, commercial companies
have strong incentives to capture product knowledge early in the
process to assess the chances of making the business case and the need
for further investments.
Production is a dominant concern in commercial companies throughout the
product development process and forces discipline and trade-offs in the
design process. This environment encourages realistic assessments of
risks and costs since doing otherwise would threaten the business case
and invite failure. For the same reasons, the environment places a high
value on knowledge for making decisions. Program managers have good
reasons to identify risks early, be intolerant of unknowns, and not
rely on testing late in the process as the main vehicle for discovering
the performance characteristics of the product. By adhering to the
business case as the key to success, program managers in leading
commercial companies are conservative in their estimates and aggressive
in risk reduction. Ultimately, adherence to the business case
strengthens the ability to say ’no“ to pressures to accept high risks
and unknowns. Practices such as prototyping, early manufacturing and
supplier involvement, completing 90 percent of engineering drawings by
critical design review, demonstrating product reliability, and
achieving statistical control of critical manufacturing processes by
production are adopted because they help ensure success.
In DOD‘s current acquisition environment, the customer is willing to
trade time and money for the highest performing weapon system possible.
That willingness drives the business case. This creates strong
incentives for the program office to take significant risks with
technologies and designs to ensure it can offer the customer a weapon
system that is a quantum leap above the competition. In addition,
because funding is secured on an annual basis in DOD, strong incentives
exist for the program office to make optimistic assumptions about
development cost and schedule. Because the customer is willing to wait
and funding is never certain, an environment exists where program
managers have good reasons to avoid the capture of knowledge and delay
testing. Since the business case in DOD places very little premium on
meeting cost and schedule targets, but a very high premium on
performance, programs succeed at the point where sunk costs make it
difficult--if not prohibitive--for decision makers to cancel them.
The practices commercial companies use to capture knowledge are not
currently used in this environment because the business case does not
favor them. Instead, DOD‘s product development environment relies on
cost-type contracting throughout the entire product development
process. Once in production, programs will cut quantities to maintain
funding or once fielded, they rely on the operations and maintenance
budget to pay for reliability problems not solved in development.
[End of section]
Chapter 5 Conclusions and Recommendations:
Conclusions:
The Department of Defense‘s (DOD) planned $700 billion investment in
weapon systems over the next 5 years requires an approach that keeps
cost, schedule, and performance risks to a minimum. This approach means
adopting and implementing an evolutionary approach to developing new
weapon systems, improving policy to more closely approximate a
knowledge-based product development process, and creating incentives
for capturing and using knowledge for decision making. Without an
evolutionary approach as its foundation, the ability to capture design
and manufacturing knowledge early in the development process is
significantly reduced. Programs, in turn, take on too much new unproven
content to meet their objectives and risks invariably increase. DOD has
made improvements in its acquisition policy by incorporating guidance
for evolutionary acquisition, creating guidelines for the development
of a basic product that can be upgraded with additional capabilities as
technologies present themselves. However, evolutionary acquisition has
yet to be consistently implemented with success on individual weapon
system acquisitions.
Regardless of whether DOD emphasized greater use of evolutionary
acquisition, acquisition programs are not capturing sufficient design
and manufacturing knowledge to make good decisions at key investment
points. The current policy establishes a good framework to develop a
product, but the policy still lacks specific criteria required to move
a program forward and does not tie knowledge to decisions for
increasing investments in the program as it moves from system
integration to system demonstration. As a result, programs often pass
through each development phase and into production with an unstable
design and insufficient knowledge about critical manufacturing
processes and product reliability. This results in greater likelihood
for inconsistent and poor results and cost and schedule problems later
in the program.
Additionally, DOD does not provide the proper incentives to encourage
the use of best practices in capturing knowledge early in its
development programs. Currently, managers are focused more on the
annual exercise of obtaining funding needed to keep their programs
viable and alive. The importance of capturing design and manufacturing
knowledge early gives way to the pressures of maintaining funding,
often resulting in the acceptance of greater risks. Raising problems on
a program early because design and manufacturing knowledge is
discovered can cause extra oversight and questions that threaten a
system‘s survival. The prevailing culture is to accept greater risks
upfront and then fix problems later in the development program.
We found that leading commercial companies over the years had found
ways to overcome these problems and had identified best practices that
resulted in the early capture of and use of design and manufacturing
knowledge. This was done by a combination of four key elements. First,
they established and used an evolutionary approach to develop products
that made the capture of design and manufacturing knowledge a more
manageable task. This framework limited the design challenge for any
one new product development by allowing risky technology, design, or
manufacturing requirements to be deferred until a future generation of
the product. DOD‘s current policy addresses this; however, it has not
had sufficient time to show how this will be implemented.
Second, each company we visited used the same basic product development
process and criteria for bringing together and integrating all of the
technologies, components, and subsystems required for the product to
ensure the design was stable and then demonstrating that the product
was producible and reliable using proven manufacturing processes. DOD‘s
policy lacks the criteria to measure design stability and process
controls. Third, successful companies used tools to capture design and
manufacturing knowledge about the product and decide about whether to
invest further based on that knowledge. Their new product development
process included key, high-level decision points before moving into
product demonstration, and again before making the production decision
that required specific, knowledge-based exit criteria. DOD‘s policy
does not require a decision to move from system integration to system
demonstration. Finally, leading companies created an environment for
their managers that emphasized capturing design and manufacturing
knowledge early, before committing substantial investments in a product
development that made cancellation a more difficult decision to make.
DOD‘s environment encourages meeting schedule milestones instead of
capturing design and manufacturing knowledge to make decisions.
Recommendations for Executive Action:
DOD should take steps to close the gaps between its current acquisition
environment and best practices. To do this, it should ensure that its
acquisition process captures specific design and manufacturing
knowledge, includes decisions at key junctures in the development
program, and provides incentives to use a knowledge-based process. Such
changes are necessary to obtain greater predictability in weapon system
programs‘ cost and schedule, to improve the quality of weapon systems
once fielded, and to deliver new capability to the war fighter faster.
More specifically, we recommend that the Secretary of Defense:
* Require the capture of specific knowledge to be used as exit criteria
for decision making at two key points--when transitioning from system
integration to system demonstration and from system demonstration into
production. The knowledge to be captured when moving from system
integration into system demonstration should include the following:
* Completed subsystem and system design reviews.
* Ninety percent of drawings completed.
* Demonstration that design meets requirements--prototype or variant
testing.
* Stakeholders‘ (cross functional design team that includes design
engineers, manufacturing, key supplier) assurance that drawings are
complete.
* Completed failure modes and effects analysis.
* Identification of key system characteristics.
* Identification of critical manufacturing processes.
* Set reliability targets and growth plan.
The knowledge to be captured when moving from system demonstration into
production should include the following:
* Demonstrated manufacturing processes.
* Built production representative prototypes.
* Tested prototypes to achieve reliability goal.
* Tested prototypes to demonstrate product in operational environment.
* Collected statistical process control data.
* Demonstration that critical processes are capable and in control.
* Require that the interim progress review, currently identified in
DOD‘s policy as that point in the process between system integration
and system demonstration, be a mandatory decision review. At this
point, the design should be demonstrated to be stable so that during
the next phase of development attention can be focused on demonstrating
manufacturing processes and product reliability. The program manager
should have proof--based on the exit criteria for moving out of system
integration in the above recommendation--that the product design is
stable. The exit criteria should be demonstrated and verified by the
program manager before the program can make the substantial investments
needed to begin manufacturing production representative prototypes in
the next phase of development--system demonstration. To ensure
visibility of demonstrated exit criteria to decision makers, the
criteria and the program‘s status in achieving them should be included
in each program‘s Defense Acquisition Executive Summary and Selected
Acquisition Reports. If the program does not meet the exit criteria,
investments should be delayed until such time as the criteria are
satisfied. To proceed without completing the required demonstrations
should require approval by the decision authority.
* Expand exit criteria for the Milestone C decision to include the
knowledge to be captured during the system demonstration phase as
identified in recommendation one. This will require that the program
office demonstrate that the critical manufacturing processes are under
statistical control and that product reliability has been demonstrated
before entering production of the new weapon system. These are best
practices and indicate that the product design is mature and the
program is ready to begin production of units for operational use that
will meet the cost, schedule, and quality goals of the program.
* To ensure that contracts support a knowledge-based process, we
further recommend that DOD structure its contracts for major weapon
system acquisitions so that (a) the capture and use of knowledge
described in recommendation one for beginning system demonstration is a
basis for DOD‘s decision to invest in the manufacturing capability to
build initial prototypes and (b) the capture and use of manufacturing
and reliability knowledge discussed in recommendation one for moving
from system demonstration to production is a basis for DOD‘s decision
to invest in production.
Agency Comments and Our Evaluation:
DOD concurred with a draft of this report and agreed with the benefits
of using design and manufacturing knowledge to make informed decisions
at key points in a system acquisition program. DOD had some comments
with regard to the details contained in the recommendations, which are
summarized below. DOD concurred with our recommendation to add exit
criteria at two key points in the acquisition process--when
transitioning from system integration to system demonstration and from
system demonstration into production. DOD believes, however, that the
milestone decision authority needs to retain flexibility in applying
the knowledge requirement for drawings. Flexibility and judgment are
management prerogatives that should exist in any decision process. We
agree there may be circumstances, such as in the development of
software, when it makes good sense to progress with less than the best
practice standard for drawings, but the DOD policy should maintain the
requirement to achieve 90 percent drawings by the completion of the
system integration phase.
DOD also concurred that critical manufacturing processes must be
demonstrated using statistical process control techniques before
production, but believes that achieving this at Milestone C, the low
rate production decision, is unlikely. It believes the criteria would
be better applied to the full rate production decision or when low rate
production quantities extend beyond 10 percent of the planned weapon
system buy. This is a reasonable approach when processes are new or
unique. However, not all critical processes will be new or unique to a
specific weapon system. Some will have been used to manufacture parts
or components for other systems or products. At a minimum, it should be
possible to demonstrate these by Milestone C. For other critical
processes that may require additional production experience to bring
under statistical process control, a program manager should have a
reasonable plan at the Milestone C decision review to bring those
processes into control by the full rate production decision, but no
later than completion of 10 percent of the planned buy.
[End of section]
Appendixes:
Appendix I: Comments from the Department of Defense:
ACQUISITION, TECHNOLOGY AND LOGISTICS:
OFFICE OF THE UNDER SECRETARY OF DEFENSE:
3000 DEFENSE PENTAGON WASHINGTON, DC 20301-3000:
19 JUN 2002:
Ms. Katherine V. Schinasi:
Director, Acquisition and Sourcing Management U.S. General Accounting
Office:
441 G Street, N.W. Washington, D.C. 20548:
Dear Ms. Schinasi:
This is the Department of Defense (DOD) response to the GAO draft
report, ’BEST PRACTICES: Capturing Design and Manufacturing Knowledge
Early Improves Acquisition Outcomes,“ dated May 15, 2002 (GAO Code
120054/GAO-02-701).
The Department concurs with the objectives of GAO‘s DRAFT report and we
agree with the benefits of using important information about a system‘s
design and critical manufacturing processes to make informed decisions.
However, we have some comments with regard to the details contained in
the recommendations.
In response to previous GAO reports, DoD has structured its acquisition
process to make progress through the acquisition life-cycle dependent
on the knowledge available at key decision points. This report adds
robustness to our already disciplined process.
We look forward to discussing these issues with the GAO. Detailed
comments are provided in the enclosure.
My action officer for this effort is Mr. Richard Sylvester, (703) 697-
6399.
Sincerely,
Donna S. Richbourg:
Director, Acquisition Initiatives:
Signed by Donna S. Richbourg:
Enclosure: As stated:
GAO DRAFT REPORT - DATED MAY 15, 2002 GAO CODE 120054/GAO-02-701:
’BEST PRACTICES: Capturing Design and Manufacturing Knowledge Early
Improves Acquisition Outcomes“:
DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATIONS:
RECOMMENDATION 1: The GAO recommended that the Secretary of Defense
require the capture of specific knowledge to be used as exit criteria
for decision making at two key points - when transitioning from system
integration to system demonstration and from system demonstration into
production. The knowledge to be captured when moving from system
integration into system demonstration should include the following:
*Completed subsystem and system design reviews:
*90% of drawings completed:
*Demonstration that design meets requirements - prototype or variant
testing:
*Stakeholders‘ (cross functional design team that includes design
engineers, manufacturing, key supplier) assurance that drawings are
complete *Completed failure modes and effects analysis:
*Identification of key system characteristics:
*Identification of critical manufacturing processes:
*Set reliability targets and growth plan:
The knowledge to be captured when moving from system demonstration into
production should include the following:
*Demonstrated manufacturing processes:
*Built production representative prototypes:
*Tested prototypes to achieve reliability goal:
*Tested prototypes to demonstrate product in operational environment:
*Collected statistical process control data:
*Demonstration that critical processes are capable and in control (pgs.
69-70/GAO Draft Report):
DOD RESPONSE: Concur. DoD agrees with the benefits of identifying
specific design and manufacturing information to support decision
making at key decision points. However, we believe that the milestone
decision authority needs to retain the flexibility to determine
application of the knowledge requirement for drawings (e.g., software
systems will not have drawings, 85% of completed drawings may be enough
when measured against urgency of requirement, etc.). However, in no
case should a system proceed without a substantial percentage of the
drawings completed. We believe the specific measures should be set by
the MDA at MS B and that the MDA should have the option to add specific
criteria to reflect specific systems.
RECOMMENDATION 2: The GAO recommended that the Secretary of Defense
require that the interim progress review, currently identified in DoD
policy as that point in the process between system integration and
system demonstration, be a mandatory decision review. At this point,
the design should be demonstrated to be stable so that during the next
phase of development attention can be focused on demonstrating
manufacturing processes and product reliability. The program manager
should have proof - based on the exit criteria for moving out of system
integration in the above recommendation - that the product design is
stable. The exit criteria should be demonstrated and verified by the
program manager before the program can make the substantial investments
needed to begin manufacturing production representative prototypes in
the next phase of development - system demonstration. To ensure
visibility of demonstrated exit criteria to decision makers, the
criteria and the program‘s status in achieving it should be included in
each program‘s Defense Acquisition Executive Summary and Selected
Acquisition Reports. If the program does not meet the exit criteria,
investments should be delayed until such time as the criteria is
satisfied. To proceed without completing the required demonstrations
should require approval by the decision authority. (p. 70/GAO Draft
Report):
DOD RESPONSE: Concur. DOD agrees with the requirement to demonstrate
design stability at a point between system integration and system
demonstration and to satisfy criteria reflecting that stability
established by the MDA at MS B. The PM should be able to demonstrate
exit criteria satisfaction by providing a report to the Milestone
Decision Authority. That data can then be considered in the context of
other key indications of program progress. A mandatory review would
only be considered if the exit criteria are not satisfied.
RECOMMENDATION 3: The GAO recommended that the Secretary of Defense
expand exit criteria for the Milestone C decision to include the
knowledge to be captured during the system demonstration phase as
identified in recommendation 1. This will require that the program
office demonstrate that the critical manufacturing processes are under
statistical control and that product reliability has been demonstrated
before entering production of the new weapon system. These are best
practices, and indicate that the product design is mature and the
program is ready to begin production of units for operational use that
will meet the cost, schedule and quality goals of the program. (p. 70/
GAO Draft Report):
DOD RESPONSE: Concur. DoD agrees that demonstration of critical
manufacturing processes must occur prior to rate production. However,
it is unlikely that the criteria suggested by the GAO could be
satisfied at MS C. MS C authorizes the program to proceed to Low Rate
Initial Production in support of Operational Test. That decision
precedes the Full Rate Production Decision where approval is given for
production in support of operational use. The criteria suggested by the
GAO would be better applied to the Full Rate Production Decision or
when LRIP extends beyond 10 percent of the total planned buy..
RECOMMENDATION 4: To ensure that contracts support a knowledge-based
process, the GAO recommended that DoD structure its contracts for major
weapon system acquisitions so that (a) the capture and use o^ knowledge
described in Recommendation 1 for beginning system demonstration is a
basis for DoD‘s decision to invest in the manufacturing capability to
build initial prototypes, and (b) the capture and use of manufacturing
and
reliability knowledge discussed in Recommendation 1 for moving from
system demonstration into production is a basis for DoD‘s decision to
invest in production. (pgs. 70-71/GAO Draft Report):
DOD RESPONSE: Concur. See additional comments in response to
recommendations 1 and 3. DoD agrees with the benefits of applying both
design and manufacturing criteria at key points in the acquisition
process and with capturing those criteria in program contracts. The
specific criteria would be most effective if tied to established breaks
(such as contract line items) designed into the contract structure.
[End of section]
Appendix II: GAO Staff Acknowledgments:
Acknowledgments:
Cheryl Andrew, Cristina Chaplain, Michael Hazard, Matthew Lea, Gary
Middleton, Michael Sullivan, Katrina Taylor, and Adam Vodraska:
[End of section]
Related GAO Products:
Defense Acquisitions: DOD Faces Challenges in Implementing Best
Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.
Best Practices: Better Matching of Needs and Resources Will Lead to
Better Weapon System Space Outcomes. GAO-01-288. Washington, D.C.:
March 8, 2001.
Best Practices: A More Constructive Test Approach Is Key to Better
Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31,
2000.
Defense Acquisition: Employing Best Practices Can Shape Better Weapon
System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000.
Best Practices: DOD Training Can Do More to Help Weapon System Programs
Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August
16, 1999.
Best Practices: Better Management of Technology Development Can Improve
Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30,
1999.
Defense Acquisition: Best Commercial Practices Can Improve Program
Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.
Defense Acquisition: Improved Program Outcomes Are Possible.
GAO/T-NSIAD-98-123. Washington, D.C.: March 18, 1998.
Best Practices: DOD Can Help Suppliers Contribute More to Weapon System
Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998.
Best Practices: Successful Application to Weapon Acquisitions Requires
Changes in DOD‘s Environment. GAO/NSIAD-98-56. Washington, D.C.:
February 24, 1998.
Major Acquisitions: Significant Changes Underway in DOD‘s Earned Value
Management Process. GAO/NSIAD-97-108. Washington, D.C.: May 5, 1997.
Best Practices: Commercial Quality Assurance Practices Offer
Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26,
1996.
FOOTNOTES
[1] U.S. General Accounting Office, Best Practices: Better Matching of
Needs and Resources Will Lead to Better Weapon System Outcomes,
GAO-01-288 (Washington, D.C.: Mar. 8, 2001) and Best Practices: Better
Management of Technology Development Can Improve Weapon System
Outcomes, GAO/NSIAD-99-162 (Washington, D.C.: July 30, 1999).
[2] DOD Directive 5000.1, The Defense Acquisition System (Oct. 23,
2000), DOD
Instruction 5000.2, Operation of the Defense Acquisition System (Apr.
5, 2002), and DOD Regulation 5000.2-R, Mandatory Procedures for Major
Defense Acquisition Programs (MDAPS) and Major Automated Information
System (MAIS) Acquisition Programs (Apr. 5, 2002).
[3] The Selected Acquisition Report provides standard, comprehensive
summary reporting of cost, schedule, and performance information for
major defense acquisition programs to the Congress.
[4] Engineering drawings can include the standard two-dimensional
drawings or newer three-dimensional drawings that are the product of
computer-aided design software systems.
[5] The fifth company wanted its critical manufacturing processes at a
minimum of 1 Cpk.
[6] Failure modes and effects analysis is a bottom-up approach to
failure identification. It should begin at the lowest level of the
product design. Through analysis potential failure modes are identified
allowing early design change to correct potential problems before
fabricating hardware--a more cost-effective time to identify and fix
problems.
[7] U.S. General Accounting Office, Best Practices: Successful
Application to Weapon Acquisition Requires Changes in DOD‘s Environment
GAO/NSIAD-98-56 (Washington, D.C.: Feb. 24, 1998).
[8] U.S. General Accounting Office, Tactical Aircraft: F-22 Delays
Indicate Initial Production Rates Should Be Lower to Reduce Risks
GAO-02-298 (Washington, D.C.: Mar. 5, 2002).
[9] DOD Directive 5000.1, The Defense Acquisition System (Oct. 23,
2000), DOD
Instruction 5000.2, Operation of the Defense Acquisition System (Apr.
5, 2002), and DOD Regulation 5000.2-R, Mandatory Procedures for Major
Defense Acquisition Programs (MDAPS) and Major Automated Information
System (MAIS) Acquisition Programs (Apr. 5, 2002).
[10] Cost-reimbursement contracts provide for payment of allowable
incurred costs, to the extent prescribed in the contracts. They are
suitable for use only when uncertainties involved in contract
performance, such as research and development work, do not permit costs
to be estimated with sufficient accuracy. In contrast, fixed-priced
contracts, except those subject to price adjustment, provide for a
preestablished firm price, place maximum risk and full responsibility
for all costs and resulting profit or loss on the contractor, and
provide maximum incentive for the contractor to control costs and
perform effectively.
GAO‘s Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO‘s commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO‘s Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ’Today‘s Reports,“ on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select ’Subscribe to daily E-mail alert for newly
released products“ under the GAO Reports heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW,
Room LM Washington,
D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.
General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.
20548:
