Best Practices
DOD Can Achieve Better Outcomes by Standardizing the Way Manufacturing Risks Are Managed Gao ID: GAO-10-439 April 22, 2010Cost growth and schedule delays are prevalent problems in acquiring defense weapon systems. Manufacturing systems has proven difficult, particularly as programs transition to production. In December 2008, the Department of Defense (DOD) issued an updated version of its acquisition policy that reflects earlier consideration of manufacturing risks. A joint defense and industry group developed manufacturing readiness levels (MRL) to support assessments of manufacturing risks. Use of MRLs on all weapon acquisition programs has been proposed. In response to a congressional request, this report assesses the manufacturing problems faced by DOD, how MRLs can address manufacturing problems, how MRLs compare to manufacturing best practices of leading commercial firms, and challenges and barriers to implementing MRLs at DOD. In conducting our work, we contacted DOD, military services, and contractors; held interviews with leading commercial firms; reviewed program documents and policy proposals; and spoke with manufacturing experts.
DOD faces problems in manufacturing weapon systems--systems cost far more and take much longer to build than estimated. Billions of dollars in cost growth occur as programs transition from development to production, and unit-cost increases are common after production begins. Several factors contribute to these problems including inattention to manufacturing during planning and design, poor supplier management, and a deficit in manufacturing knowledge among the acquisition workforce. Essentially, programs did not identify and resolve manufacturing risks early in development, but carried risks into production where they emerged as significant problems. MRLs have been proposed as new criteria for improving the way DOD identifies and manages manufacturing risks and readiness. Introduced to the defense community in 2005, MRLs were developed from an extensive body of manufacturing knowledge that includes defense, industry, and academic sources. An analysis of DOD's technical reviews that assesses how programs are progressing show that MRLs address many gaps in core manufacturing-related areas, particularly during the early acquisition phases. Several Army and Air Force centers that piloted MRLs report these metrics contributed to substantial cost benefits on a variety of technologies and major defense acquisition programs. To develop and manufacture products, the commercial firms we visited use a disciplined, gated process that emphasizes manufacturing criteria early in development. The practices they employ focus on gathering sufficient knowledge about the producibility of their products to lower risks, and include stringent manufacturing readiness criteria to measure whether the product is sufficiently mature to move forward in development. These criteria are similar to DOD's proposed MRLs in that commercial firms (1) assess producibility at each gate using clearly defined manufacturing criteria to gain knowledge about manufacturing early, (2) demonstrate manufacturing processes in a production-relevant environment, and (3) emphasize relationships with critical suppliers. However, a key difference is that commercial firms, prior to starting production, require their manufacturing processes to be in control--that is, critical processes are repeatable, sustainable, and consistently producing parts within the quality standards. DOD's proposed MRL criteria do not require that processes be in control until later. Acceptance of MRLs has grown among some industry and DOD components. Yet, DOD has been slow to adopt a policy that would require MRLs across DOD. Concerns raised by the military services have centered on when and how the MRL assessments would be used. While a joint DOD and industry group has sought to address concerns and disseminate information on benefits, a consensus has not been reached. If adopted, DOD will need to address gaps in workforce knowledge, given the decrease in the number of staff in the production and manufacturing career fields.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Michael J. Sullivan Team: Government Accountability Office: Acquisition and Sourcing Management Phone: (937) 258-7915GAO-10-439, Best Practices: DOD Can Achieve Better Outcomes by Standardizing the Way Manufacturing Risks Are Managed
This is the accessible text file for GAO report number GAO-10-439
entitled 'Best Practices: DOD Can Achieve Better Outcomes by
Standardizing the Way Manufacturing Risks Are Managed' which was
released on April 22, 2010.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
April 2010:
Best Practices:
DOD Can Achieve Better Outcomes by Standardizing the Way Manufacturing
Risks Are Managed:
GAO-10-439:
GAO Highlights:
Highlights of GAO-10-439, a report to congressional requesters.
Why GAO Did This Study:
Cost growth and schedule delays are prevalent problems in acquiring
defense weapon systems. Manufacturing systems has proven difficult,
particularly as programs transition to production. In December 2008,
the Department of Defense (DOD) issued an updated version of its
acquisition policy that reflects earlier consideration of
manufacturing risks. A joint defense and industry group developed
manufacturing readiness levels (MRL) to support assessments of
manufacturing risks. Use of MRLs on all weapon acquisition programs
has been proposed. In response to a congressional request, this report
assesses the manufacturing problems faced by DOD, how MRLs can address
manufacturing problems, how MRLs compare to manufacturing best
practices of leading commercial firms, and challenges and barriers to
implementing MRLs at DOD. In conducting our work, we contacted DOD,
military services, and contractors; held interviews with leading
commercial firms; reviewed program documents and policy proposals; and
spoke with manufacturing experts.
What GAO Found:
DOD faces problems in manufacturing weapon systems”systems cost far
more and take much longer to build than estimated. Billions of dollars
in cost growth occur as programs transition from development to
production, and unit-cost increases are common after production
begins. Several factors contribute to these problems including
inattention to manufacturing during planning and design, poor supplier
management, and a deficit in manufacturing knowledge among the
acquisition workforce. Essentially, programs did not identify and
resolve manufacturing risks early in development, but carried risks
into production where they emerged as significant problems.
MRLs have been proposed as new criteria for improving the way DOD
identifies and manages manufacturing risks and readiness. Introduced
to the defense community in 2005, MRLs were developed from an
extensive body of manufacturing knowledge that includes defense,
industry, and academic sources. An analysis of DOD‘s technical reviews
that assesses how programs are progressing show that MRLs address many
gaps in core manufacturing-related areas, particularly during the
early acquisition phases. Several Army and Air Force centers that
piloted MRLs report these metrics contributed to substantial cost
benefits on a variety of technologies and major defense acquisition
programs.
To develop and manufacture products, the commercial firms we visited
use a disciplined, gated process that emphasizes manufacturing
criteria early in development. The practices they employ focus on
gathering sufficient knowledge about the producibility of their
products to lower risks, and include stringent manufacturing readiness
criteria to measure whether the product is sufficiently mature to move
forward in development. These criteria are similar to DOD‘s proposed
MRLs in that commercial firms:
* assess producibility at each gate using clearly defined
manufacturing criteria to gain knowledge about manufacturing early,
* demonstrate manufacturing processes in a production-relevant
environment, and,
* emphasize relationships with critical suppliers.
However, a key difference is that commercial firms, prior to starting
production, require their manufacturing processes to be in control”
that is, critical processes are repeatable, sustainable, and
consistently producing parts within the quality standards. DOD‘s
proposed MRL criteria do not require that processes be in control
until later.
Acceptance of MRLs has grown among some industry and DOD components.
Yet, DOD has been slow to adopt a policy that would require MRLs
across DOD. Concerns raised by the military services have centered on
when and how the MRL assessments would be used. While a joint DOD and
industry group has sought to address concerns and disseminate
information on benefits, a consensus has not been reached. If adopted,
DOD will need to address gaps in workforce knowledge, given the
decrease in the number of staff in the production and manufacturing
career fields.
What GAO Recommends:
GAO recommends that the Secretary of Defense require the use of MRLs
across DOD programs, strengthen the MRL criteria (process control) for
production start, assess the need for tools, and assess the
manufacturing workforce to address knowledge gaps. DOD partially
concurred with the first recommendation, and concurred with the other
three.
View the full [hyperlink, http://www.gao.gov/products/GAO-10-439]. or
key components. For more information, contact Michael Sullivan at
(202) 512-4841 or sullivanm@gao.gov.
[End of section]
Contents:
Letter:
Background:
Manufacturing Problems Are Attributed to Several Factors during the
Planning and Design Phases of Selected DOD Weapons Programs:
MRLs Have Been Proposed to Improve the Way DOD Identifies and Manages
Manufacturing Risk and Readiness:
DOD's Proposed MRLs Embody Many Best Practices of Leading Commercial
Firms:
MRLs Are Hampered by Lack of an Agencywide Policy and Manufacturing
Workforce Concerns:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Manufacturing Readiness Level (MRL) Definitions:
Appendix III: Manufacturing Readiness Level (MRL) Threads and
Subthreads (Risk Areas):
Appendix IV: Comments from the Department of Defense:
Appendix V: GAO Contact and Staff Acknowledgments:
Related GAO Products:
Tables:
Table 1: Basic Manufacturing Readiness Level Definitions:
Table 2: Basic Manufacturing Threads (Risk Areas) for MRL 1-10:
Table 3: Many Manufacturing Criteria Used by Leading Commercial Firms
Are Similar to DOD's MRLs:
Table 4: Percent of Manufacturing Workforce Decrease from 2001 to 2007:
Figures:
Figure 1: Distribution of Average Procurement Unit-Cost Growth after
Production Decision for Major Defense Acquisition Programs:
Figure 2: Contributing Factors to Manufacturing Problems for Four DOD
Case-Study Programs:
Figure 3: Relationship of MRLs to System Milestones and Technology
Readiness Levels (TRL):
Figure 4: GE CT Scanner Using Advanced Scintillator Material:
Figure 5: Honeywell Uses Three Producibility Models and MRL Workshop:
Figure 6: GE Aviation's Turbine Airfoils Lean Lab Proves Out
Production Processes:
Figure 7: Leading Commercial Firms Use Statistical Controls to Ensure
Quality Products:
Abbreviations:
CT: Computed Tomography:
DOD: Department of Defense:
MRL: Manufacturing Readiness Level:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
April 22, 2010:
The Honorable Bill Nelson:
Chairman:
Subcommittee on Emerging Threats and Capabilities:
Committee on Armed Services:
United States Senate:
The Honorable Jack Reed:
United States Senate:
The Department of Defense (DOD) has a well-documented history of
taking much longer and spending much more than originally planned to
develop and acquire its weapons systems. In particular, as systems
transition from development to production, programs experience
significant manufacturing problems. While DOD has made some progress
over the last two decades in addressing the problem--including policy
changes and advocating the use of best practices for product
development--GAO's recent weapon system reviews show that
manufacturing problems, among others, continue to hinder acquisition
cost, schedule, and performance outcomes.
It is essential to find better ways of doing business and, in
particular, to make sure systems are manufactured on time and cost-
effectively. To this end, leading commercial companies have achieved
more predictable outcomes from their manufacturing efforts because
they understand producibility--the relative ease of producing designs
of an item, product, or system economically with available production
techniques--and identify manufacturing risks early and manage them
effectively throughout a product's development life cycle.
On December 8, 2008, DOD issued a revised version of its policy
instruction on operation of the defense acquisition system that, among
other things, recognizes the need to consider manufacturing risks
earlier in the acquisition life cycle and assesses risks prior to key
decision points. In response to the need for the department to better
design and produce more affordable weapon systems, and to give
decision makers and managers better visibility into their program
risks, a joint defense and industry working group was established in
2004 to develop manufacturing readiness levels (MRL), a measurement
scale designed to improve the management and communication of
manufacturing risk and readiness throughout acquisitions. Similar to
technology metrics that measure the readiness of a technology, MRLs
are new manufacturing criteria that measures the manufacturing
maturity or readiness of a given technology, manufacturing process,
system, or element of a weapon system at various phases of the
acquisition life cycle.
In response to a request from the Senate Subcommittee on Emerging
Threats and Capabilities and Senator Reed, we reviewed DOD's efforts
to adopt MRLs. This report addresses (1) the manufacturing problems
experienced by selected DOD programs, (2) how MRLs can address DOD's
manufacturing problems, (3) how proposed MRLs compare to manufacturing
best practices of leading commercial companies, and (4) the challenges
and barriers to implementing MRLs at DOD.
To meet these objectives, we compared the manufacturing practices of
DOD and its large prime contractors with those of leading commercial
companies. We performed an aggregate analysis of DOD programs from our
annual weapons assessment.[Footnote 1] We also evaluated four major
defense weapon systems in production with known cost and schedule
problems to gain in-depth insights as to the nature and causes of
problems. We also evaluated two defense systems known to be producing
systems within cost and schedule goals and compared their practices to
those employed by commercial firms. We examined program documentation
and policy proposals, and held discussions with manufacturing and
systems-engineering officials from DOD program offices, prime
contractors, and the Defense Contract Management Agency. We also
reviewed lessons learned from DOD programs that pilot-tested MRLs. We
met with officials from the Office of the Secretary of Defense, Air
Force, Army, and Navy, Missile Defense Agency, Joint Defense
Manufacturing Technology Panel, Defense Acquisition University,
National Center for Advanced Technologies, and National Defense
Industrial Association to discuss manufacturing topics and MRLs. On
manufacturing workforce issues, we interviewed officials responsible
for planning activities within each of the military services and the
Defense Management Contract Agency. We compared manufacturing and
production considerations in the prior version of DOD's policy
instruction on operation of the defense acquisition system[Footnote 2]
to those in the current version of the policy instruction.[Footnote 3]
To identify manufacturing best practices of leading commercial
companies, we interviewed and obtained documentation from
manufacturing, quality, and supplier personnel at five companies, and
reported on four companies: GE Aviation, an aerospace company; GE
Healthcare, a producer of healthcare products and services; Honeywell
Aerospace, a provider of aircraft integrated avionics, engines,
systems, and services; Siemens Mobility, a producer of light rail
cars. We selected companies that manufacture complex products and have
won awards for quality manufacturing. Appendix I includes additional
details about our scope and methodology. We conducted this performance
audit from January 2009 to February 2010 in accordance with generally
accepted government auditing standards. These standards require that
we plan and perform the audit to obtain sufficient, appropriate
evidence to provide a reasonable basis for our findings and
conclusions based on our audit objectives.
We are making recommendations to the Secretary of Defense to require
an assessment of the manufacturing readiness across DOD programs using
MRL criteria, examine strengthening the MRL criteria related to
process capability and control, assess analytical model needs and
tools to support MRL assessments, and assess the manufacturing
workforce knowledge and skills base and develop a plan to address
DOD's current and future workforce knowledge gaps. In commenting on a
draft of this report, DOD partially concurred with the first
recommendation, and concurred with the other recommendations.
Background:
In recognition of the lack of manufacturing knowledge at key decision
points and the need to develop more affordable weapon systems, DOD
made recent changes to its policy. In 2008, the department made
constructive changes to its policy instruction on operation of the
defense acquisition system. It also developed MRLs as a measure that
could strengthen the way the department manages and develops
manufacturing-intensive systems. In 2004, the Joint Defense
Manufacturing Technology Panel[Footnote 4] sponsored a joint defense
and industry working group to design and develop MRLs for programs
across DOD. In May 2005, MRLs were first introduced to the defense
community in DOD's Technology Readiness Assessment Deskbook for
science and technology and acquisition managers to consider.
As new manufacturing readiness criteria, MRLs are a measurement scale
designed to provide a common metric and vocabulary for assessing
manufacturing maturity and risk. MRL assessments identify the risks
and manufacturing readiness of a particular technology, manufacturing
process, weapon system, subsystem, or element of a legacy program at
key milestones throughout the acquisition life cycle. There are 10
basic MRLs designed to be roughly congruent with comparable levels of
technology readiness levels for ease of use and understanding. Table 1
shows the MRLs and basic definitions (see appendix II for the detailed
MRL definitions).
Table 1: Basic Manufacturing Readiness Level Definitions:
MRL: 1;
Description: Basic manufacturing implications identified.
MRL: 2;
Description: Manufacturing concepts identified.
MRL: 3;
Description: Manufacturing proof of concept developed.
MRL: 4;
Description: Capability to produce the technology in a laboratory
environment.
MRL: 5;
Description: Capability to produce prototype components in a
production-relevant environment.
MRL: 6;
Description: Capability to produce a prototype system or subsystem in
a production-relevant environment.
MRL: 7;
Description: Capability to produce systems, subsystems, or components
in a production-representative environment.
MRL: 8;
Description: Pilot line capability demonstrated; ready to begin low-
rate initial production.
MRL: 9;
Description: Low-rate production demonstrated; capability in place to
begin full-rate production.
MRL: 10;
Description: Full-rate production demonstrated, and lean production
practices in place.
Source: Joint Defense Manufacturing Technology Panel.
[End of table]
The working group also developed a set of elements called "threads" to
provide acquisition managers and those conducting assessments an
understanding of the manufacturing risk areas (see table 2). For these
threads, desired progress is defined for each MRL, to provide an
understanding of risks as readiness levels increase from one MRL to
the next. Conceptually, these threads are manufacturing elements that
are essential to programs as they plan, prepare for, and manage the
activities necessary to develop a product. For example, the materials
thread requires an assessment of potential supplier capability by MRL
3 and an assessment of critical first-tier suppliers by MRL 7.
Likewise, the manufacturing personnel thread calls for identifying new
manufacturing skills by MRL 3 and identifying manufacturing workforce
requirements for the pilot line by MRL 7.
Table 2: Basic Manufacturing Threads (Risk Areas) for MRL 1-10:
Thread (risk areas): Technology and the Industrial Base; Description:
Requires an analysis of the capability of the national technology and
industrial base to support the design, development, production,
operation, uninterrupted maintenance support of the system and
eventual disposal (environmental impacts).
Thread (risk areas): Design;
Description: Requires an understanding of the maturity and stability
of the evolving system design and any related impact on manufacturing
readiness.
Thread (risk areas): Cost and Funding;
Description: Requires an analysis of the adequacy of funding to
achieve target manufacturing maturity levels. Examines the risk
associated with reaching manufacturing cost targets.
Thread (risk areas): Materials;
Description: Requires an analysis of the risks associated with
materials (including basic/raw materials, components, semi-finished
parts, and subassemblies).
Thread (risk areas): Process Capability and Control;
Description: Requires an analysis of the risks that the manufacturing
processes are able to reflect the design intent (repeatability and
affordability) of key characteristics.
Thread (risk areas): Quality Management;
Description: Requires an analysis of the risks and management efforts
to control quality, and foster continuous improvement.
Thread (risk areas): Manufacturing Personnel;
Description: Requires an assessment of the required skills,
availability, and required number of personnel to support the
manufacturing effort.
Thread (risk areas): Facilities;
Description: Requires an analysis of the capabilities and capacity of
key manufacturing facilities (prime, subcontractor, supplier, vendor,
and maintenance/repair).
Thread (risk areas): Manufacturing Management;
Description: Requires an analysis of the orchestration of all elements
needed to translate the design into an integrated and fielded system
(meeting program goals for affordability and availability).
Source: Joint Defense Manufacturing Technology Panel.
[End of table]
As shown, each basic thread (risk area) has a description and general
requirements for assessing risks for each thread. The working group
further decomposed these MRL threads into subthreads to provide users
a detailed understanding of the various kinds of manufacturing risks.
See appendix III for a detailed breakdown of these threads (risk
areas) for each MRL.
DOD's Long-standing History of Manufacturing Problems:
GAO has conducted an extensive body of work that highlights many of
the manufacturing-related problems that both DOD and its prime
contractors have faced. In many respects, DOD has recognized the
nature of these problems throughout the years and has taken a number
of proactive steps to address them. GAO's work has drawn on lessons
learned and best practices to recommend ways for DOD to improve the
way it develops and manufactures its weapon systems. Examples from our
reports include the following:
* In 1996, GAO reported the practices that world-class commercial
organizations had adopted to more efficiently produce quality
products, to improve DOD's quality assurance program.[Footnote 5] DOD
was spending $1.5 billion extra per year on military-unique quality
assurance requirements for major acquisitions and billions more on
cost and schedule overruns to correct problems. GAO concluded that
repeated unstable designs, poor process controls, and poor transition
to production caused the manufacturing quality problems. While DOD had
taken some actions, its culture was cited as the biggest reason for
slow adoption and unimplemented recommendations.
* In 1998, GAO reported on best commercial practices to offer ways to
improve the process DOD uses to manage suppliers engaged in developing
and producing major weapon systems.[Footnote 6] In assessing defense
contractors and two case studies of munitions programs, the report
concluded that suppliers were critical in the amount of technological
innovation they contribute to the final product.
* In 2002, GAO reported on how best practices could offer improvements
to the way DOD develops new weapon systems, primarily the design and
manufacturing aspects of the acquisition process.[Footnote 7] DOD's
record showed a history of taking longer and spending more than
planned to develop and acquire weapon systems, which reduced its
buying power. The report identified and recommended best practices for
capturing and using design and manufacturing knowledge early and new
development processes that included high-level decision points and
knowledge-based exit criteria before key decisions on production are
made. Essentially, one of the high-level decision points has become
what GAO commonly refers to as Knowledge Point 3--the point when a
program has demonstrated the manufacturing processes are mature. The
report also recommended a best practice that includes a standard
called the Process Capability Index (Cpk), a process performance
measurement that quantifies how closely a product is running to its
specification limits. The index indicates how well the processes
statistical performance meets its control limit requirement.
* In 2008, GAO reported on how DOD and its defense contractors can
improve the quality of major weapon systems.[Footnote 8] We reported
that if DOD continued to employ the same acquisition practices as it
has in the past, the cost of designing and developing its systems
could continue to exceed estimates by billions of dollars. Quality
problems were identified as the cause for cost overruns, schedule
delays, and reduced weapon-system availability. Like DOD prime
contractors, leading commercial firms rely on many practices related
to systems engineering, manufacturing, and supplier quality, but they
were more disciplined, and had institutionalized processes to ensure
quality.
* Since 2003, GAO has issued a series of annual assessment reports on
selected weapons programs, increasing from 77 to 96 programs reviewed.
[Footnote 9] At $296 billion, the cumulative cost growth for DOD
programs reported in 2009 was found to be higher than it had been five
years earlier, and the overall performance of weapon system programs
was still poor. Although the cost growth and the 22-month average
delay in delivering initial capabilities was not attributed to
manufacturing alone, the lack of production maturity was cited as one
of three key knowledge areas contributing to the department's cost
growth, schedule delay, and performance problems.
Revised Policy Incorporates Manufacturing Best Practices:
DOD's December 2008 revision to its policy instruction on operation of
the defense acquisition system[Footnote 10] incorporates a number of
the best practices we identified in our previous work. The instruction
covers the entire life cycle and considers manufacturing risks earlier
in the acquisition life-cycle framework. In a November 2003 report on
DOD's May 2003 revision to its policy, we reported that much of the
revised policy agrees with GAO's extensive body of work and that of
successful commercial firms. While we assessed DOD's revised policy as
providing a good framework for capturing knowledge about critical
technologies, product design and manufacturing processes, we reported
in 2006 that acquisition officials were not effectively implementing
the acquisition policy's knowledge-based process.[Footnote 11] We
reported that the effective implementation of policy was limited by
the absence of effective controls that require compliance and specific
criteria for clearly demonstrating that acceptable levels of knowledge
about technology, design, and manufacturing have been attained at
critical junctures before making further investments in a program. We
concluded that without specific criteria--or standards against which a
judgment or decision is quantifiably based--decision makers are
permitted to make decisions on the basis of subjective judgment. The
December 2008 revised policy instruction establishes target maturity
criteria for measuring risks associated with manufacturing processes
at milestone decision points.[Footnote 12]
During the material solutions phase, prior to milestone A, the 2008
policy instruction requires the analysis of alternatives to assess
"manufacturing feasibility." During the technology development phase,
prior to milestone B, the instruction states the following:
* Prototype systems or appropriate component-level prototyping shall
be employed to "evaluate manufacturing processes."
* A successful preliminary design review will "identify remaining
design, integration, and manufacturing risks."
* A program may exit the technology development phase when "the
technology and manufacturing processes for that program or increment
have been assessed and demonstrated in a relevant environment" and
"manufacturing risks have been identified."
After milestone B, one of the purposes of the engineering and
manufacturing development phase is to "develop an affordable and
executable manufacturing process." The instruction says that: "the
maturity of critical manufacturing processes" is to be described in a
post-critical design review assessment; system capability and
manufacturing process demonstration shall show "that system production
can be supported by demonstrated manufacturing processes;" and the
system capability and manufacturing process demonstration effort shall
end, among other things, when "manufacturing processes have been
effectively demonstrated in a pilot line environment, prior to
milestone C."
Finally, at milestone C, the instruction establishes two entrance
criteria for the production and deployment phase, which include "no
significant manufacturing risks" and "manufacturing processes [are]
under control (if Milestone C is full-rate production)." Low-rate
initial production follows in order to ensure an "adequate and
efficient manufacturing capability." In order to receive full-rate
production approval, the following must be shown:
1. "demonstrated control of the manufacturing process,"
2. "the collection of statistical process control data," and:
3. "demonstrated control and capability of other critical processes."
Even with the updated policy instruction in place that includes
guidance for most knowledge-based practices, inconsistent
implementation has hindered DOD's past efforts to reform its
acquisition practices. For example, we reported in 2006 that DOD was
not effectively implementing the knowledge-based approach process and
evolutionary approach emphasized in its policy.[Footnote 13] While the
policy outlined specific knowledge-based process of concept refinement
and technology development to help ensure a sound business case is
developed before committing to a new development program, we found
that almost 80 percent of the programs we reviewed were permitted to
bypass this process.
Manufacturing Problems Are Attributed to Several Factors during the
Planning and Design Phases of Selected DOD Weapons Programs:
Defense acquisition programs continue to have problems manufacturing
weapon systems. As a result, systems cost far more and take far longer
to produce than estimated. Many programs authorized to enter
production experienced billions of dollars in cost growth after the
authorization--nearly two-thirds of those programs reported increases
in average procurement unit costs. Several factors contribute to these
issues during the planning and design phases. These include the
inattention to manufacturing during planning and design, poor supplier
management, and lack of a knowledgeable manufacturing workforce.
Essentially, some of these programs moved into production without
considering manufacturing risks earlier in development. This hindered
managers from later managing those risks until they became
problematic, and also led to subsequent problems with supplier
management, such as prime contractors conducting little oversight of
suppliers. Some programs also had an inadequate workforce--in terms of
insufficient knowledge and numbers--to effectively manage and oversee
defense manufacturing efforts.
Manufacturing Contributed to Growth in Cost and Delays in Schedule:
Defense acquisition programs continue to be troubled by unstable
requirements, immature technology, and a lack of manufacturing
knowledge early in design, resulting in more costly products that take
longer to produce. Our 2009 annual assessment shows that total
research and development costs were 42 percent higher than originally
estimated. These higher costs reflect in part the learning that takes
place as manufacturing processes are established and used to produce
the first prototypes.
Even programs that have been authorized to begin production have
experienced substantial cost growth after the production decision.
Production performance can be measured by examining the cost growth as
expressed in changes to average procurement unit cost. This represents
the value DOD gets for the procurement dollars invested in a certain
program and shows the net effect of procurement cost growth and
quantity changes. Figure 1 shows the levels of average procurement
unit-cost growth for selected major defense acquisition programs.
[Footnote 14]
Figure 1: Distribution of Average Procurement Unit-Cost Growth after a
Production Decision for Major Defense Acquisition Programs:
[Refer to PDF for image: pie-chart]
Programs with less than 5% growth: 39%;
Programs with 5% to 10% growth: 29%;
Programs with 11% to 15% growth: 16%;
Programs with more than 15% growth: 10%;
Source: GAO analysis of DOD data.
Note: Data include all major defense acquisition programs that entered
production in fiscal year 2000 or later.
[End of figure]
As indicated in figure 1, nearly two-thirds of programs that entered
production after 2000 reported more than a 5 percent increase in
average unit cost growth, while 32 percent of programs reported
average unit cost growth that ranged from 11 percent to more than 15
percent. One program reported a 25 percent increase in average
procurement unit cost. Further, 42 percent of those programs
experienced production cost increases when procured quantities
decreased or remained the same. For example, the Black Hawk
helicopter's 2007 production estimate had no increase in quantities
since 2005, yet its production cost increased $2.3 billion, and
average procurement unit cost rose by 13 percent. The Joint Air-to-
Surface Standoff Missile had an 8 percent quantity decrease since the
2004 production decision; but the production costs increased by $561
million and average procurement unit cost increased by 25 percent.
As for schedule growth, DOD has continued to experience delays in
delivering new or modified weapon systems to the warfighter. Over 50
percent of current programs in production have encountered some form
of delay after the production decision, when manufacturing processes
should be in control. Consequently, warfighters often must operate
costly legacy systems longer than expected, find alternatives to fill
capability gaps, or go without the capability altogether.
The four DOD weapon systems we selected for in-depth review with known
cost, schedule, and performance problems reported several key factors
that contributed to manufacturing problems. These include the
inattention to manufacturing during planning and design, poor planning
for supplier management, and lack of a knowledgeable manufacturing
workforce. Capturing critical manufacturing knowledge during the
planning and design phases before entering production helps to ensure
that a weapon system will work as intended and can be manufactured
efficiently to meet cost, schedule, and quality targets. The programs
in our review often lacked manufacturing knowledge at key decision
points, which led to cost growth and schedule delays. For example, the
Joint Air-to-Surface Standoff Missile program--an autonomous, air-to-
ground missile designed to destroy high-value targets--experienced a
critical unit-cost breach due to missile reliability problems not
being addressed early in the design phase.[Footnote 15] Also, the
Electromagnetic Aircraft Launch System--a new catapult technology
being developed for the Navy's newest class of aircraft carriers--had
experienced problems manufacturing compatible materials, which
resulted in cost growth and schedule delays and was the focus of
recent congressional interest. Figure 2 summarizes contributing
factors for manufacturing problems experienced by the four DOD weapon
systems.
Figure 2: Contributing Factors to Manufacturing Problems for Four DOD
Case-Study Programs:
[Refer to PDF for image: illustrated table, containing photograph for
each program]
DOD Program: Exoatmospheric Kill Vehicle;
Source of manufacturing problems: Inattention to manufacturing during
planning and design: [Check];
Source of manufacturing problems: Poor supplier management planning:
[Check];
Source of manufacturing problems: Lack of workforce knowledge and
skills: [Check];
General problems:
* Immature technologies caused development problems;
* Cost and schedule problems increased total cost of the interceptor.
DOD Program: Electromagnetic Aircraft Launch System;
Source of manufacturing problems: Inattention to manufacturing during
planning and design: [Check];
Source of manufacturing problems: Poor supplier management planning:
[Empty];
Source of manufacturing problems: Lack of workforce knowledge and
skills: [Check];
General problems:
* Development resulted in cost growth and schedule delays.
DOD Program: H-1 Helicopter Upgrade Program;
Source of manufacturing problems: Inattention to manufacturing during
planning and design: [Check];
Source of manufacturing problems: Poor supplier management planning:
[Check];
Source of manufacturing problems: Lack of workforce knowledge and
skills: [Check];
General problems:
* Decision to remanufacture increased costs (utility helicopter
configuration);
* Systems engineering and configuration management challenges.
DOD Program: Joint Air-to-Surface Standoff Missile;
Source of manufacturing problems: Inattention to manufacturing during
planning and design: [Check];
Source of manufacturing problems: Poor supplier management planning:
[Check];
Source of manufacturing problems: Lack of workforce knowledge and
skills: [Check];
General problems:
* Increased costs and schedule delays;
* Reliability problems.
Source: GAO analysis of Army, Air Force, Navy, and Missile Defense
Agency data. Images: Missile Defense Agency and Boeing public Web site
per GMG program office (top); CVN-21 Program Office 050708-D-8455H-001
Washington, D.C. (July 8, 2005) U.S. Navy graphic (released) (second
from top); USMC Light/Attack Helicopter (H-1) Program Office, PMA276.
(third from top); Integrated Test 2 accomplished December 2006
(bottom).
[End of figure]
As indicated, most of the programs had more than one major problem
related to manufacturing. These issues illustrate the major problems
we discussed with defense and contractor officials, but do not
encompass all the manufacturing problems experienced by the programs.
For example, a recent Air Force study reports that manufacturing and
quality assurance requirements are not included in the contracts to
develop weapon systems, which could affect the contractor's approach
to manufacturing. Officials from the Defense Contract Management
Agency--a DOD component that works directly with defense suppliers to
ensure that supplies and services are delivered on time, at projected
cost, and meet performance requirements--also reported similar
contract issues that could affect contractor performance on
manufacturing.
Manufacturing Was Overlooked during Early Development:
Each of the four programs we examined did not give manufacturing
strong consideration during the early planning and design phases.
Programs were moved into production largely without considering
manufacturing risks earlier in the acquisition process, as
demonstrated by the experiences of the Exoatmospheric Kill Vehicle and
the H-1 helicopter upgrade program. The Exoatmospheric Kill Vehicle
was designed to intercept and destroy high-speed ballistic missile
warheads in mid-flight, while the H-1 upgrade program converts the
attack helicopter and the utility helicopter to the AH-1Z and UH-1Y
configurations, respectively.
The Exoatmospheric Kill Vehicle program was put on an accelerated
development schedule in response to a directive to develop and deploy,
at the earliest possible date, ballistic missile defense drawing on
the best technologies available. According to the contractor, it
bypassed some of its normal development-review processes to accelerate
delivery of the vehicle, which also resulted in a high acceptance of
manufacturing risks without sufficient identification and management
of risk-mitigation plans. For example, the program went into
production without completing qualification testing. In addition, the
contractor continued to incorporate design changes while supplier
production was ongoing, resulting in rework and disruption to the
production line. Early lots of kill vehicles were built manually by
engineers in the absence of automated production processes, which
caused dissimilarities among vehicles in the fleet and will make
refurbishments difficult.[Footnote 16]
For several reasons, the H-1 helicopter upgrade program did not
include manufacturing in the early phases of planning and also
proceeded to production before its design was mature, according to the
contractor. First, the program underestimated the complexity of
updating and remanufacturing the aircraft without historical drawings.
The emphasis was placed on minimizing development costs and resources
were not available to assess manufacturing challenges early in the
redesign process. Furthermore, the program started low-rate production
before completing operational evaluation testing. As a result, the
problems uncovered during testing had to be corrected on aircraft that
were on the assembly line. Also, constant change orders and factory
bottlenecks, among other problems, affected program costs and
schedules. The schedule pressure allowed little opportunity to remedy
the manufacturing problems, resulting in more complicated and
expensive fixes. Ultimately the schedule slowed and the costs
increased to the point that the program abandoned the remanufacturing
upgrade and, instead, opted to purchase newly manufactured aircraft
cabins for the UH-1Y configuration.
Poor Planning Led to Supplier Problems:
Inattention to manufacturing during planning and design led to
subsequent problems with supplier management in two major defense
acquisition programs we reviewed. Specifically, the prime contractors
did not give adequate attention to managing their suppliers. For
example, program officials for the Joint Air-to-Surface Standoff
Missile told us that the responsibility for manufacturing processes
and discipline shifted in the 1990s from the government to the defense
contractors. The government started to rely on the prime contractor to
ensure quality and reliability, particularly with subtier suppliers.
In this case, the program office told us that the prime contractor for
the missile program relied on the subtier suppliers to self-report
their capabilities and did not engage in effective oversight of their
work, which led to defective parts. The program office recently
recruited experts in manufacturing to help the prime contractor
address their supplier problems more effectively.
In the Exoatmospheric Kill Vehicle program, supplier quality was
inconsistent, resulting in unnecessary rework and uncovering problems
late in production. For many suppliers, the kill vehicle program
represents a small portion of their business, so the emphasis on
quality was often lacking. Further, the program was initially procured
as a capability based program, rather than requirements based program.
Thus, the prime contractor did not impose requirements on the
subcontractors to comply with stringent requirements for space
programs. In turn, the subcontractors did not implement sufficient
requirements which led to recurring quality issues.
Lack of Manufacturing Knowledge Contributed to Problems:
Some DOD programs and prime contractors had an inadequate defense
manufacturing workforce--both in terms of numbers and experience--to
effectively manage and oversee manufacturing efforts, which resulted
in schedule delays or cost inefficiencies. The manufacturing workforce
includes occupations such as specialists in quality assurance,
business, manufacturing engineering, industrial engineering, and
production control. In many cases, the programs lacked manufacturing
expertise early in development, which hindered the program's ability
to later manage manufacturing risks. For example, the contractor for
the Electromagnetic Air Launch System did not have sufficient systems-
engineering personnel involved in the design to help it transition
from development to production. As a result, the program encountered
schedule delays and cost increases. DOD conducted a program assessment
review, which led the program office and contractor to increase
systems engineering staff.
For the Exoatmospheric Kill Vehicle program, the contractor's
workforce and manufacturing processes could not readily undertake the
rigors of production for a space-based capability, part of which must
be manufactured in a clean room environment, and all of which commands
rigorous processes and procedures due to highly technical designs. The
contractor's hourly assembly personnel were trained to build tactical
missiles on a high-rate production line and were not sufficiently
trained in the quality-control standards required by clean-room
manufacturing, such as carefully controlling foreign-object debris,
specially maintaining the clean room, and using a partner in certain
high-level tasks to ensure all steps are properly followed. These
standards were not institutionalized, and the contractor eventually
had to modify its facilities and production standards to correct the
manufacturing problems. The facility had to be retooled and
reconfigured late in development. The contractor also experienced high
turnover in its workforce due to the increasing demands associated
with working in a clean-room environment and working long hours.
MRLs Have Been Proposed to Improve the Way DOD Identifies and Manages
Manufacturing Risk and Readiness:
The Joint Defense Manufacturing Technology Panel working group has
proposed MRLs as new manufacturing readiness criteria that could
improve weapon system outcomes by standardizing the way programs
identify and manage manufacturing risks associated with developing and
fielding advanced weapon systems. MRLs were first introduced to the
defense community in DOD's 2005 Technology Readiness Assessment
Deskbook as an important activity for science and technology and
acquisition managers to consider. An analysis by the working group
shows that MRLs address many of the manufacturing issues not covered
by DOD's technical reviews, particularly reviews conducted in the
early phases of acquisition. In their development, comprehensive
efforts were undertaken to design and develop MRLs from DOD as well as
industry resources. For example, the working group formulated MRLs
from a manufacturing knowledge base of defense, industry, and academia
to address two key areas of risk--immature product technologies and
immature manufacturing capability. The working group also designed
MRLs as a structured and disciplined approach for the way
manufacturing risk and readiness is expected to be identified and
assessed. The working group also developed a set of tools that include
a deskbook, checklist, and a website to help managers and users apply
MRLs and conduct assessments. In addition, the Army and Air Force
report that their use of MRLs on pilot programs contributed to
substantial cost benefits on a variety of programs, including major
acquisition programs.
MRLs Were Developed from Knowledge-Based Resources on Manufacturing:
To develop MRLs, the working group conducted comprehensive sessions
with industry participants to ensure the metrics and vocabulary for
assessing manufacturing readiness would be an all-inclusive body of
knowledge. Officials stated that a mature set of manufacturing
knowledge resources already existed but it was scattered and not
consistently applied in a disciplined way that aligned with the DOD
acquisition life-cycle framework. In their formulation, MRLs were
developed from an extensive body of manufacturing knowledge that
included, but was not limited to, the following defense, industry, and
academic sources:
* DOD Instruction 5000.02, Operation of the Defense Acquisition System
(Dec. 8, 2008),
* Navy best-practices manual for using templates on design and
manufacturing best practices,
* Air Force manufacturing development guide,
* military standards and specifications, and:
* Malcolm Baldrige quality award criteria.
Other standards and technical sources were obtained from the Institute
of Electrical and Electronics Engineers, the International Standards
Organization on quality management systems, automotive industry
quality standards, and the supplier model from the Massachusetts
Institute of Technology.
Analysis Shows MRLs Address Manufacturing Gaps in DOD's Technical
Reviews:
An analysis conducted by the working group shows that MRLs address
many of the manufacturing gaps identified in several of DOD's
technical reviews[Footnote 17] that provide program oversight and
determine how well programs are meeting expected goals, particularly
the reviews conducted in the early acquisition phases. According to
the working group, addressing these manufacturing gaps is fundamental
to improving the way programs plan, design, and prepare for
manufacturing. For example, the working group's analysis shows that
DOD's current systems-engineering technical review checklist used for
preliminary design reviews[Footnote 18] has only 27 of 759 total
questions that deal with core manufacturing-related questions, whereas
the MRL 6 assessment checklist for this juncture has 169 core
manufacturing questions. More importantly, the technical review
checklist did not address key manufacturing discipline in the areas of
program management, systems engineering, requirements management, risk
management, and program schedule. Similarly, the technical review
checklist used for critical design reviews[Footnote 19] has only 22 of
824 total questions that deal with core manufacturing questions,
whereas the MRL 7 assessment checklist for this juncture has 162 core
questions. Core manufacturing disciplines were not addressed in the
specific areas of management metrics, manufacturing planning,
requirements management, system verification, and other areas.
Finally, DOD's technical review checklist used for production
readiness reviews[Footnote 20] has 194 of 613 total questions that
deal with core manufacturing questions. While the MRL 8 assessment
checklist has 14 fewer core questions on manufacturing at this
juncture, the working group stated these core manufacturing questions
are addressed earlier in the acquisition framework, which is
reflective of commercial best practices where such manufacturing
topics and discipline are addressed, in contrast to DOD's current
practice.
Draft Deskbook Explains MRL Application and Assessments:
The draft MRL deskbook is a detailed instructional resource on how to
apply MRLs and conduct assessments of manufacturing risk and
readiness, such as how to structure and apply evaluations to a
technology, component, manufacturing process, weapon system, or
subsystem using the MRL definitions. It also demonstrates how
assessments should be carried out at various phases by the managers of
science and technology projects and technology demonstration projects
intending to transition directly to the acquisition community, as well
as acquisition program managers and the people involved in conducting
assessments. According to the working group, MRLs can not only be used
to improve how DOD manages and communicates manufacturing risk and
readiness, but can also give decision makers and manager's better
visibility into program risks. For example, a variety of manufacturing
status and risk evaluations have been performed for years as part of
defense acquisition programs in a variety of forms--for example,
production readiness reviews, manufacturing management/production
capability reviews, etc. However, these structured and managed reviews
do not use a uniform metric to measure and communicate manufacturing
risk and readiness.
MRLs, when used in combination with technology readiness levels, are
expected to address two key risk areas--immature product technologies
and immature manufacturing capability. The draft deskbook says that it
is common for manufacturing readiness to be paced by technology
readiness or design stability, and that it is not until the product
technology and product design are stable that manufacturing processes
will be able to mature. MRLs can also be used to define manufacturing
readiness and risk at the system or subsystem level. For these
reasons, the MRL definitions were designed to include a target level
of technology readiness as a prerequisite for each level of
manufacturing readiness. Figure 3 shows the relationship of MRLs to
system milestones and technology readiness levels in the defense
acquisition life-cycle framework.
Figure 3: Relationship of MRLs to System Milestones and Technology
Readiness Levels (TRL):
[Refer to PDF for image: illustrated table]
Material Solution Analysis:
MRL 1: Basic manufacturing implications identified;
TRL 1: Basic principles observed.
MRL 2: Manufacturing concepts identified;
TRL 2: Concept formulated.
MRL 3: Manufacturing proof of concept developed;
TRL 3: Proof of concept.
MRL 4: Capability to produce the technology in a laboratory
environment;
TRL 4: Breadboard in laboratory.
Milestone A.
Technology Development:
MRL 5: Capability to produce prototype components in a production
relevant environment;
TRL 5: Breadboard in representative environment.
MRL 6: Capability to produce a prototype system or subsystem in a
production relevant environment;
TRL 6: Prototype in representative environment.
Milestone B.
Engineering and Manufacturing Development:
MRL 7: Capability to produce systems, subsystems or components in a
production representative environment;
TRL 7: Prototype in operational environment.
MRL 8: Pilot line capability demonstrated; ready to begin low rate
initial production;
TRL 7: Prototype in operational environment.
Production and Deployment:
MRL 9: Low rate production demonstrated; capability in place to begin
full rate production. Production cost targets are met;
TRL 8: System qualification.
MRL 10: Full rate production demonstrated and lean production
practices in place. Production unit cost goals are met;
TRL 9: Mission proven.
Source: GAO analysis of DOD chart.
Note: Alignment of MRLs and TRLs within the DOD acquisition framework
are generalized and may not align exactly as illustrated.
[End of figure]
MRL Assessments Provide Basis for Identifying, Planning, and Managing
Program Risks:
MRL assessments are intended to leverage better manufacturing
knowledge, enabling managers to be aware of problems or risks early in
development, when they are easier to resolve and before significant
investments are made. In turn, these risks can be addressed earlier in
the life cycle when costs are lower. For example, the ability to
transition technology smoothly and efficiently from the laboratories,
onto the factory floor, and into the field is a critical enabler for
evolutionary acquisition.
Assessments can be applied to a technology, manufacturing process,
weapon system, or subsystem using the definitions as a standard. As
part of the assessment, a comparison is made between the actual MRLs
and the target MRL levels. The difference between the two identifies
the risks and forms the basis for assisting managers to develop a plan-
-called a manufacturing maturation plan--to remove or reduce them.
Risks should be identified throughout the life cycle and, when targets
are not met, the plan updated to ensure the appropriate MRL will be
achieved at the next decision point. The manufacturing maturation plan
identifies manufacturing risks and provides a plan for mitigating each
risk area throughout the duration of the technology or product-
development program. The draft MRL deskbook says every assessment of
manufacturing readiness should have an associated plan for areas where
the MRL has not achieved its target level. The deskbook requires a
manufacturing maturation plan to include the most essential items in
planning for the maturity of an element of assessment that is below
its target MRL. These include a statement of the problem that
describes areas where manufacturing readiness falls short of the
target MRLs, including key factors and driving issues, solution
options and consequences of each option, and a maturation plan with a
schedule and funding breakout. Other information should include the
status of funding to execute the manufacturing plan and specific
actions to be taken and by whom, and the MRL to be achieved and when
it will be achieved.
MRL Pilot Programs Show Positive Benefits:
Army and Air Force programs have pilot-tested MRLs on science and
technology and some major acquisition programs in an effort to
increase the manufacturing readiness and maturity to higher levels
appropriate to the phase of development. Both services performed MRL
assessments on selected pilot programs to address manufacturing risks
and assess technology transition. The Army reports numerous benefits
from the use of MRLs such as manufacturing efficiencies, improved
labor utilization, and cost benefits. Similarly, the Air Force has
used MRLs to manage its manufacturing risks associated with new
technologies, yielding tangible benefits. While MRLs cannot take full
credit for all benefits derived in the pilot programs, officials noted
they are a good way to manage, mitigate, and communicate--between
science and technology, acquisition, the user, and the system
developer--readiness and risks early and throughout the acquisition
process to avoid major consequences from manufacturing-related
problems. These programs provide insight on how the acquisition
community can utilize MRLs within weapon system programs.
Army:
In 2004, the Army's Aviation and Missile Research, Development and
Engineering Center began applying MRLs to various technologies in
concept development, including those technologies transitioning to
engineering and manufacturing development. Officials stated that
without cost and manufacturing readiness planning, science and
technology programs face certain barriers to transition, resulting in:
(1) high unit production cost caused by a focus on technology without
regard to affordability; and (2) manufacturing problems caused by
design complexity resulting in a technology that is not feasible to
manufacture. For example, the Army has applied MRLs to many programs,
including warfighter-protection materials, Micro-Electro-Mechanical
Systems, embedded sensors, and helicopter cabin structures. The
warfighter-protection program--the next generation of helmets and body
gear--reported that it was able to reduce scrap by 60 percent and
reduced touch labor by 20 to 40 percent. On programs where cost
benefits could be roughly calculated, the Army believes that MRLs,
among other improvement initiatives, contributed to the $426 million
in benefits on seven programs. MRLs were also used as a metric in the
Technology Transition Agreement to communicate manufacturing maturity
and facilitate a smooth transition to the acquisition community.
Air Force:
Air Force officials we met with discussed using MRLs to assess and
identify gaps and understand risks in manufacturing maturity that
would delay technology transition into an advanced systems development
program or a fielded system upgrade. The Air Force has conducted
several MRL assessments on advanced technology demonstrations and
major defense acquisition programs, including the MQ-9 Reaper Unmanned
Aircraft, Joint Strike Fighter, Advance Medium-Range Air-to-Air
Missile, X-band thin radar array, and Sensor Hardening for Tactical
Systems. Officials reported that the use of MRLs have contributed
millions of dollars in cost avoidance, increased production rates, and
has accelerated technology transition. For example, the Air Force
reported realizing $65 million in savings by addressing problems with
a costly manual drilling process. MRLs were used to raise new drilling
technology from MRL 4 to MRL 9, achieving a unit-cost savings of
$17,000 per aircraft from reduced tooling, manpower, floor space
usage, and time.
Because of MRL assessment's success on advanced technology programs,
the Assistant Secretary of the Air Force for Acquisition directed the
program office to perform MRL assessments on key MQ-9 Reaper
manufacturing processes and technologies. The MQ-9 Reaper is an
unmanned aerial vehicle designed to provide a ground attack capability
during reconnaissance and surveillance missions. Officials stated that
the MRL assessment results have (1) identified five areas that needed
review prior to a milestone C production decision; (2) identified two
risks to full-rate production--mitigations are in progress; and (3)
provided evidence to support the contractor's ability to meet the
production goal of two aircraft per month. To ensure that
manufacturing requirements are enforced, officials have developed
policy for programs managers to assess manufacturing readiness at key
decision points. To support that policy, the Air Force has developed
training for integrated product teams to execute the manufacturing
readiness assessments. Also in August 2009, the Air Force Institute of
Technology established a Manufacturing Readiness Assessment course to
provide training for the assessments within the Air Force and is
currently open to all services and industry.
DOD's Proposed MRLs Embody Many Best Practices of Leading Commercial
Firms:
To successfully develop and manufacture their products, the commercial
firms we visited used a disciplined, gated process that emphasized
manufacturing criteria early and throughout the product's development.
To measure manufacturing maturity, these firms developed processes
that give manufacturing readiness and producibility primary importance
throughout the product-development process, focusing on producing a
product, not developing a technology. The goal is business
profitability, and manufacturing maturity is important to this process
from the earliest stages.
The best practices they employed were focused on gathering a
sufficient amount of knowledge about their products' producibility in
order to lower manufacturing risks and included stringent
manufacturing readiness criteria--to measure whether the product was
mature enough to move forward in its development. In most respects,
these criteria are similar to DOD's proposed MRLs. For example, as
with MRLs, commercial firms:
* assess producibility at each gate using clearly defined
manufacturing readiness criteria,
* gain knowledge about manufacturing early,
* demonstrate manufacturing processes in a production-relevant
environment, and:
* emphasize the importance of effective supply-chain management.
Essentially, commercial firms emphasize these criteria in order to
maximize their understanding of manufacturing issues, to mitigate
manufacturing risks that could affect business profitability or
schedule goals for getting the product to market. DOD's MRLs were
designed to mitigate similar manufacturing risks. However, the
difference is that the commercial firms we visited required that their
manufacturing processes be in control prior to low-rate production,
whereas DOD's proposed MRL criteria do not require as early control of
the manufacturing process.
DOD's MRLs Are Similar to Manufacturing Criteria Used by Leading Firms:
Leading commercial firms use manufacturing readiness criteria, similar
to DOD's MRLs, to assess the producibility of a system, gathering
knowledge about the producibility of a product and the maturity of the
manufacturing process. These criteria are applied early, even before a
product formally enters into development, to identify and manage
manufacturing risks and gaps. Additional manufacturing readiness
criteria are applied through all the stages of a product's development
and production until the product is ready for commercial release. The
firms we visited used manufacturing readiness criteria to measure both
the readiness of the product or material to enter into development and
to proceed through the necessary gates. Table 3 below shows examples
of manufacturing readiness criteria that are common to both the MRLs
and the commercial criteria, to illustrate their similarities. Both
emphasized identifying risks and developing plans to mitigate these
risks, setting realistic cost goals, and proving out manufacturing
processes, material, and products.
Table 3: Many Manufacturing Criteria Used by Leading Commercial Firms
Are Similar to DOD's MRLs:
MRL/phases: MRL 1-3; Pre-Concept Development (Invention Stage);
Commercial manufacturing criteria and DOD MRLs:
* Relevant materials and processes evaluated for manufacturability;
* Cost models developed for new processes;
* Critical manufacturing processes identified.
MRL/phases: MRL 4; Concept Development;
Commercial manufacturing criteria and DOD MRLs:
* Risk-mitigation plans in place for management of manufacturing risks;
* Key materials issues identified;
* Manufacturing strategy developed and integrated with acquisition
strategy.
MRL/phases: MRL 5-6; Technology Development;
Commercial manufacturing criteria and DOD MRLs:
* Basic design requirements defined and all critical technology and
components tested and evaluated;
* Critical suppliers identified/supply chain in place;
* Realistic cost targets are set;
* Manufacturing processes and materials demonstrated in a production-
relevant environment.
MRL/phases: MRL 7; Product Development;
Commercial manufacturing criteria and DOD MRLs:
* Product requirements and features well-defined;
* Pilot lines' yield-data gathered and assessed;
* Manufacturing processes demonstrated in a production-representative
environment.
MRL/phases: MRL 8; Production (Preparation);
Commercial manufacturing criteria and DOD MRLs:
* Quality targets demonstrated on pilot line;
* Manufacturing processes verified for low-rate production on pilot
line;
* Yield and rates required to begin low-rate production verified;
* Manufacturing plan completed and all key manufacturing risks
mitigated.
Source: GAO analysis of DOD and commercial data.
[End of table]
Best Practice: Commercial Companies Emphasize Manufacturing Criteria
Early and at Every Stage of the Product-Development Life Cycle:
Each commercial firm we visited developed a disciplined framework for
product development that assessed producibility at each gate using
clearly defined manufacturing-maturity criteria that are similar in
many respects to DOD's MRLs. These include assessments of all aspects
of manufacturing technology and risk, supply-chain issues, production
facilities and tooling, and materials. Throughout the product-
development life cycle, these criteria were applied to determine entry
or exit into the next phase and led to informed decisions about
whether the product was ready to move forward in its development.
Manufacturing risks--such as those found in new manufacturing
technologies or production facilities, new or revolutionary materials
or supply-chain issues--were assessed at each step. Deliverables,
including risk-identification and mitigation plans, manufacturing
plans, and funding and resource needs, were required at each gate in
order to progress to the next product-development gate. Targets were
developed for each gate, including cost, schedule, and yield goals,
and the product team was responsible for either meeting these targets
or having risk-mitigation plans in place if the targets had not been
met.
GE Aviation exemplifies this disciplined process, using a highly
structured gated process with detailed checklists for entry and exit
into each phase. Like DOD's MRLs, these checklists contain
increasingly detailed criteria--as they move from product start to
production--for evaluating manufacturing technologies, cost drivers,
materials, and supply-chain issues. Structured teams are brought
together, tools are identified for execution and control of the
process, and scheduled reviews are conducted with defined deliverables
and checklists for each milestone. At each milestone, a vigorous
review of the plans for the product's development and manufacturing
and risk-reduction efforts highlights issues before they become
problems. The firm's goal is to have mature processes by production.
To achieve this, it considers manufacturing readiness throughout. Each
project's team is cross-functional and includes senior management, mid-
management and the project team. This robust review process leverages
expertise across GE Aviation, reduces risk, and highlights issues
before they become problems.
As with all the commercial firms we visited, GE Aviation requires
strong management involvement at each gate, along with decision
reviews to determine if enough knowledge is available and risk-
mitigation plans are in place to proceed or if actions to address and
mitigate manufacturing risks can show a viable way forward. This
allows management to resolve problems rather than pass them on to the
next phase. At project start, which corresponds to MRL 4, the senior
leadership team and product leadership team generate the product idea
and assess the need for the project. They provide linkage between the
business strategy and the project and develop the high-level project
strategy. They identify any new product material or manufacturing
processes and begin to develop a risk-reduction strategy for these
issues. By the time the product enters the preliminary design phase,
senior leadership and project teams agree on the approach to the
project. At this time, product directors must have a manufacturing
plan in place in order to identify how they are going to achieve
manufacturing readiness. Technical risks are identified in the
manufacturing plan, as well as risk-abatement strategies for materials
and manufacturing processes and supply-chain risks. The plan has to
show how issues will be successfully addressed by the detailed design
phase, when leadership, the project team, and customers agree on the
product to be delivered. If agreement is reached, they freeze the
project plan and a decision is made to fund or terminate the project.
Multidisciplinary Team/Manufacturing Experts:
In the commercial firms we visited, product-development teams were
multidisciplinary, generally including management, manufacturing,
quality, finance, suppliers, and engineering, with necessary skills
available to assess manufacturing readiness. Leading firms recognize
the value of having a knowledgeable, well-trained, and skilled
manufacturing engineering workforce involved in these
multidisciplinary teams from the beginning and throughout the process.
When Honeywell reorganized its aerospace business in 2005, it created
an advanced manufacturing engineering organization to focus on
manufacturing concerns in the earliest phases of new product-
development programs. This organization consists of engineers to
support various manufacturing disciplines in Honeywell. An important
part of this advanced engineering organization is its technology
group, which consists of a select number of technology fellows with
extensive expertise in key manufacturing disciplines that touch nearly
all the products Honeywell produces. Honeywell retains highly skilled
manufacturing expertise through this program and uses these
experienced and knowledgeable manufacturing engineers to oversee each
project's manufacturing assessments.
Maturing Technology and Manufacturing Processes:
Commercial firms focus on maturing and validating technology and
manufacturing processes before these are associated with a product and
before entry into the gated process. They keep invention and unproven
technologies in the technology base until their producibility at the
scale needed can be proven. As an example, GE Healthcare's Gemstone
scintillator underwent years of laboratory development on a small
scale until GE Healthcare was satisfied that this material was ready
to be used on its computed tomography (CT) scanners. Scintillators
work by converting the X-rays in the CT scanner into visible light. GE
Healthcare had been manufacturing its own scintillators since the late
1980s, but it needed an improved one that worked faster, for better
clarity of vision and to reduce the amount of exposure to radiation.
In 2001, the firm began basic composition development at the
laboratory scale and narrowed down the alternatives to find the
material with the best properties for this use. Even at this early
stage, several years before the material would enter into GE
Healthcare's gated process, there was early engagement by the chemists
with the manufacturing side. Before they decided on a solution, a
determination was made that it could produce them with sufficient
yield and quality: even if a material had the best optical qualities,
it had to balance this with its producibility. GE Healthcare tested
thousands of alternatives to determine what could meet its technical
requirements and be producible in the quantities needed. The firm
narrowed it down to a garnet-based, rare-earth minerals composite, and
began producing it in small but increasing quantities. After narrowing
the field to this garnet-based compound, GE Healthcare began to
determine its suppliers and what equipment was needed. The firm then
began building its first pilot plant to produce the material and the
scintillators, 2 years before the scintillator entered the firm's
gated process. Figure 4 shows a photo of a CT scanner that uses the
scintillator technology.
Figure 4: GE CT Scanner Using Advanced Scintillator Material:
[Refer to PDF for image: photograph]
GE Healthcare matured its CT Scanner material years before project
start to validate the material's producibility.
Source: Copyright © General Electric Company USA. All rights reserved.
[End of figure]
Best Practice: Commercial Firms Have Adopted DOD's MRLs or Are
Employing Similar Criteria in Their Product-Development Process:
Because leading commercial firms focus on producibility as a key
element to successfully develop products, they use rigorous analysis
methods to assess producibility and to identify and manage
manufacturing risks and gaps. They apply these methods and tools early
and throughout product development and use them to manage their
product development on a daily basis. This commercial approach is a
process in which quality is designed into a product and manufacturing
processes are brought into statistical control to reduce defects, in
contrast to practices employed by many defense contractors where
problems are identified and corrected after a product is produced.
Some firms were familiar with the DOD MRL proposal and had taken steps
to use the concepts at their own companies. Honeywell, for example,
determined that early decisions were responsible for many production
issues and so they developed analytical tools and models that support
evaluations of manufacturing and risk throughout the product-
development life cycle. In 2005, Honeywell engineers began looking for
a way to measure manufacturing readiness and producibility, since they
realized that early program decisions were driving many production
issues and that by the time a product entered engineering and
manufacturing development, it was too late to efficiently affect these
issues. Some of these issues include cost overruns, quality problems,
low-yield issues, service and maintainability inefficiencies, and
supply-chain problems.
A literature search led them to DOD's MRLs and they realized that
these could provide the type of metric needed for a quantitative
assessment. Honeywell then evolved its own criteria from these MRLs,
modified to meet Honeywell's needs and expanded to address concerns
such as design, obsolescence, and testability issues. Their MRL
Maturity Model assessment tool, which evolved from an early version of
DOD's MRLs, is the main tool in the assessment and is built upon three
enabling producibility analysis tools. The model provides an MRL score
for the product "as is," which is then compared to the MRL score
desired to exit the phase. This model gives the firm a systematic way
to be sure all the information is considered and the right questions
are asked by less-experienced engineers who support the program. This
MRL tool was developed 5 years ago and has evolved in an iterative,
continuous improvement process since then, based on feedback from its
users. Figure 5 shows a simplified depiction of this MRL model and the
three enabling tools.
Figure 5: Honeywell Uses Three Producibility Models and MRL Workshop:
[Refer to PDF for image: illustration]
Manufacturing Complexity Model:
* Identifies design attributes driving manufacturing complexity.
Yield Prediction Model:
* Quantifies anticipated yield of proposed design concepts.
Design for Manufacturing Scorecard Analysis:
* Quantifies the impact of design for manufacturing violations.
MRL Workshop:
* Review manufacturing maturity artifacts against evaluation criteria
to gain consensus on ratings for each category.
The above combine to yield:
MRL Assessment Tool:
* Built on inputs from three producibility analysis tools and MRL
workshop to evaluate manufacturing maturity and identify gaps.
MRL Maturity Model Output:
* MRL score.
Source: GAO analysis of Honeywell data.
[End of figure]
The output of this tool is an MRL assessment score that can identify
gaps or risks. For example, spreadsheets show the MRL scoring at a
glance for each of the elements evaluated, pinpointing the gaps; risk
worksheets to quantify the risks; and action plans to close the gaps
and mitigate these risks. It links to the firm's gated process,
providing entry and exit criteria and feedback on how to meet these
criteria. The important information obtained is not necessarily what
MRL level the item is at currently, but rather the robustness of the
gap-closure plan to get to the desired level for the next gate. The
application of the MRL tool helps identify what these key gaps are and
what steps are required to close them.
The three enabling producibility tools that provide support for this
assessment and early input on the producibility risks are: a Design
for Manufacturing Model, a Product Complexity Model and a Yield
Prediction Model:
* Manufacturing Complexity Model: This model identifies the design
features that are driving manufacturing complexity into the design and
enables scenarios to be evaluated to see what actions can be taken to
simplify the design. Higher-complexity designs generally cost more and
are higher risk, so the goal is to identify alternative design
solutions that minimize complexity, but still meet all the performance
requirements.
* Yield Prediction Model: Honeywell has also developed yield
prediction models based on statistical principles that correlate
opportunities for defects in a design to established process
capability benchmarks. This approach is used to predict yield during
early design activities based on knowledge of the manufacturing
processes used and the complexity of the design.
* Design for Manufacturing Scorecard analysis: The third Honeywell-
developed tool is a design for manufacturing scorecard, which
quantifies how well the design adheres to recommended best practices.
The goal of using the tool is to provide feedback to the designers so
that they see how their design decisions directly affect producibility
and help pinpoint improvement areas early in the process.
Honeywell then conducts an MRL workshop, with a team led by an
engineer from its Advanced Manufacturing Engineering group that
includes the program manager and various subject-matter experts. This
team reviews the tools and the MRL criteria to gain consensus on
ratings for each category. Honeywell's Manufacturing Maturity Model,
with input from these enabling tools, is used to develop an MRL score
for the product. These assessments provide early producibility
evaluations essential to mitigating design-driven risks. Since many
producibility issues are driven by early design architecture
decisions, these tools provide a way to analyze these decisions early
and make the necessary performance and producibility trades through
"virtual prototyping" long before actual hardware is built. The MRL
score provides the necessary framework to ask the questions that such
an analysis needs to answer.
After the MRL assessment is complete and the MRL scores and risk-
mitigation plans are approved, the MRL analysis and risk mitigations
are incorporated into the daily schedule of the program office. The
office continually monitors the MRL levels, updating them and working
toward its risk mitigation goals.
Best Practice: Leading Firms Prove Out Manufacturing Tooling,
Equipment, and Processes before Entry into Production:
Companies we visited spent years prior to production developing and
proving out their manufacturing processes, including building test
articles on pilot production facilities to perfect these processes.
This allowed them to perfect and validate these processes, eliminate
waste and scale up gradually to the required manufacturing level. They
reduce errors and inefficiencies with the purpose of retiring
manufacturing risks.
GE Aviation officials told us that certain advanced manufacturing
technologies achieve significant cost savings by getting the costs
lower earlier in the process and decreasing cycle time for faster
implementation. An example of manufacturing techniques or processes
that have made a big difference in costs, accuracy, and reliability
include processes for drilling small shaped holes for turbine airfoils.
GE Aviation's Turbine Airfoils Lean Lab provides a mock-up of a
production facility or process, where such technologies and production
processes can be tested to eliminate waste, scrap, and excess steps.
They focus on one part family or process, such as the turbine airfoil
shaped-hole manufacturing. The turbine airfoil is a part of the jet
engine that generates power--it extracts horsepower from the high-
temperature, high-speed combusted gasses. Turbine airfoil blades
require hundreds of cooling holes that help maintain part integrity at
elevated operating temperatures. Traditionally, round holes were used,
but the technology has evolved to compound-angle-shaped holes, which
improve cooling effectiveness and reduce engine stress. These type of
holes cannot be economically produced by traditional methods and
require improved manufacturing techniques. Advanced laser drilling was
determined to be feasible, and GE Aviation decided to initiate the
program through the Lean Lab to ensure manufacturing readiness of the
process.
GE Aviation officials compared their processes in this case to DOD's
MRLs. Prior to entering their gated process, they began making
investments in potential technologies, including tooling (MRL 1-3). As
the gated process began, risks were identified and risk-abatement
plans were put in place (MRL 4). GE Aviation then set up the Lean Lab
to test the way the airfoil would actually be built. New processes
were introduced that included new laser methods for hole drilling,
improved robotic technology, machining, and grinding (MRL 5-6). The
managers then ran the pilot production line for some time to
manufacture these airfoils using actual production operators to be
confident that the process would translate to the production line.
Adjustments were made to improve efficiency and retested on the line
until they were satisfied that they had worked out the best
procedures. GE had tooling-design experts on the team at the Lean Lab
to provide rapid part and tool manufacturing. Processes were brought
into statistical control in order to take the complexity out of
manufacturing, simplify the process, and reduce waste (MRL 7-8). They
then dismantled the production line at the Lean Lab, took it to the
manufacturing facility, and set it up exactly the same, with no
variations allowed (MRL 9). This seamless introduction of the new
manufacturing technology and the lean principles developed in the lab
are expected to save many millions of dollars across GE Aviation, on
production of this part family alone. Figure 6 shows a photo of GE
Aviation's Lean Lab setup.
Figure 6: GE Aviation's Turbine Airfoils Lean Lab Proves Out
Production Processes:
[Refer to PDF for image: photograph]
GE Aviation's Turbine Airfoils Lean Lab provided a seamless way to
introduce new manufacturing processes.
Source: Copyright © General Electric Company USA. All rights reserved.
[End of figure]
GE Healthcare provides another example of proving out manufacturing
processes prior to production in their development of the Gemstone
scintillator for use on their CT Scanners. In 2003, the technology for
this transitioned into the firm's formal gated process or product
start-up, and it began a detailed and extensive development of the
manufacturing process. The firm built a pilot plant for this purpose
and began manufacturing the composite in increasing amounts. In this
first pilot plant, it was able to process the materials in increased
quantities from what it produced in the lab. GE Healthcare verified
that it had the right technologies to minimize manufacturing risks. In
the laboratory environment, the firm had already answered the question
"Can this composite be made with the desired properties?" and now
asked "Can it be made with sufficient yield and quality to be
manufactured in the desired amounts?" This early engagement with
manufacturing enabled the firm to develop the process and reduce
errors and inefficiencies with the purpose of reducing manufacturing
risks.
GE Healthcare then built a second pilot production plant that further
increased the amount produced above that of the first pilot plant. The
firm continued its focus on gaining knowledge early, but on a larger
scale: building the pilot plants was important to perfecting the
process and gaining knowledge about the material's producibility. At
this stage, which coincides with MRL 8, it eliminated most of the
technical risks involved in manufacturing the material. The firm then
began to build its full-scale facility, which was ready 18 months
before product launch.
When the full-scale production facility was completed, further scale-
up of the material's manufacturing became the focus. Changes to the
design were made as needed to facilitate this. Any remaining
manufacturing risks were eliminated prior to entry into the next
stage, the product-validation stage. The Food and Drug Administration
requires validation of finished medical devices. GE Healthcare told us
that this means that all the equipment, processes, procedures, and
factory workers are the same as will be used in actual production.
Through use of the pilot plants to perfect the manufacturing of the
scintillator material, GE Healthcare was able to produce production-
representative material to satisfy this requirement.
Best Practice: Commercial Firms Work Closely with Suppliers, Who Must
Meet High Quality Standards for Parts and Supplies:
Commercial firms focus on developing strong relationships with their
suppliers to ensure quality parts are provided in a timely manner.
This begins with rigorous supplier-selection criteria to create a
strong supplier base to provide quality parts. Similarly, DOD's MRL
supply-chain thread focuses on supplier capability throughout the
acquisition life cycle, from as early as pre-milestone A (MRL 3),
where initial assessment of the supply chain begins, through MRL 5,
where supply-chain sources have been identified, and continuing to MRL
8, where the supply-chain should be stable and adequate to support low-
rate production. Commercial firms generally have long-term
relationships with these suppliers and can identify the supplier that
is the best source of material or parts early, well before production
begins. Leading commercial firms apply the same standards to these
suppliers as they apply to their own manufacturing processes, such as
ISO 9000[Footnote 21] or other quality standards. Throughout product
development and production, they establish effective communications
with their suppliers so they can continually assess their performance.
These firms work closely with their suppliers to retain these
beneficial relationships, providing training where necessary and
assistance if manufacturing problems arise.
GE Healthcare suppliers have to be validated before production begins,
but qualifying them starts in the design phase. Suppliers are expected
to meet the ISO 9000 standards and the Food and Drug Administration's
medical devices standards, but GE Healthcare's own standards are more
stringent that those. The supplier-qualification process ensures that
suppliers meet GE Healthcare's requirements, have a quality system
that provides the appropriate controls for the part provided and meet
regulations and requirements of multiple agencies, such as the Food
and Drug Administration. Once a supplier is qualified, it becomes an
approved supplier.
GE Healthcare also audits most of its suppliers and looks for issues
such as lapsed ISO 9000 certification or a failed review. If it finds
these things, GE Healthcare will ask the supplier for a plan to
correct the deficiency and reaudit the supplier. GE Healthcare does
annual risk assessments on the suppliers, based on data gathered
during these audits, with sole-source or single-source suppliers being
a high risk. If a supplier falls out of qualified status, GE
Healthcare will do more frequent assessments. It constantly monitors
the suppliers for quality. It helps the supplier get to the quality
needed, but quality goals must be met.
Siemens is a global company that employs about 70,000 people in the
United States. We visited Siemens Mobility Division, which builds
light rail cars for public transit. Siemens places special emphasis on
its supplier relationships, since it knows its suppliers can contract
to other rail-car builders, as there is competition for suppliers in
this market. If it has a good relationship with its suppliers, it can
continue to benefit from the relationships with high-quality
suppliers. Once it qualifies a supplier, it takes the responsibility
for keeping the supplier qualified, providing technical assistance if
necessary to keep the supplier in its pipeline. Even as early as the
bid phase of the contract, Siemens knows who it will need as suppliers
and if any particular supplier is new or challenged in some respect.
Siemens applies a three-step supplier-qualification process to its
suppliers. This starts with a supplier self-assessment. The firm's
supplier-qualification personnel then visit the supplier's plant and
evaluate the supplier on the same self-assessment form, to determine
if the supplier will make it to the vendor-qualification list. Once a
supplier is on the approved vendor-qualification list, Siemens does
risk ratings for these vendors to be sure it can keep them on the
qualified-vendor list. The firm updates these assessments if the
vendor situation changes, rating the vendor at low risk if it is fully
qualified and working with it if some aspects are not qualified.
Siemens takes responsibility for keeping the approved suppliers
qualified, since finding and qualifying new vendors can be time-
consuming and risky. It tries not to overload any one supplier,
because some of their suppliers are small or specialty operations, so
it keeps a pool of qualified suppliers for as many parts or materials
as it can.
Commercial Firms Require That Manufacturing Processes Be in Control
Earlier Than DOD's MRLs:
Although the firms we visited used manufacturing readiness criteria
similar to DOD's proposed MRLs, one important difference we observed
is that the commercial best practice is to have manufacturing
processes in control prior to the production decision, while DOD's
MRLs require manufacturing processes and procedures to be established
and controlled during MRL 9, which occurs after the milestone C
production decision, which authorizes a program to enter low-rate
initial production, or equivalent.[Footnote 22] Although DOD's MRLs
incorporate many of the commercial manufacturing best practices into
their manufacturing design and implementation criteria, the process
controls criteria would be met too late in the process to achieve
their full effect. DOD's MRL matrix states that low-rate production
yield and rate targets should be achieved at MRL 9, after the
production decision has been made. The commercial firms we talked to
emphasized that production processes must be in control before this
decision is made. They realize that they are unable to make
predictions about production performance until the process is stable
and defects are predictable. Not achieving process control could
result in low quality, extensive rework and waste, and not meeting
cost and schedule targets. Firms established pilot lines to prove out
production material, processes, and tooling, and worked to get
processes under control before the system could move from the pilot
line to production line. Figure 7 shows a depiction of the commercial
manufacturing process approach.
Figure 7: Leading Commercial Firms Use Statistical Controls to Ensure
Quality Products:
[Refer to PDF for image: illustration]
Input:
* Raw material, components, sub-components.
Controllable inputs:
* Temperature.
Uncontrollable inputs:
* Material properties.
The above combine to form: Process.
From Process:
Quality characteristics.
Lead to:
Measure Process Control:
Production decision.
Source: GAO analysis of commercial firm data.
[End of figure]
The companies we visited used various approaches to build process
capability and provide timely information on whether manufactured
components, subsystems, or systems meet design specification. For
example, GE Aviation uses a statistical measurement, called Z sigma
level, to determine whether its processes have been brought under
control or if variations in its manufacturing process could affect the
quality of the product. The product is not moved into production until
the firm is satisfied that these processes are in control. Similarly,
GE Healthcare's milestone process requires that a set of quality
targets are part of the program and that those quality targets are
met. Measures of process control vary from company to company, such as
using yield or scrap and rework rates or sigma levels, but each looks
carefully at those measures to ensure they carried no product-quality
risk and uses this information to determine if the product is ready to
be manufactured.
Two Successful DOD Programs Used Criteria Similar to Commercial Firms:
Two DOD programs, the Army's Lakota aircraft and the Missile Defense
Agency's Standard Missile 3 Block 1A, that had successful
manufacturing outcomes employed some of the same practices as leading
commercial firms. Both used a type of manufacturing readiness criteria
to evaluate whether the programs were ready to enter into production
and both programs focused on manufacturability as a key indicator of
program success, using well-developed technology and a conservative
approach in design and development.
The Lakota aircraft, a light utility helicopter that conducts
noncombat missions, was a mature aircraft design when the Army entered
into the contract with the European Aeronautic Defence and Space
Company to purchase this commercially available helicopter. The
program shows how careful attention to manufacturing readiness can
reduce program risks. According to program office officials, the
contractor was chosen in part because of its manufacturing track
record, and it completed extensive planning, both internally and with
its supplier base, to ensure on-time and reliable deliveries.
Production planning and preparation were accomplished, including
assessments of the manufacturing processes, capabilities, and
facilities. These assessments determined that the program was low risk
and ready for full-rate production. The Lakota is currently in full-
rate production and has met its cost and schedule targets.
The Standard Missile 3 is a ship-based, antiballistic missile used by
the Aegis ballistic missile defense system. Similar to the Lakota, the
system met its cost and schedule goals by using an incremental, low-
risk approach. Like the commercial firms we visited, the program built
knowledge through the use of a type of manufacturing readiness
criteria, which allowed the early identification of risk and
implementation of mitigation strategies. The Standard Missile 3 Block
IA was also on target for manufacturing cost and schedule and reported
a lower cost per unit than was originally estimated on its production
buys. As in the successful commercial firms we visited, manufacturing
issues were considered very early in the design phase, leading to
minimal changes in the program from flight test to production.
MRLs Are Hampered by Lack of an Agencywide Policy and Manufacturing
Workforce Concerns:
While acceptance of MRLs is growing within DOD and the defense
industry, the services' leadership appears to be resistant, and
adoption efforts have been slow. For example, obtaining agreement on a
policy that would institutionalize MRLs defensewide has proven
difficult. Concerns raised by the military-service policymakers have
centered on when and how the MRL assessments would be used. Officials
responsible for the draft policy have promoted MRLs as an initiative
that can address the manufacturing element in the design and
production of weapon systems, citing commercial best practices that
employ similar methods, and benefits derived from pilot programs.
While extensive efforts have been made to promote the benefits of MRLs
in support of a revised draft policy, it has taken nearly 2 years to
allay concerns and it has not yet been approved. DOD is likely to face
serious challenges even if an agreement is reached to approve the
policy, however, because the number of DOD's production and
manufacturing career-field employees has diminished, particularly
within the Air Force. Although the services are at the beginning
stages of revitalizing their production and manufacturing workforce,
DOD currently does not have adequate in-house expertise with the
requisite knowledge to assess manufacturing throughout DOD.
Essentially, the military services and Defense Contract Management
Agency have identified knowledge and manpower gaps in their
manufacturing workforce and believe that any initiative deploying MRLs
defensewide could be hampered as a result.
Draft Policy to Institutionalize MRLs Has Proven Difficult, but the
DOD Community Is Starting to See Its Value:
While acceptance of MRLs is growing within DOD and the defense
industry, the Army's, Navy's, and Air Force's leadership appears to be
resistant and adoption efforts have been slow. For example, a July
2008 draft MRL policy memorandum garnered disagreement among the
military-service policymakers. The military services' leadership
agreed that MRLs provide value in the early acquisition phases but
disagreed with the policy's intent to formalize the process. For
example, the MRL policy memorandum stated that on the basis of
analyses by GAO and the Defense Science Board--as well as positive
results on two Air Force pilot programs--that acquisition category I
programs be assessed using the MRL scale. In particular, the draft
policy included provisions that would require programs at milestone B
to be assessed at MRL 6 or beyond for all critical technologies;
programs at milestone C to be assessed at MRL 8 for all critical
technologies; procedures to be coordinated for including assessments
of manufacturing readiness in addition to technology readiness
assessments at milestone B and C; and incorporation of guidance into
training materials and guidebooks on best practices for addressing
manufacturing from the earliest stages of development through
production and sustainment.
In response to the draft policy, each of the military services issued
memorandums in July 2008 to the Under Secretary of Defense
(Acquisition, Technology and Logistics) or the Director, Defense
Research and Engineering, stating they support MRLs and their use
earlier in the acquisition process but they saw limited value in doing
formal assessments prior to milestone C. In general, the services had
concerns on when and how MRL assessments would be used. More
specifically, their concerns included the following:
* Evaluation results that could be used as the basis for go/no go
decisions.
* A growing number of assessments being levied on acquisition programs.
* Resources required to prove out multiple production lines in a
competitive prototyping environment during the technology-development
phase.
Since 2008, officials responsible for the draft policy memorandum have
been working to address concerns raised by the services. According to
the working group, most concerns pointed to a need to clarify how the
information is intended to be used by decision makers at key
milestones, particularly at the earlier milestones. According to the
working group officials we interviewed, the intent is to inform
decision makers with critical information--such as manufacturing risk
and readiness measures, as appropriate to the phase of acquisition--so
that knowledge-based decisions can be made earlier in the process to
influence better outcomes in terms of cost and schedule in the later
acquisition phases. Moreover, they cite that similar methods are
employed by leading commercial firms as a best practice, plus the fact
that MRL pilot programs have already demonstrated significant
benefits. The revised MRL draft policy has not yet been approved.
Officials familiar with the status of the draft policy stated that the
leadership at one of the military services is still opposed to the
idea of standardizing MRLs across DOD, and efforts to get approval
have not yet occurred within the Office of the Director, Defense
Research and Engineering.
DOD experienced similar problems introducing technology readiness
levels. There was opposition to the use of technology readiness
levels, but they became a standard for programs to follow, and the
standard that technologies should be demonstrated in a relevant
environment became a statutory requirement for all major acquisition
programs seeking to enter system development.[Footnote 23] Programs
report benefits from using technology readiness levels.
Some officials believe that MRLs could significantly reduce cost
growth. For example, the Army and Air Force have reported MRLs were a
factor that contributed to benefits of hundreds of millions of dollars
in reduced program costs, improved schedule, and better performance of
products.
MRL Acceptance Is Growing within DOD and Defense Industry:
A number of Army, Air Force, and Missile Defense Agency programs--as
well as defense contractors--have embraced MRLs as the method for
assessing manufacturing maturity, risk, and readiness. For example,
some Army commands have opted to use them on their science and
technology efforts that have manufacturing elements, and have
developed a formal process for identifying them. Similarly, two of
three Air Force product centers under the materiel command--the
Aeronautical Systems Center and the Air Armament Center--have recently
issued local policy that mandate the use of MRLs. For example, in a
policy memorandum by the Aeronautical Systems Center, dated October
13, 2009, all programs are now required to have manufacturing
readiness assessments using MRLs, prior to each major milestone
review. The memorandum acknowledged that the transition to production
has historically been challenging for many programs and that
manufacturing assessments are a key tool to ensure that programs are
ready to begin production. The Missile Defense Agency has included
MRLs as part of their assessment criteria. In addition, senior missile
defense manufacturing personnel have developed and conducted training
on how to conduct these assessments.
Similarly, a number of defense contractors have implemented MRLs as a
discipline for identifying, managing, and communicating manufacturing
risk and readiness. These contractors report a number of benefits
using the MRLs, including reductions in program costs and improved
production schedule. For example, in 2006, Raytheon participated in
pilot MRL program assessments involving the Advanced Medium-Range Air-
to-Air Missile and a portfolio of other programs and concluded the
approach makes good business sense to lower risk. Raytheon claimed
cost reductions of 30 percent or more could be achieved by using MRLs.
Raytheon officials state that the combination of technology and
manufacturing assessment processes changes the culture by driving a
collaborative partnership between programs, design, and manufacturing
engineering earlier in the product-development life cycle where
maturity efforts can have the greatest effect on improving program
affordability and predictability. As a result, Raytheon is deploying
MRLs as a standard across the organization. Lockheed Martin is
exploring ways to integrate MRLs within its existing review processes.
As previously discussed, Honeywell adopted MRLs for use on both its
defense and commercial products, and developed several models as an
analysis-based approach to quantify their producibility risks.
Manufacturing Workforce Knowledge and Manpower Gaps May Impede
Implementation of MRLs:
The services are in the beginning stages of revitalizing their
manufacturing workforce, largely in response to a February 2006
Defense Science Board task force report on "The Manufacturing
Technology Program: A Key to Affordably Equipping the Future Force."
The report acknowledged that both the manufacturing expertise in the
workforce and program funding have declined, thus eliminating much of
the engineering and manufacturing talent across DOD and the industrial
base. The report concluded that what was once a promising career field
in the military services--with promotion paths, training, and
professional development--has been systematically eliminated over the
past few decades. Table 4 shows the decrease in the manufacturing
career field across DOD from 2001 to 2007.
Table 4: Percent of Manufacturing Workforce Decrease from 2001 to 2007:
Army:
2001: 2,427;
2002: 2,333;
2003: 2,215;
2004: 2,226;
2005: 2,287;
2006: 2,193;
2007: 2,083;
Percent reduction: 14*.
Navy:
2001: 1,997;
2002: 2,297;
2003: 2,259;
2004: 2,232;
2005: 2,032;
2006: 2,000;
2007: 1,960;
Percent reduction:
