Best Practices
Using A Knowledge-Based Approach To Improve Weapon Acquisition Gao ID: GAO-04-386SP January 1, 2004At the request of Congress, we have been examining ways the Department of Defense (DOD) can optimize its investment in weapons systems, drawing on lessons learned from the best, mostly commercial, product development efforts. Leading commercial firms we have studied have developed increasingly sophisticated products in less time and at lower cost. Key to their success is their knowledge-based approach to the acquisition of new products. A knowledge-based approach is supported by incentives that encourage realism and candor. This booklet highlights the result of our work to date.
GAO-04-386SP, Best Practices: Using A Knowledge-Based Approach To Improve Weapon Acquisition
This is the accessible text file for GAO report number GAO-04-386SP
entitled 'Best Practices: Using A Knowledge-Based Approach To Improve
Weapon Acquisition' which was released on January 30, 2004.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
GAO-04-386SP:
Best Practices:
Using A Knowledge-Based Approach To Improve Weapon Acquisition:
This booklet describes the knowledge-based approach successful
organizations generally follow to deliver sophisticated products in
less time and at lower costs.
The Department of Defense (DOD) is on the threshold of several major
investments in acquisition programs that are likely to dominate budget
and doctrinal debates well into the next decade. Over the next 5 years
alone, DOD‘s overall investments are expected to average $150 billion a
year as DOD works to keep legacy systems as well as transform our
national defense capabilities for the future. To meet this challenge,
it is essential that sound foundations for investments in systems be
laid now so that the resulting programs can be executed within
estimates of available resources.
At the request of the Congress, we have been examining ways DOD can
optimize its investment in weapon systems, drawing on lessons learned
from the best, mostly commercial, product development efforts. Leading
commercial firms we have studied have developed increasingly
sophisticated products in less time and at lower cost. Key to their
success is their knowledge-based approach to the acquisition of new
products. A knowledge-based approach is supported by incentives that
encourage realism and candor.
This booklet highlights the results of our work to date. We continue to
explore additional facets of the acquisition process to identify best
practices. More details on our work can be found in the reports cited
in this brochure.
GAO‘s best practices work is done under the direction of Katherine V.
Schinasi. For more on this work, go to [hyperlink,
http://www.gao.gov/bestpractices]. If you have questions or would like
to discuss our reviews, please contact Ms. Schinasi at (202) 512-4841
or schinasik@gao.gov.
Signed by:
Jack L. Brock Jr.
Managing Director:
Acquisition and Sourcing Management:
[End of section]
A Knowledge-Based Approach Puts Acquisition Programs In A Better
Position To Succeed:
Leading commercial firms expect that their managers will deliver high
quality products on time and within budgets. Doing otherwise could
result in losing a customer in the short term and losing the company in
the long term. Thus, these firms have adopted practices that put their
individual program in a good position to succeed in meeting these
expectations on individual products. Collectively, these practices
ensure that a high level of knowledge exists about critical facets of
the product at key junctures during its development and is used to
deliver capability as promised. DOD has recognized the need to adopt
ways of doing business that enable it to achieve similar results for
its weapons acquisitions.
Our reviews have shown that there are three critical junctures at which
firms must have knowledge to make large investment decisions. First,
before a product development is started, a match must be made between
the customers‘ needs and the available resources”technical and
engineering knowledge, time, and funding. Second, a product‘s design
must demonstrate its ability to meet performance requirements and be
stable about midway through development. Third, the developer must show
that the product can be manufactured within cost, schedule, and quality
targets and is demonstrated to be reliable before production begins. If
the knowledge attained at each juncture does not confirm the business
case on which the acquisition was originally justified, the program
does not go forward.
In applying the knowledge-based approach, the most-leveraged decision
point of the three junctures is matching the customer‘s needs with the
developer‘s resources. This initial decision sets the stage for the
eventual outcome”desirable or problematic. The match is ultimately
achieved in every development program, but in successful development
programs, it occurs before product development. In successful programs,
negotiations and tradeoffs occur before product development is started
to ensure that a match exists between customer expectations and
developer resources. Leading firms thus make an important distinction
between technology development and product development. Technologies
that are not mature continue to be developed in the technology
base”they are not included in a product development.
With achievable requirements and commitment of sufficient investment to
complete the development, programs are better able to deliver products
at cost and on schedule. When knowledge lags, a number of risks are
introduced into the acquisition process that can result in cost
overruns, schedule delays, and inconsistent product performance.
An approach that enables organizations to achieve a match between needs
and resources is evolutionary product development. Under this approach,
basic requirements are achieved first, with additional capabilities
planned for future generations of the product. Because product
development is incremental, achieving knowledge is more manageable.
Commercial companies have found that trying to capture the knowledge
needed to stabilize the design of a product with considerable new
technical content is an unwieldy task”especially if the goal is to
reduce development cycle times and get the product to the marketplace
as quickly as possible. With evolutionary development, a product‘s
design uses only components and subsystems whose reliability has been
proven through past use or testing.
Knowledge Points at a Glance:
The knowledge-based process followed by leading organizations is
highlighted and further discussed below.
Figure: Knowledge Points at a Glance:
[See PDF for image]
This figure is an illustration of knowledge points, as follows:
Technology Development: Leads to:
Product Development: Program launch and Knowledge Point 1 occur at the
beginning of product development; Knowledge point 2 occurs in the mid-
point of product development, which leads to:
Production: Knowledge point 3 occurs at the beginning of production.
* Knowledge point 1: Resources and needs match. Knowledge point 1
occurs when a sound business case is made for the product”that is, a
match is made between the customer‘s requirements and the product
developer‘s available resources in terms of knowledge, time, and money.
To determine their available resources, successful companies rely on
current and valid information from predecessor programs, new
technologies that have demonstrated a high level of maturity, system
engineering data, and experienced people. Successful companies also
communicate extensively with customers to match their wants and needs
with the firm‘s available resources and with its ability to manufacture
an appropriate product.
* Knowledge point 2: Product design is stable. Knowledge point 2 occurs
when a company determines that a product‘s design is stable”that is, it
will meet customer requirements and cost and schedule targets. A best
practice is to achieve design stability at the product‘s critical
design review, usually held midway through development. In DOD, the
critical design review occurs when the first phase of product
development”product integration”has been completed and the second
phase”product demonstration”is about to begin. Completion of at least
90 percent of engineering drawings at the critical design review
provides tangible evidence that the design is stable.
* Knowledge point 3: Production processes are mature. This level of
knowledge is achieved when it has been demonstrated that the product can
be manufactured within cost, schedule, and quality targets. A best
practice is to ensure that all key manufacturing processes are in
statistical control”that is, they are repeatable, sustainable, and
capable of consistently producing parts within the product‘s quality
tolerances and standards”at the start of production. It is important
that the product‘s reliability be demonstrated before production
begins, as investments can increase significantly if defective parts
need to be repaired or reworked.
[End of figure]
Measuring Success at Key Knowledge Points:
The organizations we have studied apply a variety of tools and measures
to gauge the level of knowledge that they have attained before making
major investment decisions.
* Measuring technology readiness:
Many programs now use an analytical tool”technology readiness levels
(TRL)”that can assess the maturity level of technology as well as the
risk that maturity poses if the technology is included in a product
development. The experiences of the DOD and commercial technology
development cases we reviewed indicate that demonstrating a high level
of maturity before allowing new technologies into product development
programs puts those programs in a better position to succeed. Simply
put, the more mature technology is at the start of the program, the
more likely the program will succeed in meeting its objectives.
There are nine TRLs, each denoting a level of demonstrated performance,
beginning with concept”the lowest level of readiness”to application,
where the technology has been ’flight proven“ under mission conditions.
The higher the TRL, the smaller the gap between the technology‘s
maturity and the product‘s requirements, and the lower the risk of
including the technology in the product‘s development. Our best
practices work has shown that TRL 7”demonstration of a technology in an
operational environment”is the level of technology needed to minimize
risks when launching an acquisition program.
Using TRLs to assess a technology‘s maturity helps decisionmakers make
informed choices about product development. If a technology lacks
maturity, decisionmakers can choose to delay product development until
the technology has matured sufficiently or lower the product‘s
requirements so that a less advanced but proven technology can be used.
The more a technology has been proven, the more likely the product is to
meet its objectives and the more likely the company can minimize
unexpected problems and avoid potential schedule delays and cost
overruns.
* Measuring design readiness:
The organizations we studied also understood the importance of having
disciplined design reviews and getting agreement from the stakeholders
that the product‘s design had been demonstrated to meet requirements
before beginning initial manufacturing. Each organization had a design
review process that began at the component level, continued through the
subsystem level, and culminated with a critical design review of the
integrated system to determine if the product was ready to progress to
the next phase of development.
In addition to design engineers, a cross-functional team of
stakeholders in the process included key suppliers, manufacturing
representatives, and service and maintenance representatives. From past
experience, leading organizations have discovered that cross-functional
teams provide a complete perspective of the product.
At critical design review, the leading organizations we studied
generally require that at least 90 percent of the engineering drawings
be completed. They consider engineering drawings to be a good measure
of the demonstrated stability of a product‘s design because the
drawings represent the language used by engineers to communicate to the
manufacturers the details of a new product design”what it looks like,
how its components interface, how it functions, how to build it, and
what critical materials and processes are required to fabricate and
test it.
Other Important Tools and Practices:
Our reports have identified other tools and practices that enable
successful implementation of the approach. These include:
* Systems engineering for identifying gaps between requirements and
resources so that they can be reconciled through effective trade-offs
before product development.
* Employment of integrated product teams to bring together in a single
organization the different functions needed to design and manufacture a
product, such as engineering, finance, test and evaluation, and
manufacturing.
* Supplier management approaches that optimize relationships with both
contractors and subcontractors.
* Using earned value management techniques to track a projects progress
and assess its ability to meet cost and scheduling goals.
* The use of a variety of testing and evaluation tools and techniques to
validate a product‘s performance early and throughout development.
* The use of targeted, hands-on methods to ensure that program offices
are trained on key practices.
* Collection of statistical process control data to ensure manufacturing
processes are consistently producing parts within quality standards.
* Considering reasonable operating and support costs and the readiness
or availability of equipment as requirements equal in importance to
other performance characteristics.
Coupling the knowledge-based approach with sound tools and techniques
for oversight, strengthening the workforce, and ensuring the right
knowledge is attained at the right times, increases an organization‘s
potential to meet cost, scheduling, and performance targets. It is
still essential, however, that the right incentives and resources be in
place to encourage program managers to employ such measures.
Following A Knowledge-Based Acquisition Process Is Critical For Weapon
Acquisitions:
The majority of weapon system acquisitions we have reviewed over the
past several decades experienced problems during acquisition that drove
up costs and schedules and increased technical risks. Many programs
have been restructured by DOD in the face of delays and cost growth; a
few have been canceled. We have found that these problems are largely
rooted in the failure to match customer‘s needs with the developer‘s
resources”technical knowledge, timing, and funding--when starting
product development. In other words, commitments were made to
delivering capability without knowing whether technologies being
pursued could really work as intended. Time and costs were consistently
underestimated. Problems that surfaced early cascaded throughout
development and magnified the risks facing the program.
When Knowledge Is Not Attained At Key Junctures:
* Launching an acquisition program before requirements and available
resources are matched can result in a product that fails to perform as
expected, costs more, or takes longer to develop.
* Failure to ensure design stability about halfway through product
development can result in design changes that are more costly to
correct later in development or after the product is fielded.
* Entering production before manufacturing processes are under control
can result in product defects that require additional resources to
rework or scrap”a costly and inefficient practice.
On a number of programs, DOD has shown that it can manage its weapon
system acquisition process to ensure important knowledge about
technology and requirements, design, and manufacturing is captured and
used to make informed and timely decisions before committing to
substantial development and production investments. Moreover, over the
past few years, DOD has made constructive changes to its acquisition
policy to embrace best practices”especially those related to technology
maturity and separating technology development from product
development. If faithfully implemented, DOD‘s policies of evolutionary
acquisition and phased requirements will make it easier to match
resources with needs before starting a new product development. Funding
pressures and the need to modernize systems across the Department will
continue to make it paramount for DOD to adopt practices that can save
money and deliver new capabilities quicker.
However, a number of incentives”many tied to funding”continue to
undermine DOD‘s ability to achieve a match between needs and resources
at the onset of weapon acquisition programs. Unlike the commercial world
where the focus is on delivering a product to market, DOD‘s system
focuses on competing for resources. In the competition for funding,
managers are encouraged to launch product developments before
technologies are mature. Because a proven way to win support for a new
weapons acquisition program is to promote unprecedented performance
features and design characteristics, managers have implicit incentives
to rely on immature technologies. Moreover, because funding is
competitive and DOD‘s forecasts of cost, schedule, and performance are
largely based on immature technologies and other unknowns, estimates
tend to be squeezed into insufficient profiles of available funding.
Other factors, such as short tenures and career pressures, discourage
program managers from saying no to requirements that are later
discovered to have been unreasonable. Thus, to meet the investment
challenge of modernizing its forces, DOD will need not only to
implement policies that embrace evolutionary, knowledge-based
acquisitions, it will need to instill incentives that encourage realism
and candor in the acquisition process and sustain its commitment to
improving business practices.
[End of section]
Best Practices Reports:
These reports can be found on the GAO Web site at [hyperlink,
http://www.gao.gov].
Defense Acquisitions: Assessments of Major Weapon Programs, GAO-03-476,
Washington, D.C.: May 2003.
Best Practices: Setting Requirements Differently Could Reduce Weapon
Systems‘ Total Ownership Costs. GAO-03-57. Washington, D.C.: February
11, 2003.
Best Practices: Capturing Design and Manufacturing Knowledge Early
Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15,
2002.
Defense Acquisitions: DOD Faces Challenges in Implementing Best
Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.
Best Practices: DOD Teaming Practices Not Achieving Potential Results.
GAO-01-510, Washington, D.C.: April 2001.
Best Practices: Better Matching of Needs and Resources Will Lead to
Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8,
2001.
Best Practices: A More Constructive Test Approach Is Key to Better
Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31,
2000.
Defense Acquisitions: Employing Best Practices Can Shape Better Weapon
System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000.
Best Practices: DOD Training Can Do More to Help Weapon System Programs
Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August 16,
1999.
Best Practices: Better Management of Technology Development Can Improve
Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30,
1999.
Best Practices: DOD Can Help Suppliers Contribute More to Weapon System
Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998.
Best Practices: Successful Application to Weapon Acquisition Requires
Changes in DOD‘s Environment. GAO/NSIAD-98-56. Washington, D.C.:
February 24, 1998.
Best Practices: Commercial Quality Assurance Practices Offer
Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26,
1996.
