Defense Acquisitions
DOD's Revised Policy Emphasizes Best Practices, but More Controls Are Needed Gao ID: GAO-04-53 November 10, 2003The Department of Defense's (DOD) investment in new weapon systems is expected to exceed $1 trillion from fiscal years 2003 to 2009. To reduce the risk of cost and schedule overruns, DOD revamped its acquisition policy in May 2003. The policy provides detailed guidance on how weapon systems acquisitions should be managed. The Senate report accompanying the National Defense Authorization Act for Fiscal Year 2004 required GAO to determine whether DOD's policy supports knowledge-based, evolutionary acquisitions and whether the policy provides the necessary controls for DOD to ensure successful outcomes, such as meeting cost and schedule goals. The report also required GAO to assess whether the policy is responsive to certain requirements in the Bob Stump National Defense Authorization Act for Fiscal Year 2003 concerning DOD's management of the acquisition process.
DOD's new policy supports knowledge-based, evolutionary acquisitions by adopting lessons learned from successful commercial companies. One of those lessons is a knowledge-based approach, which requires program managers to attain the right knowledge at critical junctures--also known as knowledge points--so they can make informed investment decisions throughout the acquisition process. The policy also embraces an evolutionary or phased development approach, which sets up a more manageable environment for attaining knowledge. The customer may not get the ultimate capability right away, but the product is available sooner and at a lower cost. Leading firms have used these approaches--which form the backbone of what GAO calls the best practices model--to determine whether a project can be accomplished with the time and money available and to reduce risks before moving a product to the next stage of development. By adopting best practices in the acquisition policy, DOD's leadership has taken a significant step forward. The next step is to provide the necessary controls to ensure a knowledge-based, evolutionary approach. Implementing the necessary controls at all three knowledge points along the acquisition process helps decision makers ensure a knowledge-based approach is followed. Without controls in the form of measurable criteria that decision makers must consider, DOD runs the risk of making decisions based on overly optimistic assumptions. Each successive knowledge point builds on the preceding one, and having clearly established controls helps decision makers gauge progress in meeting goals and ensuring successful outcomes. DOD was responsive to the requirements in the Bob Stump National Defense Authorization Act for Fiscal Year 2003. DOD's responses reflected the committee's specific concerns about the application of certain statutory and regulatory requirements to the new evolutionary acquisition process, for more guidance for implementing spiral development, and about technology readiness at program initiations.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-04-53, Defense Acquisitions: DOD's Revised Policy Emphasizes Best Practices, but More Controls Are Needed
This is the accessible text file for GAO report number GAO-04-53
entitled 'Defense Acquisitions: DOD's Revised Policy Emphasizes Best
Practices, but More Controls Are Needed' which was released on November
10, 2003.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Senate and House Committees on Armed Services:
United States General Accounting Office:
GAO:
November 2003:
Defense Acquisitions:
DOD's Revised Policy Emphasizes Best Practices, but More Controls Are
Needed:
GAO-04-53:
GAO Highlights:
Highlights of GAO-04-53, a report to the Senate and House Committees
on Armed Services
Why GAO Did This Study:
The Department of Defense‘s (DOD) investment in new weapon systems is
expected to exceed $1 trillion from fiscal years 2003 to 2009. To
reduce the risk of cost and schedule overruns, DOD revamped its
acquisition policy in May 2003. The policy provides detailed guidance
on how weapon systems acquisitions should be managed.
The Senate report accompanying the National Defense Authorization Act
for Fiscal Year 2004 required GAO to determine whether DOD‘s policy
supports knowledge-based, evolutionary acquisitions and whether the
policy provides the necessary controls for DOD to ensure successful
outcomes, such as meeting cost and schedule goals.
The report also required GAO to assess whether the policy is
responsive to certain requirements in the Bob Stump National Defense
Authorization Act for Fiscal Year 2003 concerning DOD‘s management of
the acquisition process.
What GAO Found:
DOD‘s new policy supports knowledge-based, evolutionary acquisitions
by adopting lessons learned from successful commercial companies. One
of those lessons is a knowledge-based approach, which requires program
managers to attain the right knowledge at critical junctures”also
known as knowledge points”so they can make informed investment
decisions throughout the acquisition process. The policy also embraces
an evolutionary or phased development approach, which sets up a more
manageable environment for attaining knowledge. The customer may not
get the ultimate capability right away, but the product is available
sooner and at a lower cost. Leading firms have used these approaches”
which form the backbone of what GAO calls the best practices model”to
determine whether a project can be accomplished with the time and
money available and to reduce risks before moving a product to the
next stage of development.
By adopting best practices in the acquisition policy, DOD‘s leadership
has taken a significant step forward. The next step is to provide the
necessary controls to ensure a knowledge-based, evolutionary approach.
Implementing the necessary controls at all three knowledge points
along the acquisition process helps decision makers ensure a knowledge-
based approach is followed. Without controls in the form of measurable
criteria that decision makers must consider, DOD runs the risk of
making decisions based on overly optimistic assumptions. Each
successive knowledge point builds on the preceding one, and having
clearly established controls helps decision makers gauge progress in
meeting goals and ensuring successful outcomes.
DOD was responsive to the requirements in the Bob Stump National
Defense Authorization Act for Fiscal Year 2003. DOD‘s responses
reflected the committee‘s specific concerns about the application of
certain statutory and regulatory requirements to the new evolutionary
acquisition process, for more guidance for implementing spiral
development, and about technology readiness at program initiations.
What GAO Recommends:
GAO recommends that the Secretary of Defense strengthen DOD‘s
acquisition policy by requiring additional controls to ensure decision
makers will follow a knowledge-based, evolutionary approach. DOD
partially concurred with GAO‘s recommendations.
www.gao.gov/cgi-bin/getrpt?GAO-04-53.
To view the full product, including the scope and methodology, click
on the link above. For more information, contact Katherine V. Schinasi
at (202) 512-4841 or schinasik@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
DOD's Revised Policy Provides a Framework for Knowledge-Based,
Evolutionary Acquisitions:
Controls Are Needed to Ensure Knowledge-Based Approach Is Followed:
DOD Is Responsive to Congressional Requirements about New Acquisition
Process:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Scope and Methodology:
Appendix I: Section 802, Evolutionary Acquisition:
Appendix II: Section 803, Spiral Development:
Appendix III: Section 804, Technology Maturity, and Section 822,
Independent Technology Readiness Assessments:
Appendix IV: Comments from the Department of Defense:
Related GAO Products:
Tables:
Table 1: Excerpts of Best Practices Contained in DOD's New Policy:
Table 2: Comparison of Controls Used in Best Practices Model and DOD
Policy:
Table 3: How DOD Policy Addresses Section 802's Statutory and
Regulatory Requirements:
Table 4: How DOD Responded to Section 802's Requirements Regarding
Specific Matters for Each Increment of an Evolutionary Acquisition
Process:
Table 5: How DOD Policy Addresses Section 803's Requirements:
Table 6: How DOD Responded to Section 804 and Section 822 Requirements:
Figures:
Figure 1: Distribution of Development Funds for Fiscal Year 2003:
Figure 2: Comparison of DOD's Framework and Commercial Best Practices
Model:
Abbreviations:
DOD: Department of Defense:
DODDDOD Directive:
DODI: DOD Instruction:
MDA: Milestone Decision Authority:
TEMP: Test and Evaluation Master Plan:
United States General Accounting Office:
Washington, DC 20548:
November 10, 2003:
The Honorable John W. Warner:
Chairman:
The Honorable Carl Levin:
Ranking Minority Member:
Committee on Armed Services:
United States Senate:
The Honorable Duncan Hunter:
Chairman:
The Honorable Ike Skelton:
Ranking Minority Member:
Committee on Armed Services:
House of Representatives:
The Department of Defense (DOD) plans to spend more than $1†trillion
from fiscal years 2003 to 2009 for developing and procuring weapon
systems. To get the most out of its investment, DOD has set goals to
develop and procure weapons faster and at less cost. Guiding this
effort is DODís newly revised acquisition policy. Issued in May 2003,
the policy seeks to foster greater efficiency while building
flexibility in the acquisition process. The policy embraces a
knowledge-based, evolutionary framework that emphasizes shorter
development times.
DOD intends to use its policy to improve its record for meeting cost
and schedule estimates and for delivering capabilities as promised.
Congress has expressed support for DODís efforts to revamp its
acquisition process and has established requirements that DOD must take
to ensure a disciplined approach. Recognizing that a consistent and
disciplined application of policies will be key to achieving desired
outcomes, the Senate Armed Services Committe directed us to assess
DOD's current acquisition policies to determine whether they (1)
support knowledge-based, evolutionary acquisitions, (2) provide the
necessary controls to ensure DOD's policy intent is followed, and (3)
respond to specific requirements in the Bob Stump National Defense
Authorization Act for Fiscal Year 2003 (Public Law 107-314) regarding
DOD plans to manage its new acquisition process.[Footnote 1]e:
Results in Brief:
DOD has made major improvements to its acquisition policy by adopting
knowledge-based, evolutionary practices used by successful commercial
companies. If properly applied, these best practices can put DOD's
decision makers in a better position to deliver high-quality products
on time and within budget. The policy requires decision makers to have
the knowledge they need before moving to the next phase of development.
To ensure that the acquisition environment is conducive to a knowledge-
based approach, the policy embraces evolutionary development, which
allows managers to develop a product in increments rather than trying
to incorporate all the desired capabilities in the first version that
comes off the production line. Leading companies who use the knowledge-
based evolutionary approach have shown that these best practices help
reduce risks at critical junctures during a productís development and
help ensure that decision makers get the most out of their investments.
DOD's recent policy changes are a significant step forward. The next
step is for DOD to provide the necessary controls to ensure a
knowledge-based, evolutionary approach is followed. For example, the
policy does not establish measures to gauge design and manufacturing
knowledge at critical junctures in the product development process.
Without specific requirements to demonstrate knowledge at key points,
the policy allows significant unknowns to be judged as acceptable
risks, leaving an opening for decision makers to make uninformed
decisions about continuing product development.
DOD was responsive to the requirements in the Defense Authorization Act
for Fiscal Year 2003. DOD's responses reflected the committee's
specific concerns about the application of certain statutory and
regulatory requirements[Footnote 2] to the new evolutionary acquisition
process, for more guidance for implementing spiral development, and
about technology readiness at program initiations.
This report makes recommendations that the Secretary of Defense
strengthen DOD's acquisition policy by requiring additional controls to
ensure decision makers will follow a knowledge-based, evolutionary
approach. DOD partially concurred with our recommendations. DOD
believes the current acquisition framework includes the controls
necessary to achieve effective results, but department officials will
continue to monitor the process to determine whether other controls are
needed to achieve the best possible outcomes. DOD agreed it should
record and justify program decisions for moving from one stage of
development to next but did not agree with the need to issue a report
outside of the department.
Background:
Traditionally, DODís strategy for acquiring major weapon systems has
been to plan programs that would achieve a big leap forward in
capability. However, because the needed technologies often are not yet
mature, programs stay in development for years until the technologies
are demonstrated. As a result, weapon systems have frequentlybeen
characterized by poor cost, schedule, and performance outcomes. This
has slowed modernization efforts, reduced the buying power of the
defense dollar, delayed capabilities for the warfighter, and forced
unplanned--and possibly unnecessary--trade-offs among programs.
Our extensive body of work shows that leading companies use a product
development model that helps reduce risks and increase knowledge when
developing new products. This best practices model enables decision
makers to be reasonably certain about their products at critical
junctures during development and helps them make informed investment
decisions. This knowledge-based process can be broken down into three
cumulative knowledge points.
* Knowledge point 1: A match must be made between the customer's
needs and the developer's available resources--technology, engineering
knowledge, time, and funding--before a program starts.
* Knowledge point 2: The product's design must be stable and must
meet performance requirements before initial manufacturing begins.
* Knowledge point 3: The product must be producible within cost,
schedule, and quality targets and demonstrated to be reliable before
production begins.
To bolster the knowledge-based process, leading companies use
evolutionary product development, an incremental approach that enables
developers to rely more on available resources rather than making
promises about unproven technologies. While the user may not initially
receive the ultimate capability under this approach, the initial
product is available sooner and at a lower, more predictable cost.
Also, leading companies know that invention cannot be scheduled and its
cost is difficult to estimate. They do not bring technology into new
product development unless that technology has been demonstrated to
meet the user's requirements. Allowing technology development to spill
over into product development puts an extra burden on decision makers
and provides a weak foundation for making product development
estimates.
DOD understands that it must improve acquisition process outcomes if
it is to modernize its forces within currently projected resources. To
help achieve this goal, DOD has revised its acquisition policy, called
the 5000 series, to reflect best practices from successful commercial
and DOD programs.[Footnote 3] The policy covers mostóbut not allmajor
acquisitions. The Secretary of Defense has delegated authority to the
Missile Defense Agency and to the National Security Space Team to
develop separate guidance for missile defense and space systems,
respectively.[Footnote 4] Approximately 35 percent of DOD's
development funds in 2003 went to these systems. (Figure 1 shows how
$43.1 billion in development funds were distributed across space,
missile defense, and systems covered by the 5000 series.) This report
addresses policy for the defense programs covered exclusively under the
5000 series.ó:
Figure 1: Distribution of Development Funds for Fiscal Year 2003:
[See PDF for image]
Note: Includes research, development, test, and evaluation budget
activities for advanced component development and prototypes; system
development and demonstration; and operational systems development.
[End of figure]
DOD's Revised Policy Provides a Framework for Knowledge-Based,
Evolutionary Acquisitions:
DOD's leaders have made significant improvements to DOD's acquisition
policy by adopting the knowledge-based, evolutionary approach used by
leading commercial companies. The revised policy has the potential to
transform DODís acquisition process by reducing risks and increasing
the chances for successful outcomes. The policy provides aframework for
developers to ask themselves at key decision points whether they have
the†knowledge they need to move to the next phase of acquisition. If
rigorously applied, this knowledge-based framework can help managers
gain the confidence they need to make significant and sound investment
decisions for major weapon systems. In placing greater emphasis on
evolutionary product development, the policy sets up a more manageable
environment for achieving knowledge. Another best practice reflected in
the policyís framework is separating technology development from
product development, which reduces technological risk at the start of
a program.
As shown in table 1, DODís policy emphasizes best practices used by
leading companies.
Table 1: Excerpts of Best Practices Contained in DOD's New Policy:
Best practices: Knowledge-based acquisition; DOD policy: DOD Directive
5000.1 (enclosure 1, p. 5): Program managers "shall provide knowledge
about key aspects of a system at key points in the acquisition process.
— shall reduce technology risk, demonstrate technologies in a relevant
environment — prior to program initiation. They shall reduce
integration risk and demonstrate product design prior to the design
readiness review. They shall reduce manufacturing risk and demonstrate
producibility prior to full-rate production.".
Best practices: Evolutionary acquisition; DOD policy: DOD Instruction
5000.2 (p. 3): "Evolutionary acquisition is the preferred DOD strategy
for rapid acquisition of mature technology for the user. — The
objective is to balance needs and available capability with resources,
and to put capability into the hands of the user quickly. The success
of the strategy depends on consistent and continuous definition of
requirements, and the maturation of technologies — that provide
increasing capability towards a materiel concept.".
Best practices: Separating technology development from product
development; DOD policy: DOD Instruction 5000.2 (p. 6): "The project
shall exit Technology Development when an affordable increment of
militarily-useful capability has been identified, the technology for
that increment has been demonstrated in a relevant environment, and a
system can be developed for production within a short timeframe
(normally less than five years).".
Source: DOD.
[End of table]
Similar to the best practices model, DOD's policy divides its
acquisition process into phases, as shown in figure 2. Key decisions
are aligned with the three critical junctures of a productís
development, or knowledge points.
Figure 2: Comparison of DOD's Framework and Commercial Best Practices
Model:
[See PDF for image]
[End of figure]
In other similarities, DODís framework pinpoints program start at
milestone B, about the same point as program start on the best
practices model. At the midway point on both approaches, a stable
product design should be demonstrated. With DODís framework, managers
are required to knowóby the time full-rate production decision review
occursówhether the product can be produced within cost, schedule, and
quality targets. This requirement occurs earlier in the best practices
model, before production begins, or at knowledge point 3. Leading
companies have used†this approach to reduce risks and to make costs and
delivery dates more predictable.
Controls Are Needed to Ensure Knowledge-Based Approach Is Followed:
While DOD has strengthened its acquisition policy with a knowledge-
based, evolutionary framework, the policy does not include many of the
same controls that leading companies rely on to attain a high level of
knowledge before making additional significant investments. Controls
are considered effective if they are backed by measurable criteria and
if decision makers are required to consider them before deciding to
advance a program to the next level. Controls used by leading companies
help decision makers gauge progress in meeting cost, schedule, and
performance goals and ensure that managers will (1) conduct activities
to capture relevant product development knowledge, (2) provide evidence
that knowledge was captured, and (3) hold decision reviews to determine
that appropriate knowledge was captured to move to the next phase.
To determine if DOD has the necessary controls, we compared controls in
DOD's policy with those used in the best practices model at three
critical junctures. Table 2 shows the presence or absence of controls
for various versions of DOD policy since 1996, including the May 2003
revision.
Table 2: Comparison of Controls Used in Best Practices Model and DOD
Policy:
[See PDF for image]
Sources: GAO and DOD.
[End of table]
At all three knowledge points, DOD's policy does not provide all the
necessary controls used by commercial companies. For example, at
program launch (milestone B) or when knowledge point 1 should be
reached, the policy requires decision makers to identify and validate a
weapon system's key performance requirements and to have a technical
solution for the system before program start. This information is then
used to form cost and schedule estimates for the product's development.
However, the policy does not emphasize the use of a disciplined systems
engineering process for balancing a customer's needs with resources to
deliver a preliminary design. The lack of effective controls at
knowledge point 1 could result in gaps between requirements and
resources being discovered later in development.
At the design readiness review or when knowledge point 2 should be
reached, DOD's policy does not require specific controls to document
that a product is ready for initial manufacturing and demonstration.
DOD's policy suggests appropriate criteria, such as number of subsystem
and system design reviews completed, percentage of drawings completed,
planned corrective actions to hardware and software deficiencies,
adequate development testing, completed failure modes and effects
analysis, identification of key system characteristics and critical
manufacturing processes, and availability of reliability targets and
growth plans. However, these criteria are not required. For example, we
found that a key indicator of a product's design stability is the
completion of 90 percent of the engineering drawings supported by
design reviews. DOD's policy does not require that a certain percentage
of drawings or design reviews be completed to ensure the design is
mature enough to enter the system demonstration phase. As a result, a
decision maker has no benchmark to consider when deciding to advance a
program to the next level of development.
Finally, at production commitment or when knowledge point 3 should be
reached, DOD's policy does not require specific controls to document
that a product can be manufactured to meet cost, schedule, and quality
targets before moving into production. For example, the policy states
there should be "no significant manufacturing risks" at the start of
low-rate production but does not define what this means or how it is to
be measured. DOD's policy does not require the demonstrated control of
manufacturing processes and the collection of statistical process
control data until full-rate production begins but even then fails to
specify a measurable control. Given that low-rate production can last
several years, a significant number of products can be manufactured
before processes are brought under control, creating a higher
probability of poor cost and schedule outcomes.
DOD Is Responsive to Congressional Requirements about New Acquisition
Process:
While supporting efforts to build more flexibility into the DOD
acquisition process and to develop weapon systems using an evolutionary
approach, Congress asked DOD to be more disciplined in its approach.
The Defense Authorization Act for Fiscal Year 2003 required DOD to
address (1) the way it plans to meet certain statutory and regulatory
requirements for managing its major acquisition programs, (2) needed
guidance for implementing spiral developments, and (3) technology
readiness (at acquisition program initiation). DOD was responsive to
all three requirements. With regard to the second requirement, a
description of the process that would be used to independently validate
that measurable exit criteria for applying a spiral development
process[Footnote 5] have been met was unclear. DOD stated that the
milestone decision authority[Footnote 6] provides that independent
validation as part of DOD's milestone approval process. DOD's responses
to the relevant sections of the act are summarized below. More detailed
comparisons are provided in appendixes I, II, and III.
Section 802: Evolutionary Acquisition:
Requirements: This section directed DOD to report on its plan to meet
certain statutory and regulatory requirements for managing its major
acquisition programs applying an evolutionary acquisition process.
These include establishing and approving operational requirements and
cost and schedule goals for each increment, meeting requirements for
operational and live fire testing for each increment, and optimizing
total system performance and minimizing total ownership costs.
DOD response: In April 2003, DOD submitted its report reflecting how
these requirements are addressed in its acquisition policy. According
to the report, the policy addresses the statutory and other
requirements applicable to all major defense acquisition programs,
including each increment of evolutionary acquisition programs. For
example, the policy requires that each program or increment of an
evolutionary acquisition have a milestone B decision to approve program
initiation and to permit entry into system development and
demonstration. The policy specifies the statutory and regulatory
information necessary to support the decision.
Section 803: Spiral Development:
Requirements: This section authorizes DOD to conduct a research and
development program for a major defense acquisition program using
spiral development only if approved by the Secretary of Defense or
authorized high-level designee.[Footnote 7] A program cannot be
conducted as a spiral development unless the Secretary of Defense or
designee approves a plan that describes such things as the program
strategy, test plans, performance parameters, and measurable exit
criteria. The section also requires the Secretary of Defense to issue
guidance addressing the appropriate processes for an independent
validation that exit criteria have been met, the operational assessment
of fieldable prototypes, and the management of these types of programs.
It further requires the Secretary to report to Congress on the status
of each program applying spiral development by September 30 of each
year from 2003 to 2008.
DOD response: DOD established a technology development strategy in the
new policy to address this requirement. The strategy must be completed
before a program can enter the technology development phase. The
strategy also documents the cost and schedule goals, the test plans,
the number of prototypes, and a program strategy for the total research
and development program. The strategy requires a test plan to ensure
the goals and exit criteria for the first technology spiral
demonstration are met, and the policy requires an independent
operational assessment for the release of each product increment to the
user. What is unclear in DOD's guidance is the process that will be
used for independently validating whether measurable cost, schedule,
and performance exit criteria have been met. However, DOD stated that
the milestone decision authority provides independent validation that
exit criteria have been met as part of DOD's milestone approval
process. As of October 23, 2003, DOD's report on the status of each
program applying spiral development was still in draft and not yet
submitted. DOD's current draft report states that there are no research
and development programs that have been approved as spiral development
programs as of September 30, 2003. Section 803 requirements were
implemented in DOD Instruction 5000.2, which was effective in May 2003.
DOD anticipates that there will be approved spiral development programs
to report in 2004.
Section 822: Independent Technology Readiness Assessments:
Requirements: This section added a requirement to section 804
of the National Defense Authorization Act for Fiscal Year 2002
(Public Law 107-107) that directed DOD to report by March of each year
between 2003 and 2006 on the maturity of technology at the initiation
of major defense acquisition programs. Each report is required to (1)
identify any major acquisition program that entered system development
and demonstration during the preceding calendar year with immature key
technology that was not demonstrated in, at minimum, a relevant
environment, as required by the new policy; (2) justify the
incorporation of any key technology on an acquisition program that does
not meet that requirement; (3) identify any instances that the Deputy
Under Secretary of Defense for Science and Technology did not concur
with the technology assessment and explain how the issue has been or
will be resolved; (4) identify each case in which a decision was made
not to conduct an independent technology readiness assessment for a
critical technology on a major defense acquisition; and (5) explain the
reasons for the decision each year through 2006.
DOD response: In March 2003, DOD reported that two programs entered
system development and demonstration in 2002 with critical technologies
that did not meet demonstration requirements and provided justification
for them.[Footnote 8] DOD did not identify or report any cases where an
independent technology readiness assessment was not conducted or where
the Under Secretary disagreed with assessment findings.
Conclusions:
DOD can maximize its $1 trillion investment in new weapons over the
next 6 years by ensuring effective implementation of the new
acquisition policy. DOD's leaders have taken noteworthy steps by
incorporating into the policy a framework that supports a knowledge-
based, evolutionary acquisition process, similar to one used by leading
commercial companies to get successful outcomes. A framework is an
important and significant step. DOD must now turn its attention to
establishing controls. As leading companies have found, having clearly
established controls to capture and use appropriate knowledge to make
decisions at critical junctures is crucial for delivering affordable
products as planned. DOD's policy addresses specific congressional
requirements and includes some controls that leading companies use to
capture knowledge at the start of a program. However, additional
controls are needed to ensure that decisions made throughout product
development are informed by demonstrated knowledge.
Recommendations for Executive Action:
DOD must design and implement necessary controls to ensure that
appropriate knowledge is captured and used at critical junctures to
make decisions about moving a program forward and investing more money.
We recommend that the Secretary of Defense require additional controls
for capturing knowledge at three key points--program launch, design
readiness review for transitioning from system integration to system
demonstration, and production commitment. The additional controls for
program launch (milestone B) should ensure the capture of knowledge
about the following:
* Cost and schedule estimates based on knowledge from a preliminary
design using systems engineering tools.
The additional controls for transitioning from system integration to
system demonstration (design readiness review) should ensure the
capture of knowledge about the following:
* Completion of 90 percent of engineering drawings.
* Completion of subsystem and system design reviews.
* Agreement from all stakeholders that drawings are complete and the
design is producible.
* Completion of failure modes and effects analysis.
* Identification of key system characteristics.
* Identification of critical manufacturing processes.
* Reliability targets and a reliability growth plan based on
demonstrated reliability rates of components and subsystems.
The additional controls for the production commitment (milestone C)
should ensure the capture of knowledge about the following:
* Completion of production representative prototypes.
* Availability of production representative prototypes to achieve
reliability goal and demonstrate the product in an operational
environment.
* Collection of statistical process control data.
* Demonstration that critical manufacturing processes are capable and
in statistical control.
Because knowledge about technology, design, and manufacturing at
critical junctures can lower DOD's investment risk, decisions that do
not satisfy knowledge-based criteria should be visible and justified.
Therefore, we also recommend that the Secretary of Defense document the
rationale for any decision to move a program to the next stage of
development without meeting the knowledge-based criteria, including
those listed in the first recommendation. The responsible milestone
decision authority should justify the decision in the program's
acquisition decision memorandum and in a report to Congress.
Agency Comments and Our Evaluation:
DOD provided us with written comments on a draft of this report. The
comments appear in appendix IV.
DOD partially concurred with our recommendation that the Secretary
require additional controls for capturing knowledge at three key
points: program launch, design readiness review for transitioning from
system integration to system demonstration, and production. DOD stated
that it agrees in principle with the advantages of using knowledge-
based controls at key points in the acquisition process to assess risk
and ensure readiness to proceed into the next phase of the acquisition
process. DOD believes the current acquisition framework includes the
controls necessary to achieve effective results, but it will continue
to monitor the process to determine whether others are necessary to
achieve the best possible outcomes.
While we believe DOD's effort to establish a solid framework for
evolutionary acquisitions is a giant step forward, our work has shown
that a disciplined application of controls in the process is needed to
implement the framework if better acquisition outcomes are to be
achieved. DOD's policy does not include all the necessary controls to
ensure a high level of product knowledge is attained and used for
making decisions to move a program forward in the product development
process. Leading product developers use additional controls, as listed
in our first recommendation, to achieve the knowledge necessary to
reduce risk to reasonable levels at critical junctures before making
additional significant investments in product development. Simply
monitoring the process may not be enough for DOD to achieve the best
outcomes. Therefore, we are retaining our recommendation that the
Secretary require additional controls at three critical points in the
acquisition process.
DOD also partially concurred with our recommendation that the Secretary
document in each program's acquisition decision memorandum and in a
report to Congress the rationale for any decision to move a program to
the next stage of development without meeting the knowledge-based
criteria, including those described in the first recommendation. DOD
agreed that it should record and be accountable for program decisions.
Decision makers will continue to use the acquisition decision
memorandum to document program decisions and the rationale for them.
DOD did not concur with the need for a report outside the department.
Because we believe strongly that knowledge-based criteria used to gauge
a product's development progress at critical junctures can lower DOD's
investment risks, we think it is important that decisions made without
satisfying knowledge-based criteria be justified in a visible and
transparent way to hold managers accountable for moving a program
forward absent this knowledge. Therefore, we are retaining our
recommendation for reporting the basis for decisions to move forward in
a report to Congress.
Scope and Methodology:
We reviewed DOD's revised and past acquisition policies, DOD Directive
5000.1, DOD Instruction 5000.2, and DOD 5000.2-R,[Footnote 9] which
provide management principles and mandatory policies and procedures for
managing acquisitions programs. We contacted an official in the Office
of the Under Secretary of Defense for Acquisition, Technology, and
Logistics who is responsible for the development of the policy to
better understand its content. We also reviewed information from the
Defense Acquisition University that provided educational material on
the policies.
We reviewed the relevant sections of the Bob Stump National Defense
Authorization Act for Fiscal Year 2003 and the accompanying Senate
Armed Services Committee report to identify the requirements applicable
to DOD's acquisition policy. We compared these requirements with DOD's
responses to determine whether they have been addressed.
Finally, we used information from more than 10 GAO products that
examine how commercial best practices can improve outcomes for various
DOD programs. During the past 6 years, we have gathered information
based on discussions and visits with the following companies:
:
3M; Boeing Commercial Airplane Group; Chrysler; Bombardier Aerospace;
Ford Motor; Hughes Space and Communication; Motorola; Xerox; Hewlett-
Packard; Caterpillar; Cummins; General Electric Aircraft Engines;
Toyota; Harris Semiconductor; Honda; Texas Instruments; John Deere;
Varian Oncology Systems; Ethicon-Endo Surgery (division of Johnson &
Johnson).
[End of table]
Although the approaches varied, these companies consistently applied
the basic processes and standards in use. We compared this information
with the acquisition framework and controls established by DOD's
policy. We concentrated on whether the policy provides a framework for
a knowledge-based, evolutionary process and the controls necessary to
carry out this intent.
We conducted our review from April 2003 to September 2003 in accordance
with generally accepted government auditing standards.
We are sending copies of this report to the Secretary of Defense; the
Secretaries of the Air Force, Army, and Navy; and the Director of the
Office of Management and Budget. We will also provide copies to others
on request. In addition, the report will be available at no charge on
the GAO Web site at http://www.gao.gov.
Please contact me at (202) 512-4841 if you have any questions
concerning this report. Other key contributors to this report were
Lily Chin, Chris DePerro, Matt Lea, Mike Sullivan, and Adam Vodraska.
Katherine V. Schinasi:
Director, Acquisition and Sourcing Management:
Signed by Katherine V. Schinasi:
[End of section]
Appendix I: Section 802, Evolutionary Acquisition:
Section 802 of the Defense Authorization Act for Fiscal Year 2003
required the Secretary of Defense to submit a report to Congress
explaining how the Department of Defense (DOD) plans to meet certain
statutory and regulatory requirements for acquisition programs
following an evolutionary approach. In April 2003, the Secretary
reported how these requirements were addressed in DOD's policy (such as
in tables of statutory and regulatory information requirements
contained in enclosure 3 of Instruction 5000.2). According to the
report, DOD's policy requires that each program--including an increment
of an evolutionary acquisition--have a milestone B decision to approve
program initiation and to permit entry into systems development and
demonstration. DOD's policy specifies the statutory and regulatory
information necessary to support the decision. We examined the policy
to ensure the statutes and regulations identified in section 802 were
addressed. Table 3 provides a list of the statutory and regulatory
requirements identified in section 802, a corresponding document and
page number where the requirement appears in DOD's policy, and a
description of the requirement from the policy.
Table 3: How DOD Policy Addresses Section 802's Statutory and
Regulatory Requirements:
Section 802 requirements: Requirements of chapter 144 of title 10,
United States Code:
Section 802 requirements: Sec. 2430, major defense acquisition program
defined; DOD policy reference examples: Department of Defense
Instruction (DODI) 5000.2, enclosure 2, p. 16; Brief description from
policy: Table E.2.T1 provides a description (criteria) and the decision
authority by acquisition program category.
Section 802 requirements: Sec. 2431, weapons development and
procurement schedules; DOD policy reference examples: Not
referenced[A]; Brief description from policy: [Empty].
Section 802 requirements: Sec. 2432, Selected Acquisition Reports; DOD
policy reference examples: DODI 5000.2, enclosure 3, p.19; Brief
description from policy: Required at milestone B or program initiation
and annually thereafter; end of quarter following milestone C decision;
full-rate production decision. Also required when there is a baseline
breach.
Section 802 requirements: Sec. 2433, unit cost reports; DOD policy
reference examples: DODI 5000.2, enclosure 3, p. 19; Brief description
from policy: Required on quarterly basis.
Section 802 requirements: Sec. 2434, independent cost estimates;
operational manpower requirements; DOD policy reference examples: DODI
5000.2, enclosure 6, p. 30 DODI 5000.2, enclosure 3, p. 19; Brief
description from policy: Required at program initiation for ships (cost
assessment only). Required at milestones B and C and full-rate
production decision.
Section 802 requirements: Sec. 2435, baseline description; DOD policy
reference examples: DODI 5000.2, enclosure 3, p. 19 DODI 5000.2, p. 8;
Brief description from policy: Required at program initiation for
ships. Required at milestones B and C and full-rate production
decision. Program deviation report required immediately upon program
deviation.
Section 802 requirements: Sec 2440, technology and industrial base
plans; DOD policy reference examples: DODI 5000.2, enclosure 3, p. 19;
Brief description from policy: Required at milestones B and C (part of
acquisition strategy).
Section 802 requirements: Sec. 139, Director of Operational Test and
Evaluation; DOD policy reference examples: Department of Defense
Directive (DODD) 5000.1, p. 3 DODI 5000.2, enclosure 3 DODI 5000.2,
enclosure 5, pp. 27-29; Brief description from policy: Cites Director
of Operational Test and Evaluation as key official of defense
acquisition system. Director responsible for assessing adequacy of
operational tests and live fire tests and evaluating operational
effectiveness, suitability, and survivability of systems.
Section 802 requirements: Sec. 181, Joint Requirements Oversight
Council; DOD policy reference examples: DODD 5000.1, p. 3 DODI 5000.2,
p. 4; Brief description from policy: Chairman of the Joint Chiefs of
Staff, with assistance of Joint Requirements Oversight Council,
responsible for assessing and providing advice regarding capability
needs for defense acquisition programs. Chairman also responsible for
validating and approving capabilities documents.
Section 802 requirements: Sec. 2366, major systems and munitions
programs: survivability testing and lethality testing required before
full-scale production; DOD policy reference examples: DODI 5000.2,
enclosure 3, p. 19 DODI 5000.2, enclosure 5, p. 29 DODI 5000.2, p. 10;
Brief description from policy: Live fire testing and reporting required
for all covered systems. Strategy or live fire waiver and alternate
plan required at milestone B. Live fire test and evaluation report
required for full-rate production decision.
Section 802 requirements: Sec. 2399, operational test and evaluation of
defense acquisition programs; DOD policy reference examples: DODI
5000.2, enclosure 3, p. 18 DODI 5000.2, pp. 9-10 DODI 5000.2, enclosure
5, pp. 27-28; Brief description from policy: Director of Operational
Test and Evaluation shall determine operational effectiveness and
suitability of system under realistic conditions. Beyond Low Rate
Initial Production Report required at full-rate production decision.
DOD may not conduct operational testing until Director of Operational
Test and Evaluation approves test plan.
Section 802 requirements: Sec. 2400, low-rate initial production of new
systems; DOD policy reference examples: DODI 5000.2, enclosure 3, p. 19
DODI 5000.2, pp. 9-10; Brief description from policy: Low-rate initial
production quantities will be determined by milestone B.
Section 802 requirements: DODD 5000.1; DOD policy reference examples:
DODD 5000.1, pp. 1, 2; Brief description from policy: Policies in
directive apply to all acquisition programs. Evolutionary acquisition
strategies are preferred approach to satisfying operational needs.
Section 802 requirements: DODI 5000.2; DOD policy reference examples:
DODI 5000.2, p. 1; Brief description from policy: Instruction applies
to all defense technology projects and acquisition projects.
Section 802 requirements: Chairman of the Joint Chiefs of Staff
Instruction 3170.01B[B]; DOD policy reference examples: DODI 5000.2, p.
4; Brief description from policy: Chairman of Joint Chiefs of Staff
provides advice through Instruction 3170.01. Capability documents
required at concept decision and at milestones A, B, and C.
Section 802 requirements: Other provisions of law and regulations
(including successor documents) that are applicable to such programs;
DOD policy reference examples: DODI 5000.2, enclosure 3, pp. 18-22;
Brief description from policy: Several other statutory, regulatory, and
contract requirements are addressed in the policy.
Sources: Defense Authorization Act for Fiscal Year 2003, DOD, and GAO.
[A] Although the statutory requirement of 10 U.S.C., section 2431 is
not specifically cited in the policy, the same summarized information
is submitted to Congress in the Selected Acquisition Reports for the
first quarter of a fiscal year as required by 10 U.S.C., section 2432.
The requirement for Selected Acquisition Reports is addressed by DOD's
acquisition policy. The statute concerning weapons development and
procurement schedules requires the Secretary of Defense to submit
budget justification documents regarding this information for each
weapon system for which fund authorization is required (and for which
procurement funds are requested in the budget) not later than 45 days
after the President submits the budget to Congress.
[B] Chairman of the Joint Chiefs of Staff Instruction 3170.01B was
revised and reissued June 24, 2003, as Joint Chiefs of Staff
Instruction 3170.01C with an accompanying manual.
[End of table]
Section 802 also required DOD to report on its plans for addressing
certain acquisition process issues regarding each increment of an
evolutionary process. DOD reported on how it plans to establish and
approve operational requirements and cost and schedule goals; meet
requirements for operational and live fire testing; monitor cost and
schedule performance; achieve interoperability; and consider total
system performance and total ownership costs. We compared DOD's
response with section 802's reporting requirements. As shown in table
4, DOD was responsive to the 802 requirements.
Table 4: How DOD Responded to Section 802's Requirements Regarding
Specific Matters for Each Increment of an Evolutionary Acquisition
Process:
Section 802 requirements: "The manner in which the Secretary plans to
establish and approve, for each increment of an evolutionary
acquisition process:
Section 802 requirements: "operational requirements;" DOD
response: "Operational Requirements: Each program is required to have
documented, approved operational requirements in accordance with
authorized Joint Staff procedures. For evolutionary acquisition
programs, the requirements documents are typically time-phased and
specify the capability expected of each increment."; Policy reference:
Capability development document, including key performance parameters,
required at milestone B or program initiation (DODI 5000.2, p. 7, and
enclosure 3, p. 20).
Section 802 requirements: "cost and schedule goals."; DOD response:
"Cost and Schedule Goals: At program initiation, each program and
program increment is required to have an Acquisition Program Baseline
approved by the Milestone Decision Authority (MDA). The Acquisition
Program Baseline includes cost and schedule goals."; Policy reference:
Acquisition program baseline required at milestone B or program
initiation (DODI 5000.2, p. 8, and enclosure 3, p. 19).
Section 802 requirements: "The manner in which the Secretary plans, for
each increment of an evolutionary acquisition process:"
Section 802 requirements: "to meet requirements for operational testing
and live fire testing;"; DOD response: "At program initiation, each
program or program increment must have a Test and Evaluation Master
Plan (TEMP) approved by the Director, Operational Test and Evaluation.
The TEMP includes requirements for operational and live fire testing.";
Policy reference: Test and Evaluation Master Plan required at milestone
B or program initiation (DODI 5000.2, enclosure 3, p. 21, and enclosure
5, p. 25).
Section 802 requirements: "to monitor cost and schedule performance;
and"; DOD response: "Progress against cost and schedule goals [for each
increment] is addressed via automated reporting systems at both the
Office of Secretary of Defense staff level and at the Service staff
level, and by reviews conducted in the context of the acquisition
oversight model."; Policy reference: DOD's policy contains an enclosure
on resource estimation (DODI 5000.2, enclosure 6, pp. 30-31). Also,
several reporting requirements address monitoring cost and schedule
performance such as acquisition program baselines and selected
acquisition reports (DODI 5000.2, enclosure 3, pp. 18-21).
Section 802 requirements: "to comply with laws requiring reports to
Congress on results testing and on cost and schedule performance."; DOD
response: "DoD acquisition policy requires independent assessment by
the operational test authority prior to release of each successive
increment to the user. All reports by statute will be submitted.";
Policy reference: The service shall perform an independent operational
assessment prior to release of each successive increment to the user
(DODI 5000.2, enclosure 5, p. 29). Also, several statutory reports are
listed in the policy addressing testing and cost and schedule
performance (DODI 5000.2, enclosure 3, pp. 18-21).
Section 802 requirements: "The manner in which the Secretary plans to
ensure that each increment of an evolutionary acquisition process is
designed:"
Section 802 requirements: "to achieve interoperability within and among
United States forces and United States coalition partners; and"; DOD
response: "Each increment of an evolutionary acquisition program is
required to have an Acquisition Program Baseline, approved by the MDA,
which includes an Interoperability key performance parameter."; Policy
reference: Acquisition program baseline required at milestone B or
program initiation (DODI 5000.2, p. 8, and enclosure 3, p. 19).
Section 802 requirements: "to optimize total system performance and
minimize total ownership costs by giving appropriate consideration to:
"logistics planning;"; "manpower, personnel, and training;"; "human,
environmental, safety, occupational health, accessibility,
survivability, operational continuity and security factors;";
"protection of critical program information; and"; "spectrum
management"; DOD response: "Each increment of an evolutionary
acquisition program is required to have an acquisition strategy,
approved by the MDA, that addresses [where applicable]: logistics
planning; manpower, personnel and training; human, environmental,
safety, occupational health; accessibility (human), survivability,
operational continuity (as required by requirements document), security
factors; critical program information; and spectrum management. These
factors are taken into account as the Department of Defense considers
total system performance and total ownership costs."; Policy reference:
Acquisition strategy required at milestone B or program initiation
(DODI 5000.2, p. 7, and enclosure 3, p. 20).
Sources: Defense Authorization Act for Fiscal Year 2003, DOD, and GAO.
[End of table]
[End of section]
Appendix II: Section 803, Spiral Development:
Section 803 of the Defense Authorization Act for Fiscal Year 2003
authorized the Secretary of Defense to conduct major defense
acquisition programs as spiral development programs. However, the
section placed a limitation on these programs. It stated that a
research and development program for a major acquisition may not be
conducted as a spiral development program unless the Secretary of
Defense or authorized high-level designee gives approval. The section
requires the Secretary of Defense to issue guidance for the
implementation of such programs to address appropriate processes for
ensuring the independent validation of exit criteria being met, the
operational assessment of fieldable prototypes, and the management of
these types of programs.
DOD responded to these requirements principally by incorporating into
the acquisition policy the requirement for a technology development
strategy. This strategy is a prerequisite for a project to enter the
technology development phase of the acquisition process, or
milestone A. Table 5 compares the spiral development plan requirements
in the act with the technology development strategy requirements in
DOD's May 2003 acquisition policy.
Table 5: How DOD Policy Addresses Section 803's Requirements:
[See PDF for image]
Sources: Defense Authorization Act for Fiscal Year 2003, DOD, and GAO.
[End of table]
As shown in the table, DOD's policy generally responded to the
requirements in the act concerning guidance for implementation of
spiral development programs. While the policy includes a technology
development strategy that requires a test plan to ensure the goals and
exit criteria for the first technology spiral demonstration are met and
an independent operational assessment for the release of each product
increment to the user, it is unclear what the process is for
independently validating that cost, schedule, and performance exit
criteria have been. However, DOD stated that the milestone decision
authority provides independent validation that exit criteria have been
met as part of DOD's milestone approval process. Section 803 also
requires that a spiral development plan include "[s]pecific cost,
schedule, and performance parameters, including measurable exit
criteria, for the first spiral to be conducted." DOD's policy
substituted "parameters" for "goals" and did not use the term
"measurable" in describing the required exit criteria.
Finally, section 803 requires the Secretary of Defense to submit to
Congress by September 30 yearly from 2003 through 2008 a status report
on each spiral development program. The report is to include
information on unit costs for the projected prototypes. As of October
23, 2003, DOD's report on the status of each program applying spiral
development was still in draft and not yet submitted. DOD's current
draft report states that there are no research and development programs
that have been approved as spiral development programs as of September
30, 2003. Section 803 requirements were implemented in DOD Instruction
5000.2, which was effective in May 2003. DOD anticipates that there
will be approved spiral development programs to report in 2004.
[End of section]
Appendix III: Section 804, Technology Maturity, and Section 822,
Independent Technology Readiness Assessments:
Section 804 of the Defense Authorization Act for Fiscal Year 2002
required DOD to report on the maturity of technology at the initiation
of major defense acquisition programs. The act directed DOD to report
by March 1 of each year between 2003 and 2006 on a requirement in DOD's
policy that technology must have been demonstrated in a relevant
environment (or, preferably, in an operational environment) to be
considered mature enough to use for product development in systems
integration. Each report is required to (1) identify any major
acquisition program that entered system development and demonstration
during the preceding calendar year with immature key technology that
was not demonstrated in, at minimum, a relevant environment, as
required by the new policy; (2) justify the incorporation of any key
technology on an acquisition program that does not meet that
requirement; (3) and identify any instances that the Deputy Under
Secretary of Defense for Science and Technology did not concur and
explain how the issue has been or will be resolved, including
information on the use of independent readiness assessments. Section
822 of the Defense Authorization Act for Fiscal Year 2003 amended
section 804 by adding a requirement that the Secretary of Defense
identify each case in which an authoritative decision has been made
within DOD not to conduct an independent technology readiness
assessment for a critical technology on a major defense acquisition
program and explain the reasons for the decision. On March 18, 2003,
DOD submitted its first report. Table 6 shows the specific requirements
for the report and DOD's response.
Table 6: How DOD Responded to Section 804 and Section 822 Requirements:
Reporting requirements: Section 804: "identify each case in which a
major defense acquisition program entered system development and
demonstration during the preceding calendar year and into which key
technology has been incorporated that does not meet the technological
maturity requirement — and provide a justification for why such key
technology was incorporated."[A]; DOD's response: DOD reported two
programs (Joint Tactical Radio System Cluster 1 and Composite Health
Care System II) that entered system development and demonstration with
key technologies that did not meet the technological maturity
requirement. While DOD did not specifically identify all the
technologies for these programs that did not meet the requirements, it
did provide its justification for why the technologies were
incorporated.
Reporting requirements: Section 804: "identify any determination of
technological maturity with which the Deputy Under Secretary of Defense
for Science and Technology did not concur and explain how the issue has
been or will be resolved."; DOD's response: DOD reported that, in all
cases, the Deputy Under Secretary of Defense for Science and Technology
concurred with the technology readiness assessment of the program
manager and the milestone decision authority.
Reporting requirements: Section 822: "identify each case in which an
authoritative decision has been made within the Department of Defense
not to conduct an independent technology readiness assessment for a
critical technology on a major defense acquisition program and explain
the reasons for the decision."; DOD's response: DOD did not report on
or identify any cases.
Sources: National Defense Authorization Acts for Fiscal Years 2002 and
2003, DOD, and GAO.
[A] Technology must have been demonstrated in a relevant environment
(or, preferably in an operational environment) to be considered mature
enough to use for product development in systems integration.
[End of table]
[End of section]
Appendix IV: Comments from the Department of Defense:
OFFICE OF THE UNDER SECRETARY OF DEFENSE:
ACQUISITION, TECHNOLOGY AND LOGISTICS:
3000 DEFENSE PENTAGON WASHINGTON, DC 20301-3000:
October 22, 2003:
Ms. Katherine Schinasi:
Director, Acquisition and Sourcing Management U.S. General Accounting
Office:
441 G Street, N.W. Washington, D.C. 20548:
Dear Ms. Schinasi:
This is the Department of Defense (DOD) response to the GAO draft
report, DEFENSE ACQUISITIONS: DoD's Revised Policy Emphasizes Best
Practices, But More Controls Are Needed, dated September 26, 2003 (GAO
Code 120214/GAO-04-53). The Department of Defense partially concurs
with the recommendations in the report. We have enclosed comments which
explain this position.
Sincerely,
Signed for:
Deidre A. Lee:
Director, Defense Procurement and Acquisition Policy:
Enclosure: As stated:
GAO DRAFT REPORT-DATED SEPTEMBER 26, 2003 GAO CODE 120214/GAO-04-53:
DEFENSE ACQUISITIONS: DOD'S REVISED POLICY EMPHASIZES BEST PRACTICES,
BUT MORE CONTROLS ARE NEEDED:
DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATIONS:
RECOMMENDATION 1: The GAO recommended that the Secretary of Defense
require additional controls for capturing knowledge at three key
points-program launch, design readiness review for transitioning from
system integration to system demonstration, and production commitment.
(p. 11/GAO Draft Report):
The additional controls for program launch (milestone B) should ensure
the capture of knowledge about the following:
* Cost and schedule estimates based on knowledge from a preliminary
design using systems engineering tools.
The additional controls for the moving from system integration to
system demonstration (design readiness review) should ensure the
capture of knowledge about the following:
* Completion of 90 percent of engineering drawings.
* Completion of subsystem and system design reviews.
* Agreement from all stakeholders that drawings are complete and the
design is producible.
* Completion of failure modes and effects analysis.
* Identification of key system characteristics.
* Identification of critical manufacturing processes.
* Reliability targets and a reliability growth plan based on
demonstrated reliability rates of components and subsystems.
The additional controls for the production commitment (milestone C)
should ensure the capture of knowledge about the following:
* Completion of production representative prototypes.
* Availability of production representative prototypes to achieve
reliability goal and demonstrate the product in an operational
environment.
* Collection of statistical process control data.
* Demonstration that critical manufacturing processes are capable and in
statistical control.
DOD RESPONSE: Partially Concur. The Department of Defense agrees in
principle with the advantages of using knowledge-based controls at key
points in the acquisition process to assess risk and ensure readiness
to proceed into the next phase of the acquisition process. The
acquisition framework was recently redesigned to include the controls
we consider necessary to achieve effective program results, including,
for example, assessments of technical and design readiness. We will
continue to monitor the acquisition process to assess the effectiveness
of these controls, and determine whether others are necessary to
achieve the best possible outcomes.
RECOMMENDATION 2: The GAO recommended that the Secretary of Defense
document the rationale for any decision to move a program to the next
stage of development without meeting the knowledge-based criteria,
including those listed in the first recommendation. The responsible
acquisition executive should justify the decision in the program's
acquisition decision memorandum and in a report to Congress. (p. 12/GAO
Draft Report):
DOD RESPONSE: Partially concur. The Department of Defense agrees that
we should be accountable for program decisions and appropriately record
the considerations in moving from one stage of development to the next.
Milestone Decision Authorities will continue to employ the Acquisition
Decision Memorandum (ADM) to document program decisions and their
rationale. We do not concur with the need for a report outside the
Department.
[End of section]
Related GAO Products:
Best Practices: Setting Requirements Differently Could Reduce
Weapon Systems' Total Ownership Costs. GAO-03-57. Washington, D.C.:
February 11, 2003.
Best Practices: Capturing Design and Manufacturing Knowledge
Early Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.:
July 15, 2002.
Defense Acquisitions: DOD Faces Challenges in Implementing Best
Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.
Best Practices: Better Matching of Needs and Resources Will Lead
to Better Weapon System Outcomes. GAO-01-288. Washington, D.C.:
March 8, 2001.
Best Practices: A More Constructive Test Approach Is Key to
Better Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.:
July 31, 2000.
Defense Acquisition: Employing Best Practices Can Shape Better Weapon
System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.:
April 26, 2000.
Best Practices: DOD Training Can Do More to Help Weapon System Program
Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.:
August 16, 1999.
Best Practices: Better Management of Technology Development Can Improve
Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30,
1999.
Defense Acquisitions: Best Commercial Practices Can Improve Program
Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.
Defense Acquisition: Improved Program Outcomes Are Possible. GAO/T-
NSIAD-98-123. Washington, D.C.: March 17, 1998.
Best Practices: DOD Can Help Suppliers Contribute More to Weapon System
Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998.
Best Practices: Successful Application to Weapon Acquisition Requires
Changes in DOD's Environment. GAO/NSIAD-98-56. Washington, D.C.:
February 24, 1998.
Best Practices: Commercial Quality Assurance Practices Offer
Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.:
August 26, 1996.
FOOTNOTES
[1] S. Rept. No. 108-46, Report of the Senate Armed Services Committee
accompanying S. 1050, National Defense Authorization Act for Fiscal
Year 2004, at page 346 (May 13, 2003).
[2] As used here, the term "regulatory requirements" refers to policies
governing DOD's acquisition system.
[3] DOD Directive 5000.1, The Defense Acquisition System, describes the
management principles for DODís acquisition programs DOD Instruction
5000.2, The Operation of the Defense Acquisition System, outlines a
framework for managing acquisition programs. Collectively, these are
known as the 5000 series.
[4] Program elements of the ballistic missile defense program enter the
formal DOD acquisition cycle at milestone C (production commitment) and
are subject to the 5000 series from that point on. To use the
streamlined process of the National Security Space Acquisition Policy,
a Space System Program Director/Program Manager must request that the
DOD Space Milestone Decision Authority (the Under Secretary of the Air
Force) grant a waiver and an exemption to the processes and procedures
described in DOD Instruction 5000.2 (the waiver authority does not
include DOD Directive 5000.1). Notwithstanding these policy exceptions,
statutory requirements for major defense acquisition programs continue
to apply to missile defense and space programs.
[5] A spiral development program is defined in section 803 of the
Defense Authorization Act for Fiscal Year 2003 as a research and
development program conducted in discrete phases or blocks, each of
which will result in the development of fieldable prototypes and will
not proceed into acquisition until specific performance parameters,
including measurable exit criteria, have been met.
[6] The milestone decision authority is typically the DOD component
acquisition executive or designee, or for certain large programs, the
head of the component or the Under Secretary of Defense for
Acquisition, Technology, and Logistics.
[7] Section 803 authorized the Secretary of Defense to delegate
authority to approve a spiral development plan to the Under Secretary
of Defense for Acquisition, Technology, and Logistics or to the senior
acquisition executive of the military department or defense agency
concerned. The authority may not be delegated further.
[8] The two programs were the Composite Health Care System II and the
Joint Tactical Radio System, Cluster 1.
[9] DOD 5000.2-R was canceled and replaced with the Interim Defense
Acquisition Guidebook.
GAO's Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW,
Room LM Washington,
D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.
General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.
20548:
