Defense Acquisitions
Perspectives on Potential Changes to Department of Defense Acquisition Management Framework Gao ID: GAO-09-295R February 27, 2009The National Defense Authorization Act (NDAA) for Fiscal Year 2008 directed GAO to report on potential modifications of the organization and structure of the Department of Defense (DOD) for Major Defense Acquisition Programs (MDAP). In preparing the report, the review was required to include the feasibility and advisability of seven potential modifications of DOD's organization and structure for MDAPs. We were charged with reporting on the feasibility and advisability of (1) establishing a process in which the commanders of combatant commands (COCOM) assess and provide input on the capabilities needed to successfully accomplish their missions over a long-term planning horizon of 15 years or more; (2) establishing a materiel solutions process for addressing identified gaps in critical warfighting capabilities, under which the Under Secretary of Defense for Acquisition, Technology, and Logistics (USD (AT&L)) circulates among the military departments and appropriate defense agencies requests for proposals for technologies and systems to address such gaps; (3) revising the acquisition process by establishing shorter, more frequent acquisition program milestones; (4) requiring the milestone decision authority (MDA) to specify at the time of milestone B approval the period of time that will be required to deliver an initial operational capability (IOC) to the commanders of the relevant COCOMs; (5) establishing a new office to provide independent cost and performance estimates; (6) requiring certifications of program status to the Defense Acquisition Executive (DAE) and Congress prior to milestone approval; and (7) modifying the role played by Chiefs of Staff of the Armed Forces in the requirements, resource allocation, and acquisition processes. In carrying out this review, as Congress directed, we obtained the views of current and prior senior DOD acquisition officials, currently serving senior military officers involved in setting requirements, and other experts including some who participated in previous reviews of DOD's organization and structure for carrying out the acquisition of major weapon systems.
Based on our discussions with acquisition experts and reviews of our own and key studies on DOD acquisition, we believe that six of the seven potential changes included in the fiscal year 2008 NDAA could help improve DOD's acquisition processes with some modifications, but only if they are accompanied by additional changes in the overall acquisition environment. Specifically, the six changes are (1) establishing a process in which the combatant commanders assess and provide input on the long-term capabilities needed to successfully accomplish their missions; (2) establishing a materiel solutions process for addressing identified gaps in critical warfighting capabilities, under which USD (AT&L) circulates among the military departments and appropriate defense agencies requests for proposals for technologies and systems to address such gaps; (3) revising the acquisition process by establishing shorter, more frequent acquisition program milestones; (4) requiring the MDA to specify at the time of milestone B approval the period of time that will be required to deliver an IOC to the relevant combatant commanders; (5) establishing a new office to provide independent cost and performance estimates; and (6) requiring certifications of program status to the DAE and Congress prior to milestone approval. The sixth potential change appears to have been instituted already in recent legislation. Finally, we could not determine whether the remaining potential change--modifying the role played by Chiefs of Staff of the Armed Forces in the requirements, resource allocation, and acquisition processes--would improve acquisition outcomes as this potential change requires further definition and more study. Our discussions with acquisition experts indicate that these changes may not achieve the desired improvement in acquisition outcomes unless they are accompanied by changes in the overall acquisition environment, its culture, and the incentives provided for success. Some of the DOD executives we interviewed--both current and former--emphatically stated that these changes will not improve the acquisition system until the department can significantly improve its prioritization of needs so that fewer programs are competing for available dollars. Furthermore, some of the experts expressed a strong belief that none of the potential changes may be acceptable to some process stakeholders because some stakeholders like the process as it is. Another expert characterized the acquisition process as a system in equilibrium. It will take strong, consistent strategic leadership to change the current culture and environment. DOD did not provide a written response to a draft of this correspondence, but provided technical comments which we have incorporated as appropriate.
GAO-09-295R, Defense Acquisitions: Perspectives on Potential Changes to Department of Defense Acquisition Management Framework
This is the accessible text file for GAO report number GAO-09-295R
entitled 'Defense Acquisitions: Perspectives on Potential Changes to
Department of Defense Acquisition Management Framework' which was
released on February 27, 2009.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
GAO-09-295R:
United States Government Accountability Office:
Washington, DC 20548:
February 27, 2009:
Congressional Committees:
Subject: Defense Acquisitions: Perspectives on Potential Changes to
Department of Defense Acquisition Management Framework:
The National Defense Authorization Act (NDAA) for Fiscal Year 2008
directed GAO to report on potential modifications of the organization
and structure of the Department of Defense (DOD) for Major Defense
Acquisition Programs (MDAP). In preparing the report, the review was
required to include the feasibility and advisability of seven potential
modifications of DOD's organization and structure for MDAPs.[Footnote
1] We were charged with reporting on the feasibility and advisability
of (1) establishing a process in which the commanders of combatant
commands (COCOM) assess and provide input on the capabilities needed to
successfully accomplish their missions over a long-term planning
horizon of 15 years or more; (2) establishing a materiel solutions
process for addressing identified gaps in critical warfighting
capabilities, under which the Under Secretary of Defense for
Acquisition, Technology, and Logistics (USD (AT&L)) circulates among
the military departments and appropriate defense agencies requests for
proposals for technologies and systems to address such gaps; (3)
revising the acquisition process by establishing shorter, more frequent
acquisition program milestones; (4) requiring the milestone decision
authority (MDA) to specify at the time of milestone B approval the
period of time that will be required to deliver an initial operational
capability (IOC) to the commanders of the relevant COCOMs; (5)
establishing a new office to provide independent cost and performance
estimates; (6) requiring certifications of program status to the
Defense Acquisition Executive (DAE) and Congress prior to milestone
approval; and (7) modifying the role played by Chiefs of Staff of the
Armed Forces in the requirements, resource allocation, and acquisition
processes.[Footnote 2] See enclosure II for the full wording of the
potential changes.
In carrying out this review, as Congress directed, we obtained the
views of current and prior senior DOD acquisition officials, currently
serving senior military officers involved in setting requirements, and
other experts including some who participated in previous reviews of
DOD's organization and structure for carrying out the acquisition of
major weapon systems.[Footnote 3] We also reviewed and made extensive
use of GAO's prior work and studies commissioned by DOD or the
President--listed in enclosure IV.
We conducted this performance audit from January 2008 to February 2009
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives. For more on our scope and
methodology, see enclosure I.
Results in Brief:
Based on our discussions with acquisition experts and reviews of our
own and key studies on DOD acquisition, we believe that six of the
seven potential changes included in the fiscal year 2008 NDAA could
help improve DOD's acquisition processes with some modifications, but
only if they are accompanied by additional changes in the overall
acquisition environment. Specifically, the six changes are (1)
establishing a process in which the combatant commanders assess and
provide input on the long-term capabilities needed to successfully
accomplish their missions; (2) establishing a materiel solutions
process for addressing identified gaps in critical warfighting
capabilities, under which USD (AT&L) circulates among the military
departments and appropriate defense agencies requests for proposals for
technologies and systems to address such gaps; (3) revising the
acquisition process by establishing shorter, more frequent acquisition
program milestones; (4) requiring the MDA to specify at the time of
milestone B approval the period of time that will be required to
deliver an IOC to the relevant combatant commanders; (5) establishing a
new office to provide independent cost and performance estimates; and
(6) requiring certifications of program status to the DAE and Congress
prior to milestone approval. The sixth potential change appears to have
been instituted already in recent legislation[Footnote 4]. Finally, we
could not determine whether the remaining potential change--modifying
the role played by Chiefs of Staff of the Armed Forces in the
requirements, resource allocation, and acquisition processes--would
improve acquisition outcomes as this potential change requires further
definition and more study.
Our discussions with acquisition experts indicate that these changes
may not achieve the desired improvement in acquisition outcomes unless
they are accompanied by changes in the overall acquisition environment,
its culture, and the incentives provided for success. Some of the DOD
executives we interviewed--both current and former--emphatically stated
that these changes will not improve the acquisition system until the
department can significantly improve its prioritization of needs so
that fewer programs are competing for available dollars. Furthermore,
some of the experts expressed a strong belief that none of the
potential changes may be acceptable to some process stakeholders
because some stakeholders like the process as it is. Another expert
characterized the acquisition process as a system in equilibrium. It
will take strong, consistent strategic leadership to change the current
culture and environment. DOD did not provide a written response to a
draft of this correspondence, but provided technical comments which we
have incorporated as appropriate.
Background:
DOD's MDAPs are carried out under an overall defense acquisition
management framework comprised of three major processes: one for
requirements definition, one for resource allocation, and one for the
development and production of specific systems.[Footnote 5] Within that
framework, the requirements process identifies, assesses, and
prioritizes warfighting needs utilizing DOD's Joint Capabilities
Integration and Development System (JCIDS); the resource allocation
process governs the distribution of financial resources across DOD and
the military services through DOD's Planning, Programming, Budgeting,
and Execution System (PPBES) to meet validated requirements; and the
Defense Acquisition System (DAS) serves to manage the execution of
product development and procurement.[Footnote 6]
DOD's overall acquisition management framework in theory integrates
three interdependent and stable processes of budget, acquisition, and
requirements and is built on a foundation that requires stability and
continuity. Within that framework, the DAS is the acquisition process
that defines how to buy weapon systems, but it does not include
requirements and budget. The system involves contracting, program
management, developmental and operational test and evaluation,
production, and initial fielding of weapon systems. While all these
activities are necessary and important, they are a subset of the
strategic-level overall management framework. Together, what DOD
decides to buy at the strategic level and how it develops, tests, and
produces weapon systems at the program level determine the capabilities
DOD delivers to the warfighter.
DOD Directive 5000.01 defines the DAS as the management process used by
DOD to provide weapons systems and other equipment to users. DOD
Instruction 5000.02 provides more specific guidance and identifies the
entrance and exit criteria necessary for an acquisition program to move
through each of the phases of the acquisition process. DOD recently
updated DOD Instruction 5000.02 in December 2008 to incorporate changes
to policy that were the result of new or revised sections of public law
enacted since 2003; approved DOD policy appearing in over 25 policy
memos; and DOD responses to GAO and DOD Office of Inspector General
reports. A comparison of the new and prior processes and an explanation
of the major changes are provided in enclosure III.
The MDA approves the program structure, including the type and number
of decision points.[Footnote 7] Milestone decision points can initiate
programs and authorize entry into the major acquisition process phases.
Decision reviews assess progress and authorize or halt further program
activity. At each milestone, the MDA (which will be the DAE or
Component Acquisition Executive (CAE) depending upon the scale or
complexity of the acquisition program) conducts a management review to
determine if a program is ready to progress to the next phase of the
framework.[Footnote 8] Milestone A review is held between the material
solution analysis and technology development phases; milestone B review
is held between the technology development and engineering and
manufacturing development phases; and milestone C review is held
between the engineering and manufacturing development and the
production and deployment phases.
GAO, DOD, and other organizations have issued a number of reports over
the past 20 or more years examining both the overall acquisition
management framework and the programmatic implementation of the DAS and
making recommendations for improvements. Five reports seemed most
relevant to the seven potential changes, the challenges facing defense
acquisition, and ways to mitigate them--the Quadrennial Defense Review,
the Center for Strategic and International Studies' (CSIS) Beyond
Goldwater-Nichols series volume two, the Defense Science Board's (DSB)
Summer Study on Transformation, the Defense Acquisition Performance
Assessment (DAPA) and the Packard Commission report,[Footnote 9] a 1986
study which became the foundation for the Goldwater-Nichols Department
of Defense Reorganization Act of 1986.[Footnote 10]
Potential Changes May Improve Acquisition Outcomes:
We believe that six of the seven potential changes--some with
modification--could improve the acquisition process and its outcomes if
accompanied by additional changes in the overall acquisition
environment. The sixth potential change appears to have been instituted
already in recent legislation.[Footnote 11] It is unclear whether the
potential change modifying the role played by Chiefs of Staff of the
Armed Forces in the requirements, resource allocation, and acquisition
processes would improve acquisition outcomes because it requires
further definition and more study.
Establishing a Process in Which the Combatant Commanders Assess and
Provide Input on the Long-Term Capabilities Needed to Successfully
Accomplish Their Missions:
Based on our work and other major acquisition reform studies, we
believe that enduring requirements reform will occur when the combatant
commanders have more influence over the requirements process to meet
their priorities. This reform is necessary to create a truly joint
military and lay the foundation for an acquisition process that is
responsive to operational needs.
COCOMs are responsible for conducting combat operations and,
ultimately, ensuring that the warfighter has the capabilities needed to
defeat threats. However, these commands currently have a role in the
requirements process that can be strengthened. A key vehicle for
informing the process is the integrated priority list (IPL), which is
nonbinding and does not obligate the military services, which currently
have primary responsibility over the requirements process, to fund any
of the COCOM requirements.[Footnote 12] However, many of the COCOMs do
not believe that their needs, which are reflected through the IPL
process, are sufficiently addressed through the department's JCIDS
process. This has created well-known problems among the services
including duplication in capabilities and persistent interoperability
problems. The potential change would provide more authority to COCOMs
to ensure their long-term needs are met. We believe if this potential
change were implemented, a more COCOM-focused requirements process
could improve joint warfighting capabilities by ensuring that the
combatant commanders--the customers--are provided the appropriate level
of input regarding the capabilities needed to execute their missions
rather than relying on the military services--the suppliers-- to drive
requirements. Our review of DOD's past studies and our own work
provides additional rationale for this change. Previous recommendations
and evidence to support this change include the following:
* GAO recently reported the results of our review of documentation
related to proposals for new capabilities that found that most--almost
70 percent--were sponsored by the military services, with little
involvement from COCOMs.[Footnote 13]
* The DAPA panel recommended that the COCOMs play the lead role in
defining needed capabilities and identifying gaps, and that the
military services and DOD agencies compete to provide solutions once
those gaps are identified. The report further stated that decisions
concerning which capabilities to acquire should be made from a joint
operational perspective. It also found that within the current
requirements identification process, the military services provide the
primary input concerning needed capabilities and control the resources
for procuring capabilities, thus the requirements frequently are not
linked to the capabilities desired by the combatant commanders.
* CSIS's Beyond Goldwater-Nichols body of work and DSB's 2005 Summer
Study on Transformation found that only COCOMs have operational
requirements and operational responsibility to employ the military
services as a joint team and therefore should have more influence in
establishing requirements.
A small majority of the experts we interviewed who commented on this
change (23 of 43) believed that a requirements process that provides
the COCOMs with more influence over which needs and gaps get filled
could improve acquisition outcomes if the COCOMs are provided needed
personnel and financial resources to do this. Additionally, some
experts stated that this change would promote joint solutions and would
be more responsive to warfighters' most urgent needs. The remaining
experts, however, told us the potential change would not improve the
process, and some believed this was because the existing JCIDS process
already considers the combatant commanders' long-term needs and that
they lack the required expertise and resources.
To implement a more COCOM-focused requirements process, DOD would need
to ensure the COCOMs have sufficient additional resources to establish
robust analytical capabilities to identify and assess their long-term
requirements. Also, while some of the participants in DOD's acquisition
system may object to a perceived shift in focus for the COCOMs from
short-term to long-term views, this objection may be addressed by
implementing a Beyond Goldwater-Nichols Phase II Report recommendation
that DOD utilize functional commands, such as the Joint Forces Command,
to provide robust analytical support to the geographical COCOMs to
identify and assess each command's long-term requirements. According to
DOD, it has taken action already that it believes is in line with the
Beyond Goldwater-Nichols Phase II Report recommendation. A recent JROC
memorandum directed that the lead functional COCOM for the battlespace
awareness, command and control, logistics, and net-centric portfolios
will coordinate on any capability documents, and the lead functional
combatant command Deputy Commander will coordinate on the associated
JROC or Joint Capability Board memoranda that validate those capability
documents.[Footnote 14] We believe this is a step in the right
direction.
In its latest Defense Acquisition Transformation Report to Congress,
DOD indicated that the Joint Staff is heavily engaging the COCOMs in
the requirements process through participation in JROC meetings. DOD
noted that in fiscal year 2008 over 75 percent of JROC meetings
included one or more COCOM general officer representatives.
Furthermore, JROC used the COCOMs' IPLs as the starting point for a
series of assessments that identified the most critical military issues
and prioritized the capability gaps for each Joint Capability Area
(JCA).[Footnote 15] Additionally, twice a year JROC visits and consults
with each COCOM to better understand their warfighting needs and to
provide feedback on JROC's and the military services' efforts to
satisfy those needs.
Establishing a Materiel Solutions Process for Addressing Identified
Gaps in Critical Warfighting Capabilities, Under Which USD (AT&L)
Circulates Among the Military Departments and Appropriate Defense
Agencies Requests for Proposals for Technologies and Systems to Address
Such Gaps:
If other important changes also take place, we believe this potential
change could significantly improve acquisition outcomes. If implemented
in conjunction with an integrated, portfolio management based
investment strategy, it would establish a single point of
accountability that has the potential to help ensure interoperability
among multiple systems; help reduce stovepiped and sometimes
duplicative acquisition programs; and enable DOD to more easily align
its portfolio of systems within its budgetary resources.
GAO and others have time and again reported on service parochialism in
DOD's requirements process and the military services may not be in the
best position to decide what the combatant commanders need to conduct
their operations, ensure systems are interoperable, and eliminate
duplication of systems. Previous recommendations and evidence to
support this change include the following:
* GAO recently reported that the JCIDS process has not yet met its
objective to identify and prioritize warfighting needs from a joint
capabilities perspective. Instead, capabilities continue to be driven
primarily by the individual military services. GAO has repeatedly
reported that DOD could improve acquisition outcomes if, at the Office
of the Secretary of Defense (OSD) level, it utilized a portfolio
management approach to guide its investment strategy rather than rely
on each military service to choose which weapons to procure.
* The DAPA panel recommended requiring the military services to compete
to satisfy the combatant commanders' requirements.
* DSB's 2005 Summer Study on Transformation recommended an
organizational structure that clearly identifies the Secretary of
Defense with support from USD (AT&L) as the decision maker for weapon
system solutions with advice from the Chairman of the Joint Chiefs of
Staff and the Joint Staff.
Of the 49 experts we interviewed who commented on this potential
change, 35 believed that it would not improve the acquisition
processes. Some of those experts told us that USD (AT&L) was not
properly resourced for this change to be effective. Also, there
appeared to be uncertainty among the experts as to how this change
would be implemented. Some agreed that the services would be strictly
tasked with providing solutions to requirements that have already been
vetted, while others seemed to interpret the change as one that would
include requirement-setting responsibility. However, the remaining
experts felt this potential change would improve the process. Some of
those experts cited benefits including reducing service parochialism
and increasing the role of OSD staff and the combatant commanders in
the requirements process.
To be effective, this potential change should be considered in
conjunction with the potential change to establish a process in which
the combatant commanders are given more responsibility for assessing
capabilities and determining requirements. Together, these changes
could help ensure focus on filling the capability gaps identified by
the combatant commanders and reducing service parochialism.
The department has reported efforts that may help in this regard. In
its latest Defense Acquisition Transformation Report to Congress, DOD
stated that it is addressing some of the perceived problems in this
area by adopting concept decision and capability portfolio management
processes under the auspices of the USD (ATL)--that are intended to
establish better OSD leadership to improve strategic investment
decisions.[Footnote 16] DOD also stated that it is seeking to challenge
and empower its acquisition workforce to yield the best value for the
taxpayer and capability for the warfighter.
Revising the Acquisition Process by Establishing Shorter, More Frequent
Acquisition Program Milestones:
This potential change, if adopted in a manner consistent with the
mandate and its supporting conference report, would require shorter,
more frequent acquisition milestones to manage requirements,
technology, design, and manufacturing risk more closely.[Footnote 17]
Based on our own body of work and review of relevant DOD studies, we
believe a requirement for four milestones would serve the purposes of
this potential change. In our opinion, this potential change could
improve program execution if the milestones are specified and aligned
with demonstrating knowledge at the appropriate time and if they are
tied to certifications to the DAE.
GAO has previously reported that the acquisition management framework
allows programs to move forward without adequate knowledge at critical
junctures. DOD programs regularly skipped a pre-acquisition milestone
A, passed milestone B with limited systems engineering knowledge and
critical technologies that were immature and not ready for product
development. Programs do not undergo another milestone review until
production begins at milestone C, sometimes up to 10 years later. These
practices left programs with limited knowledge and lengthy time gaps
between critical reviews and few opportunities for crucial management
oversight and decision making concerning the department's investment
and the warfighter's continuing need. DOD issued updated guidance
concerning the operation of its DAS in December 2008 to help address
these issues. However, it is too soon to determine how successful these
changes will be in addressing those issues.
GAO has advocated that OSD exert stronger leadership in managing DOD's
acquisition programs and has specifically recommended that OSD (1)
develop and implement an integrated enterprise-level investment
strategy that is based on a joint assessment of warfighting needs and a
full set of potential and viable alternative solutions within realistic
and affordable budget projections; (2) ensure DOD's acquisition
programs are established with sound business cases prior to entering
systems development; (3) establish a knowledge-based acquisition
process; and (4) improve leadership and accountability.
DOD's recently revised acquisition policy addresses some of these
needed changes to some degree; for example, the policy now encourages
the development of knowledge earlier in the acquisition process.
However, consistent with our prior work, we believe an additional
milestone review is needed midway through engineering and manufacturing
development--during the product development phase--to show the design
is stable, and that adequate knowledge exists and demonstrates the
weapon system design will perform as expected. We believe this is
necessary to reduce risk and optimize the department's investment.
According to DOD, the recently revised process achieves this. In
addition to the three milestone reviews, the revised instruction
requires a formal MDA decision review--the Post Critical Design Review
Assessment--during the Engineering and Manufacturing Development Phase.
Additionally, if a program did not conduct a Preliminary Design Review
prior to the milestone B decision, the MDA will conduct a Post
Preliminary Design Review Assessment following the Preliminary Design
Review. According to DOD, these MDA assessments are designed to achieve
the same substantive benefit as a milestone review, and will result in
decisions that will be documented in acquisition decision memoranda.
Figure 1 compares DOD's revised acquisition process and GAO's
recommended knowledge-based, evolutionary process.
Figure 1: Comparison of DOD's Acquisition Process and GAO's Knowledge-
Based Acquisition Process:
[Refer to PDF for image: illustration}
Current DOD process:
Pre Systems Acquisition:
Materiel solution analysis: Milestone A;
Materiel development decision (decision point);
Technology Development: Milestone B;
Systems Acquisition:
Engineering and manufacturing development:
Post Prelim design review assessment (Conditional decision point [held
if preliminary design review is after Milestone B]);
System integration design;
Post critical design review assessment (decision point);
System capacity and manufacturing process demo;
Milestone C;
Initial Operational Capacity: Production and deployment:
Low rate initial production and initial operational test and
evaluation;
Full-rate production decision review (decision point);
Full-rate production and deployment;
Sustainment:
Full Operational Capability: Operations and support:
Lifecycle sustainment;
Disposal.
Knowledge-Based Process:
Concept and technology development:
Materiel development decision (decision point);
Milestone A;
Prelim Design review (design review);
Milestone B, Program initiation; Knowledge point 1; Technology
available, time, funding, and other resources match requirements.
Product development:
Critical design review (design review);
Post critical design review assessment (decision point);
Milestone B'; Knowledge point 2; Design is stable and meets performance
requirements;
Milestone C; Knowledge point 3; Production can meet costs, schedule,
and quality targets;
Production:
Initial Operational Capacity:
Low rate initial production and initial operational test and
evaluation;
Full-rate production design review (decision point);
Full-rate production and deployment;
Full Operational Capacity:
Lifecycle sustainment;
Disposal.
Source: GAO analysis of DOD's acquisition process and GAO's best
practices.
[End of figure]
GAO's work has repeatedly demonstrated the need for DOD to follow a
knowledge-based acquisition process and to develop knowledge earlier in
the acquisition process, basing the decision to enter the system
development phase of the acquisition process with preliminary design
knowledge in hand. We agree with the department's new policy that
prescribes conducting milestones A, B, and C. We believe a fourth
milestone, centered at each program's Post Critical Design Review
Assessment midway through development, is also needed. With that in
mind, each program would have to pass through the following milestone
reviews:
* Milestone A--A review to determine the weapon system concept and
identify the technologies available to meet a validated requirement for
a new weapon system;
* Milestone B (equivalent to Knowledge Point 1)--A review to determine
through prototyping and disciplined and robust systems engineering
practices that a match is made between requirements and available
mature technology; funding, and other resources;
* Milestone B' (equivalent to Knowledge Point 2)--A review of
demonstrated systems engineering, prototyping, and test data that the
system's design is stable and meets performance requirements; and:
* Milestone C (equivalent to Knowledge Point 3)--A review to
demonstrate that manufacturing processes are in control and that the
system can be produced within cost, schedule, and quantity targets.
Consequently, shorter, more frequent acquisition program milestones
could improve DOD's accountability and the acquisition process by
providing OSD's decision makers with improved data upon which to make
decisions and more frequent opportunities to exercise oversight over
these expensive investments. Recommendations from prior studies on DOD
acquisition support realignment along these lines and include:
* The DAPA panel reported that the chief risk reduction opportunities
exist between milestone A and milestone B. Nonetheless, DOD focuses
much of its efforts on reaching and passing milestone B, which leads to
starting programs with immature technologies and a long cycle of
program instability, requirements and budget changes, schedule delays,
and repeated rebaselining.[Footnote 18] The DAPA panel recommended DOD
realign the milestone B decision to occur at preliminary design review
to reduce program risk by increasing knowledge prior to milestone B.
* The Institute for Defense Analysis (IDA) in its Acquisition
Initiatives--Phase II report recommended USD (AT&L) should reinstitute
MDA oversight of the milestone A process so that a sound systems
engineering management plan is put in place for each alternative that
will be pursued into a formal risk reduction/demonstration/validation
phase, leading to a possible milestone B formal initiation of a major
new acquisition program.[Footnote 19] The report also suggested there
should be a very high bar for waivers of an inadequate systems
engineering plan at formal program initiation at milestone B.
Regarding this potential change, 28 of the 52 experts we interviewed
who commented on this potential change believed that it would not
improve the process. Some of the experts who told us the potential
change would not improve the acquisition process indicated that the
current process and oversight reviews are adequate and noted that more
reviews do not always equate to improved outcomes. However, many of the
experts we interviewed also told us that this potential change could
improve the process depending upon how it is implemented. Some of the
experts told us that additional milestones should be accompanied with a
more uniform data gathering process, quantifiable metrics,
significantly more systems engineering knowledge, and incremental
development strategies.
We believe DOD's updated DOD Instruction 5000.02 aligns with the intent
of this potential change and GAO's best practices model. However, to be
more effective, we believe the revised policy should be implemented in
conjunction with the potential change to require the milestone decision
authority to specify, at the time of milestone B approval, the period
that will be required to deliver IOC to the relevant combatant
commanders, which is discussed in the following report segment. In
combination, these potential changes should result in more frequent
milestone reviews with shorter timeframes between milestones--both as a
result of the adoption of the milestone B' and the adoption of a more
incremental, evolutionary acquisition approach--helping to focus
development on more manageable, less aggressive increments of
capability and facilitating accountability. Additionally, strong
consistent leadership that demands and rewards the proper use of
knowledge by programs is needed.
In its latest Defense Acquisition Transformation Report to Congress,
DOD states its acquisition policy encourages the development of
knowledge earlier in the acquisition process and has placed clear
emphasis on the need for more systems engineering analysis and
prototyping earlier in the process. For example, the policy now
requires that programs involving immature technologies undergo a
milestone A review and requires a disciplined preliminary design review
prior to or soon after the start of a program at milestone B. While the
recently updated policy does not create new milestones for design
demonstration, it aligns existing milestones and other decision points
with obtaining critical knowledge.
Requiring the MDA to Specify at the Time of Milestone B Approval the
Period of Time That Will Be Required to Deliver an IOC to the Relevant
Combatant Commanders:
We believe that this potential change, if modified to require a time-
certain development period not to exceed 6 years would significantly
improve acquisition outcomes.[Footnote 20] However, it can only be
successful if the department uses an incremental, evolutionary, and
knowledge-based approach to weapon system acquisition programs. New
programs would attempt to develop and produce capabilities that are
achievable within 6 years, requiring a new program to achieve future
incremental capabilities. Limiting development to 5-or 6-year
increments (or shorter increments) would force smaller, more manageable
commitments in capabilities for each increment and make costs and
schedules more predictable. In addition, it could help to stabilize
funding by reducing the need to make up for program deviations.
In today's acquisition environment, it sometimes takes more than a
decade to produce an initial capability. As a result, programs face
increased risks that circumstances will change around them--for
example, technology changes, requirements changes, and budget
pressures--driving up costs and delaying fielding.[Footnote 21]
Requiring a milestone decision authority to specify the period required
to deliver an IOC, essentially incorporating a time-certain philosophy,
could help DOD deliver useful military capabilities in a more timely
manner. Previous recommendations and evidence to support this change
include the following:
* GAO recommended in 2007 that DOD take decisive actions to shorten
cycle times in delivering needed combat capabilities to the warfighter
to include adopting a time-certain development cycle that can deliver
an increment of new capability within 5 to 6 years after the start of
system design and development[Footnote 22] We have cited past
acquisitions, such as the F-16 and Small Diameter Bomb, as examples of
successful incremental development.
* The DAPA panel recommended that DOD make program schedules a key
performance parameter as well as shift its acquisition approach to
incorporate a time-certain development strategy. Inserting time as a
key performance parameter would encourage evolutionary development and
force more detailed up-front requirements statements.
* The Beyond Goldwater-Nichols work found that current initiatives may
prove fruitful in the next few years, for example more aggressively
managing risk at the early stages of the development process by
assessing technology maturity more accurately and giving increased
weight to getting capabilities into the field faster.
* The Packard Commission reported that an unreasonably long acquisition
cycle--10 to 15 years for major weapon systems--is the central problem
from which most other acquisition problems stem. It leads to
unnecessarily high costs of development--a 10 year acquisition cycle is
clearly more expensive than a 5 year cycle. It also leads to obsolete
technology in fielded equipment. Because long-term forecasts are
uncertain at best, users tend to err on the side of overstating the
threat.
More than half of the experts we interviewed who commented on this
potential change (26 of 49) believed it could improve the acquisition
process. Several of those experts told us this change could be
effective if it is aligned with evolutionary acquisition and time-
certain development principles. Several of the experts who commented
that this potential change may not improve acquisition results told us
that historically, DOD program managers (PM) do not control
requirements and resources or cost estimates are not realistic.
To be most effective, we believe this potential change should be
implemented in conjunction with the potential change to revise the
acquisition process by establishing shorter, more frequent acquisition
program milestones. Furthermore, the success of implementing time-
certain development may be contingent upon several factors. For
example, PMs must be empowered to control their program and defer
additional requirements until a future increment; there must be unity
among leadership in support of the program; and PM tenure agreements
need to bridge the gap between major program milestones to facilitate
accountability.
In its latest Defense Acquisition Transformation Report to Congress,
DOD agreed with the intent, but not the language, of the DAPA panel's
findings on time-certain acquisition and stated that it was
implementing that intent by selecting time-defined approaches
consistent with the users' needs at the concept decision point in the
acquisition process.
Establishing a New Office to Provide Independent Cost and Performance
Estimates:
We believe that establishing an independent assessment office could
enhance the cost estimates produced if the organization is required to
report the results of knowledge-based cost assessments for MDAPs
directly to Congress and the Secretary of Defense, and is given the
resources needed to develop thorough estimates.
The Cost Analysis Improvement Group (CAIG) is the organization
currently responsible for conducting independent cost assessments for
MDAPs and reports to the department's program analysis and evaluation
organization (PA&E).[Footnote 23] However, USD (AT&L)--through
oversight of the DAS--is the ultimate customer for CAIG estimates, and
ultimately decides whether the CAIG independent cost estimate will be
used as the basis to fund the program.[Footnote 24] Establishing an
office of independent assessment that reports its results directly to
Secretary of Defense and Congress, similar to the office of the
Director of Operational Test and Evaluation (DOT&E) would more closely
align the CAIG with the Secretary of Defense and USD (AT&L) providing a
new level of accountability and the opportunity to be more fully
integrated into the acquisition management framework.[Footnote 25]
DOD's inability to allocate funding effectively to programs is largely
driven by its acceptance of unrealistic cost estimates and a failure to
balance needs based on available resources. The uniqueness of each
program, the lack of sufficient knowledge about system requirements,
technology and design maturity, and the limited analytical tools
available are often cited as factors that contribute to optimistic
forecasts of development costs. Previous recommendations and evidence
to support this change include the following:
* GAO has reported:
- Development costs for major acquisition programs are often
underestimated at program initiation sometimes by 30 to 40 percent.
[Footnote 26] Additionally, weapon system programs are initiated
without sufficient knowledge about system requirements, technology, and
design maturity and that lacking such knowledge, managers rely on
assumptions that are consistently too optimistic, exposing programs to
unnecessary risks and ultimately cost growth and schedule delays.
[Footnote 27]
- The policy concerning independent cost estimates does not require
that the estimates be relied upon to set budgets, only that they be
considered at key acquisition decision points.[Footnote 28]
Additionally in 2008, we reported that for many of the 20 major weapons
programs we reviewed, the initial approved development baseline funding
was lower than the CAIG's cost estimate.[Footnote 29]
- The independence of DOT&E and its resulting authority to report
directly to Congress is the foundation of its effectiveness. That
independence along with its legislative mandate provides sufficient
freedom and authority to exercise effective oversight of the
operational testing and evaluation of new systems.[Footnote 30]
* The DAPA panel recommended adjusting program estimates to reflect
high confidence levels--defined as a program with an 80 percent chance
of completing development at or below estimated cost.
Overall, 46 of the 49 experts we interviewed believe a new office to
provide cost and performance estimates is not needed. Many of the
experts we interviewed believe the CAIG is independent. Some suggested
that the lack of resources within the CAIG, inadequate systems
engineering, long acquisition time frames, or adoption of immature
technologies contribute to poor cost estimates. Additionally, according
to one CAIG official, its cost estimates and independence could be
improved if the organization had more personnel resources with the
appropriate systems engineering expertise to conduct independent
systems engineering performance reviews that could then be used to
prepare improved cost estimates and risk determinations.
It is critical to note that for this change to be effective and result
in more realistic, reliable, and independent cost estimates, we believe
DOD needs to improve its cost estimating through a disciplined,
knowledge-based approach that reduces risk and uncertainty over time by
conducting detailed systems engineering early in weapons systems
development processes, shortening acquisition periods, and adopting
more mature technologies. Although many experts believed that this
change would not improve the acquisition process, we believe that
establishing an independent assessment office that provides independent
cost estimates and reports its results directly to the Secretary of
Defense and to Congress would enhance independence by better informing
stakeholders outside of the DOD acquisition community. We see no reason
why the CAIG couldn't form the basis of the suggested organization.
Finally, that office should have the appropriate resources to conduct
such knowledge-based cost assessments.
In its latest Defense Acquisition Transformation Report to Congress,
DOD stated that USD (AT&L) is focusing on prototyping to reduce
technical risk and inform costs estimates. DOD is also working to
improve control over developmental timelines through reduced risk,
better knowledge, and technical maturity.
Requiring Certifications of Program Status to the DAE and Congress
Prior to Milestone Approvals:
The National Defense Authorization Acts of Fiscal Years 2006, 2007, and
2008 changed certification standards, requiring additional
certification requirements for major defense acquisition programs
before proceeding to milestones A or B.[Footnote 31] In addition USD
(AT&L) issued a memo in May, 2007 implementing Program Manager
Agreements to establish an annual contract between the program manager
and the acquisition and requirements/resource officials.[Footnote 32]
These are binding agreements that detail the progress the program is
expected to make during the year and the resources the program will be
provided to reach these goals. USD (AT&L) also requires PMs to sign
tenure agreements with the SAE stipulating that their MDAP tenure will
correspond to the next major milestone review closest to 4 years. In
our opinion, when certifications are used in conjunction with Program
Manager Agreements, they provide more program visibility and a stronger
means for holding the PM and the other key management officials
accountable for the decisions they make managing their programs.
Furthermore, DOD's strategy to improve acquisition outcomes rests not
only with providing support for individual PMs, but more importantly on
creating a department-wide environment that promotes stability,
accountability, and more informed decision making.[Footnote 33] Many of
the experts we interviewed cited the new legislation--which requires
milestone A and B certifications--as adequate to meet the needs that
this change targets. We agree.
Modifying the Role Played by Chiefs of Staff of the Armed Forces in the
Requirements, Resource Allocation, and Acquisition Processes:
It is difficult to say whether the potential change, as presented, will
generate better acquisition outcomes. In our view, this potential
change requires further definition and more study. It is unclear which
roles of the Chiefs of Staff of the Armed Forces should be changed and
how these roles should be modified to improve acquisition outcomes.
However, we note that the Beyond Goldwater-Nichols body of work
concluded that a consequence of implementing the 1986 Goldwater-Nichols
legislation is that the military service Chiefs of Staff no longer play
a large role in managing the execution of their acquisition programs.
[Footnote 34] According to the DAPA panel, the DSB 2005 Summer Study on
Transformation, and the Beyond Goldwater-Nichols body of work,
restoring the authority to manage the execution of their acquisition
programs to the military service chiefs could in turn restore
accountability for acquisition results. A chief argument of each of
these studies is that by realigning acquisition programs through a
uniformed services chain of command, the services' acquisition
organizations would be more streamlined and accountability would be
easier to enforce. A large majority of the experts we interviewed who
commented on this potential change (35 of 43) told us that modifying
the role of the Chiefs of Staff of the Armed Forces to provide them
authority to manage the execution of their acquisition programs would
not improve the acquisition process. Many of the experts told us the
military services already have control over the requirements and
resource allocation processes, and should not also have control over
the acquisition process. On the other hand, some experts commented that
changing the roles of the military Chiefs of Staff might improve the
process if done in conjunction with making the combatant commanders
responsible for determining requirements, a condition that would take
place if the potential change to allow combatant commanders more
influence concerning their long-term needs is made.
Changes Needed in Acquisition Environment, Culture, and Incentives:
Our discussions with acquisition experts indicated that the potential
changes may not achieve the desired improvement in acquisition outcomes
unless they are accompanied with changes in the overall acquisition
environment, its culture, and the incentives provided for success.
Specifically, some of the DOD executives we interviewed--both current
and former--emphatically stated that these changes will not improve the
acquisition system unless the department can significantly improve its
prioritization of needs so that there are fewer programs competing for
available dollars. Other experts we interviewed questioned the
potential of all seven changes to improve acquisition outcomes. A few
of the experts believe some process stakeholders involved in setting
requirements, providing funding, or executing programs would not agree
to the changes because they like the process as it is or do not want
their roles in the process to change. One DOD acquisition official told
us that DOD's acquisition policies are good; they just are not
implemented as intended. Another expert--a former high-ranking DOD
executive--categorically stated that the acquisition system is a system
in equilibrium with rules and regulations that allow leadership to
deviate from the process when they want. Until members of Congress and
DOD leadership collectively realize this is the case as well as the
fact that they all like the system as it is, nothing will change. Many
believe it will take strong, consistent strategic leadership to change
the existing culture and environment.
Conclusions:
We believe that six of the seven potential changes to DOD's acquisition
framework could improve acquisitions outcomes. Although many of the
experts we interviewed believed the potential changes may not achieve
the desired improvement in acquisition outcomes, they provided a wide
range of opinions concerning the pros and cons of each potential
change. Nonetheless, based on the results of this review and our work
and observations over the years of the DOD acquisition systems'
imperviousness to change, we do not believe that the changes are likely
to achieve the desired results unless they are (1) implemented together
along with other more systemic and sweeping changes and (2) accepted by
all of the stakeholders and process owners in this complex framework of
requirement setting, funding, and program execution. We point
specifically to comments made by those current and former DOD
executives who know the environment best. Some of these experts stated
that (1) the requirements setting process (JCIDS) takes too long;
generally produces requirements for capabilities that aren't achievable
in the short term, and often doesn't provide optimal results; (2) the
funding process historically accepts cost and schedule estimates based
on optimistic assumptions and little knowledge about the resources
needed to develop the weapon system required; and (3) the acquisition
process moves forward with those poor estimates, often adding
requirements and increasing cost as the development program progresses.
Finally, some of the experts we interviewed pointed to an overall lack
of discipline in the process when it comes to making resource-
constrained decisions. All of these factors sustain an environment of
high risk, low reward, and poor outcomes in the form of higher costs,
late deliveries, and less than optimal capabilities fielded to the
warfighter.
Potential incremental changes to the process are not likely to produce
lasting and meaningful improvement until the acquisition environment
and all of its process owners accept that risk in product development
must be lowered by prioritizing the needs of national security jointly
and with discipline; systems engineering knowledge is used to constrain
requirements prior to program initiation; funding programs are based on
knowledge and independent cost estimates with high levels of
confidence; and evolutionary, knowledge-based practices and processes
to reduce technology, design, and manufacturing risk are employed.
Nonetheless, we believe the potential changes discussed in this report
could have significant positive effect, but not if the environment, its
incentives, and the culture continue to resist change.
Agency Comments and Our Evaluation:
We provided a draft of this correspondence to DOD for comment. The
department did not provide a written response, but provided technical
comments which we have incorporated as appropriate.
We are sending copies of this report to the Secretary of Defense; the
Under Secretary of Defense for Acquisition, Technology and Logistics;
the Chairman of the Joint Chiefs of Staff; and the Secretaries of the
Air Force, Army, and Navy. This report will also be available at no
charge on GAO's Web site at [hyperlink, http://www.gao.gov]. If you
have any questions about this report or need additional information,
please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact
points for our Offices of Congressional Relations and Public Affairs
may be found on the last page of this report.
Key contributors to this report were Bruce Thomas, Assistant Director;
Claudia Dickey, Analyst-in-Charge; Greg Campbell; Tina Cheng; Rosa
Johnson; John Krump; Jean McSween; Robert Miller, and J. Andrew Walker.
Signed by:
Michael J. Sullivan, Director:
Acquisition and Sourcing Management:
List of Committees:
The Honorable Carl Levin:
Chairman:
The Honorable John McCain:
Ranking Member:
Committee on Armed Services:
United States Senate:
The Honorable Daniel K. Inouye:
Chairman:
The Honorable Thad Cochran:
Ranking Member:
Subcommittee on Defense:
Committee on Appropriations:
United States Senate:
The Honorable Ike Skelton:
Chairman:
The Honorable John M. McHugh:
Ranking Member:
Committee on Armed Services:
House of Representatives:
The Honorable John P. Murtha:
Chairman:
The Honorable C.W. Bill Young:
Ranking Member:
Subcommittee on Defense:
Committee on Appropriations:
House of Representatives:
[End of section]
Enclosure I: Scope and Methodology:
The 2008 National Defense Authorization Act (NDAA) directed GAO to
report on the feasibility and advisability of seven potential changes
to improve the Department of Defense's (DOD) acquisition framework for
Major Defense Acquisition Programs (MDAPs).[Footnote 35] See enclosure
II for a list of the seven potential changes. Our objective was to
determine the feasibility and advisability of the potential
modifications.
To determine the feasibility and advisability of the seven potential
changes included in the mandate, we considered the results of GAO's own
body of work and relevant works produced by DOD and others, and the
results of our interviews with 53 acquisition experts. We identified
and analyzed recommendations from prior GAO and DOD work that address
Congress's potential changes, and considered the opinions of the 53
experts we interviewed. Enclosure IV provides a list of DOD acquisition
studies and GAO reports used to inform our work. We reviewed relevant
law and DOD policies, directives, and guidance as well as the history
of the department's DAS and management framework.
We conducted semistructured interviews with 53 identified experts. We
selected the experts from among currently serving senior DOD
acquisition officials, senior military officers involved in setting
requirements for the joint staff, the military services, the Combatant
Commands (COCOM), individuals who formerly served as senior DOD
acquisition officials, participants in previous reviews of the
organization and structure of DOD for the acquisition of major weapon
systems, academicians, and other subject matter experts. The interview
questions were designed to determine the expert's opinions concerning
the seven potential changes, as well as determine other systemic
problems in DOD's overall acquisition management framework and other
potential changes that might result in better acquisition outcomes. We
pretested the questions during initial interviews with DOD officials
located in Dayton, Ohio and at the Pentagon. Based on the results of
these interviews, we refined our questions to ensure they were open
ended and allowed the respondents the opportunity to provide not only
opinions on the potential changes, but also their reasoning.
Additionally, the nature of the questions allowed the experts to
provide additional opinions concerning problems with DOD's acquisition
processes and potential solutions. To summarize and analyze the
information obtained during our interviews, we recorded and verified
the experts' responses in a Web-based data collection instrument. The
content for the responses were then coded and verified to determine the
extent to which the experts believed the seven potential changes,
would, would not, or might improve DOD's acquisition processes, and to
identify other potential challenges and solutions.
We conducted this performance audit from January 2008 to February 2009
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.
[End of section]
Enclosure II: Potential Modifications to the DOD Acquisition Process
from the National Defense Authorization Act for 2008:
1. Revising the acquisition process for major defense acquisition
programs by establishing shorter, more frequent acquisition program
milestones;
2. Requiring certifications of program status to the defense
acquisition executive and Congress prior to milestone approval for
major defense acquisition programs;
3. Establishing a new office (to be known as the "Office of Independent
Assessment") to provide independent cost estimates and performance
estimates for major defense acquisition programs;
4. Requiring the milestone decision authority for a major defense
acquisition program to specify, at the time of Milestone B approval, or
Key Decision Point B approval, as applicable, the period of time that
will be required to deliver an IOC to the relevant combatant
commanders;
5. Establishing a material solutions process for addressing identified
gaps in critical warfighting capabilities, under which process the
Under Secretary of Defense for Acquisition, Technology, and Logistics
circulates among the military departments and appropriate Defense
Agencies a request for proposals for technologies and systems to
address such gaps;
6. Modifying the role played by chiefs of staff of the Armed Forces in
the requirements, resource allocation, and acquisition processes; and:
7. Establishing a process in which the commanders of the combatant
commands assess, and provide input on, the capabilities needed to
successfully accomplish the missions in the operational and contingency
plans of their commands over a long-term planning horizon of 15 years
or more, taking into account expected changes in threats, the geo-
political environment, and doctrine, training and operational concepts.
[End of section]
Enclosure III: Comparison of DOD‘s Current and Prior Acquisition
Processes:
Figure: Comparison of DOD‘s Current and Prior Acquisition Processes:
[Refer to PDF for image: illustration]
Current DOD Process (as of December 2008):
Pre Systems Acquisition:
Materiel solution analysis:
Materiel development design (decision point);
Milestone A;
Technology DEvelopment;
Prelim design review (design review) (held here or after Milestone B
hold system integration design review);
Milestone B;
Systems Acquisition:
Engineering and manufacturing development:
Post prelim design review assessment (conditional decision point [held
if preliminary design review is after milestone B]);
System integration design (design review);
Post critical design review assessment (decision point);
System capability and manufacturing process demo;
Milestone C;
Initial Operational Capability:
Production and deployment;
Low-rate initial production and initial operational test and
evaluation;
Full-rate production decision review (decision point);
Full-rate production and deployment;
Sustainment:
Full Operational Capability:
Lifecycle sustainment;
Disposal.
Prior DOD process (May 2003):
Pre Systems Acquisition:
Concept refinement;
Concept decision (decision point);
Milestone A;
Technology development;
Milestone B;
Systems Acquisition:
System development and demonstration;
Prelim design review (design review);
System integration;
Critical design review (design review);
Design readiness review (decision point);
System demonstration;
Milestone C;
Initial Operational Capability:
Production and deployment;
Low-rate initial production and initial operational test and
evaluation;
Full-rate production decision review (decision point);
Full-rate production and deployment;
Sustainment:
Full Operational Capability:
Lifecycle sustainment;
Disposal.
Source: GAO analysis of DOD's current and prior acquisition process.
[End of figure]
Major differences between the 2003 and 2008 versions of the DOD
instruction 5000.02 are:
* The Materiel Development Decision (MDD) replaces the Concept
Decision. A MDD is required regardless of where the program intends to
enter the acquisition process.
* The Materiel Solution Analysis (MSA) Phase replaces the Concept
Refinement Phase. The Joint Capabilities Integration and Development
System (JCIDS) process no longer includes an Analysis of Materiel and
Non-Materiel Alternatives. Non-materiel solutions will be handled via
JCIDS; however, all analysis of alternative materiel solutions will be
accomplished by the Analysis of Alternatives (AOA) during MSA. The
Milestone Decision Authority (MDA) will approve the materiel solution
at milestone A.
* The Technology Development Phase (TD) now includes a mandatory
requirement for competitive prototyping of the system or key-system
elements. A Preliminary Design Review (PDR) may be conducted for the
candidate designs, and a PDR report will be provided to the MDA with
recommended requirements trades. The final Capabilities Development
Document should contain trade-offs determined during the TD phase.
* The Engineering & Manufacturing Development (EMD) phase replaces
System Development and Demonstration. The two major efforts have been
renamed. The Program Manager must provide a PDR report, and must
provide a Critical Design Review (CDR) report to the MDA.
* A Post-CDR Assessment replaces the Design Readiness Review. The MDA
will determine if the results of the CDR warrant continuing EMD to
milestone C.
[End of section]
Enclosure IV: Acquisition Studies:
Studies on DOD Acquisition:
President's Blue Ribbon Commission on Defense Management, A Formula for
Action: A Report to the President on Defense Acquisition. The Packard
Commission. Washington, D.C.: April 1, 1986. [hyperlink,
http://www.ndu.edu/library/pbrc/pbrc.html].
Defense Acquisition Performance Assessment Report. Department of
Defense Acquisition Performance Assessment Panel. Washington, D.C.:
January 27, 2006. [hyperlink, http://www.acq.osd.mil/dapaproject].
Beyond Goldwater-Nichols: Defense Reform for a New Strategic Era, Phase
1 Report. Center for Strategic and International Studies. Washington,
D.C.: March 2004. [hyperlink, http://www.csis.org/isp/bgn].
Beyond Goldwater-Nichols: U.S. Government and Defense Reform for a New
Strategic Era, Phase 2 Report. Center for Strategic and International
Studies. Washington, D.C.: July 2005. [hyperlink,
http://www.csis.org/isp/bgn].
The Future of the National Guard and Reserves: Beyond Goldwater-
Nichols: Department of Defense Acquisition and Planning, Programming,
Budgeting, and Execution System Reform, Phase 3 Report. Center for
Strategic and International Studies. Washington, D.C.: August 2006.
[hyperlink, http://www.csis.org/isp/bgn].
Beyond Goldwater-Nichols: Department of Defense Acquisition and
Planning, Programming, Budgeting, and Execution System Reform, An
Annotated Brief. Center for Strategic and International Studies.
Washington, D.C.: August 1, 2006. [hyperlink,
http://www.csis.org/isp/bgn].
Transformation: A Progress Assessment Volume I. Defense Science Board,
2005 Summer Study. Washington, D.C.: February 2006. [hyperlink,
http://www.acq.osd.mil/dsb/reports.htm#2005].
Defense Acquisition Transformation, Report to Congress. John Warner
National Defense Authorization Act Fiscal Year 2007, Section 807.
Washington, D.C.: March 2008: [hyperlink,
http://www.acq.osd.mil/at/docs/804_defense_acquisition_transformation_re
port_mar_2008.pdf].
Report to Congress on Program Manager Empowerment and Accountability.
Department of Defense. Washington, D.C.: August 2007. [hyperlink,
http://www.acq.osd.mil/at/docs/Program_Manager_Empowerment_and_Accountab
ility.pdf].
Joint Defense Capabilities Study, Improving DOD Strategic Planning,
Resourcing and Execution to Satisfy Joint Capabilities. Department of
Defense. Washington, D.C.: January 2004. [hyperlink,
http://www.acq.osd.mil/actd/articles/JointDefenseCapabilitiesStudyFinalR
eport_January2004.pdf].
Related GAO Products:
Defense Acquisitions: Fundamental Changes Are Needed to Improve Weapon
Program Outcomes. [hyperlink,
http://www.gao.gov/products/GAO-08-1159T]. Washington, D.C.: September
25, 2008.
Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve
Major Weapon System Program Outcomes. [hyperlink,
http://www.gao.gov/products/GAO-08-619]. Washington, D.C.: July 2,
2008.
Defense Acquisitions: Better Weapon Program Outcomes Require
Discipline, Accountability, and Fundamental Changes in the Acquisition
Environment. [hyperlink, http://www.gao.gov/products/GAO-08-782T].
Washington, D.C.: June 3, 2008.
Defense Acquisitions: Results of Annual Assessment of DOD Weapon
Programs. [hyperlink, http://www.gao.gov/products/GAO-08-674T].
Washington, D.C.: April 29, 2008.
Defense Acquisitions: Assessments of Selected Weapons Programs.
[hyperlink, http://www.gao.gov/products/GAO-08-467SP]. Washington,
D.C.: March 31, 2008.
Defense Management: DOD Needs to Reexamine Its Extensive Reliance on
Contractors and continue to Improve Management and Oversight.
[hyperlink, http://www.gao.gov/products/GAO-08-572T]. Washington, D.C.:
March 11, 2008.
Defense Contracting: Additional Conflict of Interest Safeguards Needed
for Certain DOD Contractor Employees. [hyperlink,
http://www.gao.gov/products/GAO-08-169]. Washington, D.C.: March 7,
2008.
Best Practices: Increased Focus Requirements and Oversight Needed to
Improve DOD's Acquisition Environment and Weapon System Quality.
[hyperlink, http://www.gao.gov/products/GAO-08-294]. Washington, D.C.:
February 1, 2008.
Defense Acquisitions: Department of Defense Actions on Program Manager
Empowerment and Accountability. [hyperlink,
http://www.gao.gov/products/GAO-08-62R]. Washington, D.C.: November 9,
2007.
Best Practices: An Integrated Portfolio Management Approach to Weapon
System Investments Could Improve DOD's Acquisition Outcomes.
[hyperlink, http://www.gao.gov/products/GAO-07-388]. Washington, D.C.:
March 30, 2007.
DOD Acquisition Outcomes: A Case for Change. [hyperlink,
http://www.gao.gov/products/GAO-06-257T]. Washington, D.C.: November
15, 2006.
Best Practices: Stronger Practices Needed to Improve DOD Technology
Transition Processes. [hyperlink,
http://www.gao.gov/products/GAO-06-883]. Washington, D.C.: September
14, 2006.
Best Practices: Better Support of Weapon System Program Managers Needed
to Improve Outcomes. [hyperlink,
http://www.gao.gov/products/GAO-06-110]. Washington, D.C.: November 30,
2005.
[End of section]
Footnotes:
[1] Pub. L. No. 110-181, § 813 (2008).
[2] In assessing the seven potential changes, we defined feasible
changes as those changes that are reasonable to implement, and
advisable changes as those that could improve MDAP outcomes.
[3] DOD issued updated guidance (DOD Instruction 5000.02) concerning
the operation of its defense acquisition system in December 2008. We
conducted our interviews with the experts prior to DOD issuing the
updated guidance. We believe there is nothing in the updated guidance
that would change the opinions provided to us by the experts or the
results of our engagement.
[4] Pub L. No. 109-163, § 801 (2006), Pub. L. No. 109-364, § 805
(2007), Pub. L. No. 110-181, § 943 (2008).
[5] Federal statutes, located primarily in title 10 of the United
States Code, along with the Federal Acquisition Regulation, the Defense
Federal Acquisition Regulation Supplement, defense regulation and
guidance provide the framework for the defense acquisition system.
[6] JCIDS falls under the oversight of the Chairman, Joint Chiefs of
Staff and is defined in CJCSI 3170.01F. PPBES falls under the oversight
of the Deputy Secretary of Defense and is defined in DOD Financial
Management Regulation 7045.14-R. DAS falls under the oversight of the
USD (AT&L) and is defined in DOD Directive 5000.01 and DOD Instruction
5000.02.
[7] According to DOD, the MDA is the designated individual who has
overall responsibility for an investment. This person has the authority
to approve an investment's progression in the acquisition process and
is responsible for reporting cost, schedule, and performance results.
For example, the milestone decision authority for a major defense
acquisition program, when not delegated to the component level, is the
USD (AT&L).
[8] The DAE is the USD (AT&L) who has responsibility for supervising
the DAS. The DAE takes precedence on all acquisition matters after the
Secretary and the Deputy Secretary of Defense. In the military
departments, the officials delegated as CAEs (also called Service
Acquisition Executives (SAE)) are respectively, the Assistant Secretary
of the Army (Acquisition, Logistics, and Technology) (ASA (AL&T)), the
Assistant Secretary of the Navy (Research, Development and Acquisition)
(ASN (RD&A)), and the Assistant Secretary of the Air Force
(Acquisition) (ASAF (A)). The CAEs are responsible for all acquisition
functions within their components.
[9] DOD, 2006 Quadrennial Defense Review Report (Washington, D.C.,
February 2006); CSIS, Beyond Goldwater-Nichols: U.S. Government and
Defense Reform for a New Strategic Era, vol. 2 (Washington, D.C., July
2005); DSB, Defense Science Board Summer Study on Transformation: A
Progress Assessment, vol. 1. (Washington, D.C., February 2006); DOD,
Defense Acquisition Performance Assessment Project Report (Washington,
D.C., January 2006); DOD, A Quest for Excellence: Final Report to the
President by the President's Blue Ribbon Commission on Defense
Management (Washington, D.C., June 1986).
[10] See also the Goldwater-Nichols Department of Defense
Reorganization Act of 1986, Pub. L. No. 99-433.
[11] Pub. L. No. 109-163 § 801 (2006); Pub. L. No. 109-364 § 805
(2007), and Pub. L. No. 110-181 § 943 (2008).
[12] The IPL defines the COCOM's highest priority capability gaps for
the near term, including shortfalls that may adversely affect COCOM
missions. These lists are prepared by the COCOMs and submitted to the
Joint Chiefs of Staff and then to the Joint Requirements Oversight
Council (JROC). After JROC reviews the IPL and decides to endorse the
requirements, the approved document is given to the Vice Chairman of
the Joint Chiefs and finally the Chairman of the Joint Chiefs for final
review. The document is analyzed against established programs within
the services to see if needs can be met through existing programs. If a
capability is needed that does not have a corresponding program in
development, then the Joint Staff will determine the most appropriate
organization to further develop the requirement.
[13] GAO, Defense Acquisitions: DOD's Requirements Determination
Process Has Not Been Effective in Prioritizing Joint Capabilities,
[hyperlink, http://www.gao.gov/products/GAO-08-1060]. (Washington,
D.C.: Sept. 25, 2008).
[14] The Joint Staff, Joint Requirements Oversight Council. JROCM 130-
08, June 20, 2008. Assignment of Joint Potential Designators and
Coordination by Combatant Commands on Capabilities Documents.
[15] JCA provide a common medium to discuss and describe capabilities
across many DOD activities and processes. For example, Battle Space
Awareness is the Joint Capability Area that encompasses the ability to
understand dispositions and intentions as well as characteristics and
conditions of the operational environment that bear on national and
military decision making.
[16] Concept decision (now the Material Development Decision under the
updated 5000.02 instruction) is the first decision point of the DAS. It
authorizes entry into the material solution analysis phase.
[17] The Conference Report for the National Defense Authorization Act
for Fiscal Year 2008 proposes seven milestones. H.R. Conf. Rep. No. 110-
477, at 951 (2007).
[18] According to the GAO Cost Assessment Guide, rebaselining should
occur rarely, as infrequently as once in the life of a program or
project. Schedule rebaselining should occur only when a schedule
variance is significant enough to limit its utility as a predictor of
future schedule performance. GAO, Cost Assessment Guide: Best Practices
for Estimating and Managing Program Costs, Exposure Draft, [hyperlink,
http://www.gao.gov/products/GAO-07-1134SP] (Washington, D.C.: July
2007), p. 251.
[19] IDA, Acquisition Initiatives Review, Phase II, (Alexandria, VA.:
February 2006).
[20] According to DOD, a time-certain development period should be
tailored to the characteristics of the weapon system under development.
For example, 6 years to develop and produce major acquisitions, such as
a nuclear submarine or an aircraft carrier, may not be realistic.
[21] CJCSI 3170.01F, May 1, 2007, requires the Capability Development
Document (CDD) and Capability Production Document (CPD) to include the
definition of IOC and the required IOC date. The CDD and CPD are
approved and validated by the JROC, and the Acquisition Program
Baseline (APB) is approved by the MDA. DODI 5000.02, December 8, 2008,
requires the APB to include the definition of system IOC and the IOC
date. However, no time limitations are specified in either instruction.
[22] GAO, Tactical Aircraft: DOD Needs a Joint and Integrated
Investment Strategy, [hyperlink,
http://www.gao.gov/products/GAO-07-415] (Washington, D.C.: Apr. 2,
2007).
[23] DOD Directive 5000.04-Cost Analysis Improvement Group, dated
August 16, 2006, describes the responsibilities of the CAIG.
[24] According to DOD, the CAIG prepares a signature report on each of
its estimates, and provides this report directly to the USD (AT&L). The
CAIG independent cost estimates are prepared in accordance with the
existing statutory requirements 10 USC § 2434, and it is presented for
consideration at the Defense Acquisition Board milestone review
meetings.
[25] DOT&E is the principal staff assistant and senior advisor to the
Secretary of Defense on operational test and evaluation (OT&E) in DOD.
DOT&E's responsibilities include issuing DOD OT&E policy and
procedures; reviewing and analyzing the results of OT&E conducted for
each MDAP; providing independent assessments to Secretary of Defense,
the USD (AT&L), and Congress; making budgetary and financial
recommendations to the Secretary of Defense regarding OT&E; and
providing oversight to ensure OT&E for MDAPs is adequate to confirm
operational effectiveness and suitability of the defense system in
combat use.
[26] GAO, Defense Acquisitions: A Knowledge-Based Funding Approach
Could Improve Major Weapon System Program Outcomes, [hyperlink,
http://www.gao.gov/products/GAO-08-619] (Washington, D.C.: July 2,
2008).
[27] GAO, Defense Acquisitions: Better Weapon Program Outcomes Require
Discipline, Accountability, and Fundamental Changes in the Acquisition
Environment, [hyperlink, http://www.gao.gov/products/GAO-08-782T]
(Washington, D.C.: June 3, 2008).
[28] GAO, Space Acquisitions: DOD Needs to Take More Action to Address
Unrealistic Initial Cost Estimates of Space Systems, [hyperlink,
http://www.gao.gov/products/GAO-07-96] (Washington, D.C.: November 17,
2006).
[29] [hyperlink, http://www.gao.gov/products/GAO-08-619].
[30] GAO, Test and Evaluation: Impact of DOD's Office of the Director
of Operational Test and Evaluation, NSIAD-98-22 (Washington, D.C.:
October 24, 1997).
[31] 10 USC § 2366A and § 2366B have certification criteria for both
milestone A and B reviews.
[32] USD (AT&L) Memorandum, Program Management Tenure and
Accountability, May 25, 2007.
[33] GAO, Defense Acquisitions: Department of Defense Actions on
Program Manager Empowerment and Accountability, [hyperlink,
http://www.gao.gov/products/GAO-08-62R] (Washington, D.C.: March 11,
2008).
[34] Goldwater Nichols Department of Defense Reorganization Act of
1986, Pub. L. No. 99-433.
[35] Pub. L. No. 110-181.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
