Defense Acquisitions
A Knowledge-Based Funding Approach Could Improve Major Weapon System Program Outcomes Gao ID: GAO-08-619 July 2, 2008The Department of Defense (DOD) expects the cost to develop and procure the major weapon systems in its current portfolio to total $1.6 trillion. With increased competition for funding within DOD and across the federal government, effectively managing these acquisitions is critical. Yet DOD programs too often experience poor outcomes--like increased costs and delayed fielding of needed capabilities to the warfighter. In 2006, this Committee mandated that GAO report on DOD's processes for identifying needs and allocating resources for its weapon system programs. In 2007, GAO reported that DOD consistently commits to more programs than it can support. This follow-on report assesses DOD's funding approach, identifies key factors that influence the effectiveness of this approach, and identifies practices that could help improve DOD's approach. To conduct its work, GAO assessed 20 major weapon programs in DOD's current portfolio--5 in detail--and reviewed relevant DOD policy and guidance, prior GAO work, and other relevant literature. GAO also reviewed the practices of selected successful companies.
DOD often does not commit full funding to develop its major weapon systems when they are initiated, despite the department's policy to do so. For a majority of the weapon system programs GAO reviewed, costs have exceeded the funding levels initially planned for and reflected in the Future Years Defense Program (FYDP)--DOD's investment strategy. To compensate for these shortfalls, DOD makes unplanned and inefficient funding adjustments, like moving money from one program to another, deferring costs into the future, or reducing procurement quantities. DOD's flawed funding process is largely driven by decision makers' willingness to accept unrealistic cost estimates and DOD's commitment to more programs than it can support. DOD often underestimates development costs--due in part to a lack of knowledge and optimistic assumptions about requirements and critical technologies. At the same time, DOD's continued failure to balance its needs with available resources promotes unhealthy competition among programs for funding. This creates incentives for service and program officials to establish requirements that make their particular weapon systems stand out, with less consideration of the resources needed to develop them. Ultimately, DOD tends to push the need for funding to the future rather than limit program length or adjust requirements. The successful commercial companies that GAO has previously reviewed achieve adequate and stable funding for product development programs by following a disciplined, knowledge-based approach to estimating program costs; using manageable development cycles to increase the predictability of funding needs and the likelihood of program success; and using portfolio management practices to make decisions about which programs to pursue. Once programs are approved, these companies firmly commit to fully fund them.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-08-619, Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve Major Weapon System Program Outcomes
This is the accessible text file for GAO report number GAO-08-619
entitled 'Defense Acquisitions: A Knowledge-Based Funding Approach
Could Improve Major Weapon System Program Outcomes' which was released
on July 3, 2008.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Committee on Armed Services, U.S. Senate:
United States Government Accountability Office:
GAO:
June 2008:
Defense Acquisitions:
A Knowledge-Based Funding Approach Could Improve Major Weapon System
Program Outcomes:
Defense Acquisitions:
GAO-08-619:
GAO Highlights:
Highlights of GAO-08-619, a report to the Committee on Armed Services,
U.S. Senate.
Why GAO Did This Study:
The Department of Defense (DOD) expects the cost to develop and procure
the major weapon systems in its current portfolio to total $1.6
trillion. With increased competition for funding within DOD and across
the federal government, effectively managing these acquisitions is
critical. Yet DOD programs too often experience poor outcomes”like
increased costs and delayed fielding of needed capabilities to the
warfighter.
In 2006, this Committee mandated that GAO report on DOD‘s processes for
identifying needs and allocating resources for its weapon system
programs. In 2007, GAO reported that DOD consistently commits to more
programs than it can support. This follow-on report assesses DOD‘s
funding approach, identifies key factors that influence the
effectiveness of this approach, and identifies practices that could
help improve DOD‘s approach.
To conduct its work, GAO assessed 20 major weapon programs in DOD‘s
current portfolio”5 in detail”and reviewed relevant DOD policy and
guidance, prior GAO work, and other relevant literature. GAO also
reviewed the practices of selected successful companies.
What GAO Found:
DOD often does not commit full funding to develop its major weapon
systems when they are initiated, despite the department‘s policy to do
so. For a majority of the weapon system programs GAO reviewed, costs
have exceeded the funding levels initially planned for and reflected in
the Future Years Defense Program (FYDP)”DOD‘s investment strategy. To
compensate for these shortfalls, DOD makes unplanned and inefficient
funding adjustments, like moving money from one program to another,
deferring costs into the future, or reducing procurement quantities.
Figure: Funding Shortfalls at the Start of Development for Five Major
Weapon System Programs:
Program: Multi-mission Maritime Aircraft;
Level of funding established in the FYDP in the year the program was
initiated: 32%;
Level of funding the program needed to be fully funded in the initial
FYDP: 35%;
Funding required beyond the initial FYDP to complete development: 33%;
Total: 100%.
Program: Warfighter Information Network-Tactical;
Level of funding established in the FYDP in the year the program was
initiated: 21%;
Level of funding the program needed to be fully funded in the initial
FYDP: 49%;
Funding required beyond the initial FYDP to complete development: 30%;
Total: 100%.
Program: Future Combat System;
Level of funding established in the FYDP in the year the program was
initiated: 26%;
Level of funding the program needed to be fully funded in the initial
FYDP: 41%;
Funding required beyond the initial FYDP to complete development: 33%;
Total: 100%.
Program: Joint Strike Fighter;
Level of funding established in the FYDP in the year the program was
initiated: 64%;
Level of funding the program needed to be fully funded in the initial
FYDP: 13%;
Funding required beyond the initial FYDP to complete development: 23%;
Total: 100%.
Program: Global Hawk;
Level of funding established in the FYDP in the year the program was
initiated: 31%;
Level of funding the program needed to be fully funded in the initial
FYDP: 48%;
Funding required beyond the initial FYDP to complete development:
Total: 21%.
Source: DOD (data). GAO (analysis and presentation).
[End of figure]
DOD‘s flawed funding process is largely driven by decision makers‘
willingness to accept unrealistic cost estimates and DOD‘s commitment
to more programs than it can support. DOD often underestimates
development costs”due in part to a lack of knowledge and optimistic
assumptions about requirements and critical technologies. At the same
time, DOD‘s continued failure to balance its needs with available
resources promotes unhealthy competition among programs for funding.
This creates incentives for service and program officials to establish
requirements that make their particular weapon systems stand out, with
less consideration of the resources needed to develop them. Ultimately,
DOD tends to push the need for funding to the future rather than limit
program length or adjust requirements.
The successful commercial companies that GAO has previously reviewed
achieve adequate and stable funding for product development programs by
following a disciplined, knowledge-based approach to estimating program
costs; using manageable development cycles to increase the
predictability of funding needs and the likelihood of program success;
and using portfolio management practices to make decisions about which
programs to pursue. Once programs are approved, these companies firmly
commit to fully fund them.
What GAO Recommends:
GAO is making three recommendations aimed at increasing funding
stability and improving acquisition outcomes. DOD believes that current
policies and initiatives sufficiently address the first two
recommendations, and did not concur with the third.
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-619]. For more
information, contact Michael J. Sullivan at (202) 512-4841 or
sullivanm@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
Failure to Commit Full Funding to Weapon Systems Contributes to Poor
Acquisition Outcomes:
Unrealistic Cost Estimates and a Failure to Balance Needs with
Available Resources Underlie DOD's Flawed Funding Approach:
Proven Practices Help Ensure Accurate Cost Estimates and Adequate
Program Funding:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Evaluation:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Comments from the Department of Defense:
Related GAO Products:
Table:
Table 1: Development Cost Estimates and Baselines for 20 Major Weapon
System Programs:
Figures:
Figure 1: Simplified View of PPBE Process:
Figure 2: Shortfalls at the Start of Development for Five Major Weapon
System Programs:
Figure 3: Costs Remaining Versus Annual Appropriations for Major
Defense Acquisitions:
Figure 4: Notional Comparison of Cost Estimating Uncertainty and Levels
of Knowledge at Program Start:
Figure 5: Range of Possible Costs Narrows as Knowledge Is Gained:
Abbreviations:
AEHF: Advanced Extremely High Frequency Satellite:
CAIG: Cost Analysis Improvement Group:
CARD: Cost Analysis Requirements Description:
DAS: Defense Acquisition System:
DOD: Department of Defense:
EFV: Expeditionary Fighting Vehicle:
FCS: Future Combat System:
FYDP: Future Years Defense Program:
IDA: Institute for Defense Analysis:
JCIDS: Joint Capabilities Integration and Development System:
JSF: Joint Strike Fighter:
JTRS: Joint Tactical Radio System:
MDA: Milestone Decision Authority:
MMA: Multi-mission Maritime Aircraft:
OMB: Office of Management and Budget:
OSD: Office of the Secretary of Defense:
PPBE: Planning, Programming, Budgeting, and Execution:
SAR: Selected Acquisition Reports:
SIBRS: Space-Based Infrared System-High:
WIN-T: Warfighter Information Network-Tactical:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
July 2, 2008:
The Honorable Carl Levin:
Chairman:
The Honorable John McCain:
Ranking Member:
Committee on Armed Services:
United States Senate:
The Department of Defense (DOD) expects the cost to develop and procure
the major weapon systems in its current portfolio to total $1.6
trillion, $335 billion of which is expected to be spent over the next 5
years. Effective management of the costs of these acquisitions is
critical given the increased competition for funds within the
department to support ongoing military operations in Afghanistan and
Iraq, as well as growing pressures to reduce overall DOD spending due
to the long-term fiscal imbalances facing the federal government.
However, many of DOD's major weapon system development programs have
experienced poor outcomes--cost increases that add up to hundreds of
millions of dollars, schedule delays that add up to years, and
capabilities that fall short of what was promised.
How DOD manages its weapon system investments has been a matter of
congressional concern for many years. In fiscal year 2006, the Senate
Armed Services Committee raised concerns about DOD's poor track record
with acquisition programs and directed GAO to assess how DOD's
processes and practices for identifying requirements and allocating
resources affect the department's weapon system acquisition programs.
In March 2007, we reported that DOD lacks an effective, integrated
portfolio management approach that takes into account all of the
department's major weapon system programs and that requires tough
decisions commensurate with available resources.[Footnote 1] In short,
we noted that DOD commits to more programs than it can support. This
report, also done in response to the same Senate mandate, focuses on
DOD's funding process and its impact on major acquisitions.
Specifically, the report (1) assesses how DOD budgets for and funds its
major weapons system acquisition programs, (2) identifies key factors
that influence the effectiveness of this approach, and (3) identifies
proven processes and practices that could help improve DOD's ability to
effectively allocate resources to its acquisition programs. GAO also
has ongoing related work specifically assessing DOD's process for
identifying and prioritizing warfighting capability requirements.
To assess DOD's funding process and to identify key factors that
influence the effectiveness of that approach, we reviewed relevant DOD
policy guidance, legislation, and academic literature, and assessed
cost estimates and budget data for 20 of the 95 major weapons programs
in DOD's current portfolio--which represent more than one-third of the
total expected cost of DOD's current portfolio of major weapon system
programs. To gain further insights into the impact of DOD's funding
process on individual programs, we conducted more detailed analysis for
five of these programs: Global Hawk, Joint Strike Fighter (JSF), Future
Combat System (FCS), Warfighter Information Network-Tactical (WIN-T),
and Multi-mission Maritime Aircraft (MMA).[Footnote 2] In addition we
interviewed numerous officials from the Office of the Secretary of
Defense (OSD) as well as military service cost analysis, budgeting, and
acquisition offices. To identify proven cost estimating and budgeting
processes and practices that could be used by DOD to improve its
resource allocation process, we utilized information from our March
2007 best practices report, and conducted follow-up interviews with
officials from three of the five companies that provided input to that
report--Eli Lilly, IBM, and Motorola. We also relied on our Cost
Assessment Guide, which provides a cost-estimating methodology based on
best practices.[Footnote 3] We conducted this performance audit from
June 2007 to May 2008 in accordance with generally accepted government
auditing standards. Those standards require that we plan and perform
the audit to obtain sufficient, appropriate evidence to provide a
reasonable basis for our findings and conclusions based on our audit
objectives. We believe that the evidence obtained provides a reasonable
basis for our findings and conclusions based on our audit objectives.
Results in Brief:
DOD often does not commit full funding to its major weapon system
acquisition programs, despite the department's policy to "fully fund"
programs at the start of system development.[Footnote 4] Of the
programs we reviewed, over 75 percent were not fully funded in the
Future Years Defense Program (FYDP)--DOD's investment strategy. We
found that because programs typically had development cycles that
extended beyond the FYDP time frame, the FYDP did not capture their
full funding needs. At the same time, program costs exceeded the
funding levels initially planned for and reflected in the years covered
by the FYDP. To compensate for funding shortfalls, DOD often makes
unplanned and inefficient funding adjustments, such as moving money
between programs, deferring work and associated costs into the future,
or reducing procurement quantities. Ultimately, such reactive practices
obscure true program costs and contribute to the instability of many
programs and poor acquisition outcomes.
DOD's inability to allocate funding effectively to programs is largely
driven by the acceptance of unrealistic cost estimates and a failure to
balance needs based on available resources. Development costs for major
acquisition programs are often underestimated at program initiation--30
to 40 percent in some cases--in large part because the estimates are
based on limited knowledge and optimistic assumptions about system
requirements and critical technologies. For example, initial
development cost estimates for the Army's WIN-T communications system
were understated by at least $1.3 billion, or nearly 160 percent, in
part because the estimates assumed that commercial-off-the-shelf radio
technology would be available. This assumption proved to be wrong.
Similarly, JSF's development costs were underestimated by at least $7.1
billion, or around 20 percent, because the initial estimate assumed
certain efficiencies that never materialized. These unrealistic cost
estimates are developed in an environment where DOD commits to more
programs than available resources can support, which promotes unhealthy
competition among programs for funding. This competition creates strong
incentives for program officials to establish requirements that make
their particular weapon systems stand out from others, with less
consideration given to the resources that will be needed to develop
them. Ultimately, programs tend to push the need for funding to the
future rather than limit program length or adjust requirements.
Our past work on commercial best practices found that successful
companies were able to fully fund their product development programs
because they required programs to have realistic, knowledge-based cost
estimates; they committed to manageable product increments; and they
prioritized programs within resource constraints. To develop realistic
cost estimates, these companies followed a disciplined process through
which requirements were assessed against available resources at
multiple gated reviews prior to committing to a new product. As more
knowledge was gained and risk was reduced, costs estimates more
accurately reflected true costs. Before initiating product development,
for example, Motorola and IBM expected that actual costs would not
exceed the latest estimate by more than 5 or 10 percent. To increase
the predictability of funding needs--and funding stability--these
companies expected development cycles to be manageable (2 to 5 years).
Likewise, a study commissioned by DOD in 2006 recommended that programs
should be time-constrained with development cycles no longer than 6
years, and DOD is taking steps to pilot this concept. Finally,
companies used a portfolio management approach to prioritize
investments and allocate resources. Such an approach requires tough
decisions about approving or terminating programs. Once programs were
approved, these companies made a firm commitment to fully fund them.
Ultimately, these practices helped companies avoid committing to more
programs than they could afford and allowed them to optimize the return
on their investments.
To more effectively fund major weapon system acquisition programs and
achieve successful outcomes, we are recommending that the Secretary of
Defense develop and implement a strategy to bring DOD's current weapon
systems portfolio into balance by aligning the number of systems with
available resources in the FYDP. In addition, the Secretary should
require new programs to have manageable development cycles, realistic
cost estimates, and to have planned and programmed full funding for the
entire development cycle. Finally, the Secretary should require all
cost estimates submitted for funding a program at milestone decisions
to be reported as a range of likely costs and reflect the associated
levels of risk and uncertainty. DOD partially concurred with our first
two recommendations, but believes that they are being sufficiently
addressed through current acquisition policies and ongoing initiatives.
While we agree that if implemented appropriately, DOD's current
policies and initiatives have the potential to contribute to better
program outcomes, we have found no evidence of widespread adoption of a
knowledge-based process to better ensure adequate funding for programs.
DOD agreed with the spirit of our third recommendation--to consider
risks at key milestones--however, it did not agree that all cost
estimates submitted for funding a program at milestone decisions should
be reported as a range of likely costs. In its written comments, DOD
stated that a certain method used to calculate cost ranges can produce
misleading results. While we do not advocate a specific method for
calculating a range of potential costs, we maintain that presenting
such a range would provide decision makers with additional knowledge
about the level of risk in a proposed program.
Background:
To plan, execute, and fund its weapon system acquisition programs, DOD
relies on three principal decision-making systems: the Joint
Capabilities Integration and Development System (JCIDS), which is used
to assess gaps in warfighting capabilities and recommend solutions to
resolve those gaps; the Defense Acquisition System (DAS), which is used
to manage the development and procurement of weapon systems and other
equipment; and the Planning, Programming, Budgeting, and Execution
(PPBE) process, which is used to allocate resources. While the JCIDS
and DAS processes are driven by specific events--such as validating
requirements or receiving approval to start development--the PPBE
process is calendar driven, taking nearly 2 years to go from planning
to the beginning of budget execution.
The PPBE process is intended to provide a framework within which DOD
can articulate its strategy; identify force size, structure, and needed
equipment; set program priorities; allocate resources to individual
programs; and assess program performance. Although the different phases
of PPBE are considered sequential, because of the amount of time
required to develop and review resource requirements, the process is
continuous and concurrent with at least two phases ongoing at any given
time (see fig. 1).
Figure 1: Simplified View of PPBE Process:
[See PDF for image]
This figure is an illustration of a simplified view of PPBE Process, as
follows:
2007 PPBE:
Planning: Late 2004 to mid-2005;
Programming and budgeting: Mid-2005 to late 2006;
Execution (1 to 3 years depending on funding type): Late 2006 to 2008
and beyond.
2008 PPBE:
Planning: Late 2005 to mid-2006;
Programming and budgeting: Mid-2006 to late 2007;
Execution (1 to 3 years depending on funding type): Late 2007 to 2009
and beyond.
2009 PPBE:
Planning: Late 2006 to mid-2007;
Programming and budgeting: Mid-2007 to late 2008;
Execution (1 to 3 years depending on funding type): Late 2008 to 2010
and beyond.
2010 PPBE:
Planning: Late 2007 to mid-2008;
Programming and budgeting: Mid-2008 to late 2009;
Execution (1 to 3 years depending on funding type): Late 2009 to 2011
and beyond.
2011 PPBE:
Planning: Late 2008 to mid-2009;
Programming and budgeting: Mid-2009 to late 2010;
Execution (1 to 3 years depending on funding type): Late 2010 to 2012
and beyond.
Source: DOD (data). GAO (analysis and presentation).
[End of figure]
At the front end of the PPBE process, the Secretary of Defense provides
planning guidance to the military services and defense agencies about
the capabilities and resources required to deter and defeat threats.
The services and agencies, in turn, develop individual "programs"
(budgets) based on the planning guidance as well as fiscal guidance
provided by the Office of Management and Budget (OMB). The service and
agency program budgets are then subjected to a series of leadership
reviews in the department--OSD, Director of Program Analysis and
Evaluation, Joint Chiefs of Staff, and the Comptroller--and adjustments
are made if necessary. Responsibility for managing the PPBE process and
ensuring the budget is prepared and submitted to Congress resides with
the OSD Comptroller.
The PPBE process produces the defense portion of the President's annual
budget request to Congress as well as the FYDP--DOD's longer-term
investment strategy--which includes the department's resource
allocations for the 4 to 5 fiscal years beyond the budget.[Footnote 5]
The ultimate objective of the FYDP is to manage funds in a way that
provides combatant commanders with the best mix of forces, equipment,
and support attainable within fiscal constraints established by OMB.
Once complete, the FYDP is expected to provide an aggregate picture of
the anticipated force levels and funding needs of individual programs.
Data in the FYDP tell the Secretary of Defense, the President,
Congress, and the American people what the department expects to invest
in and how much it will spend over time.
DOD policy requires that the dollars and manpower needed to carry out
the first 5-to 6-years of a weapon system acquisition program must be
included in the FYDP. The value of the FYDP greatly depends on the
accuracy of the cost estimates supporting each individual program.
Having a realistic cost estimate provides a basis for accurate
budgeting and effective resource allocation, increasing the probability
of a program's success in meeting its targets. Two cost estimates are
required before a program is approved to start system development
(Milestone B) and at the beginning of production (Milestone C)--the
service or program office estimate and an independent cost estimate.
The independent cost estimate for most major acquisition programs is
developed by the Cost Analysis Improvement Group (CAIG). At Milestone
B, the Milestone Decision Authority (MDA)[Footnote 6] uses the two
estimates to determine the program's official cost baseline and whether
the funding reflected in the FYDP is adequate to cover the portion of
the estimated costs of the program that fall within the FYDP time
period.[Footnote 7] If not, the services are expected to adjust the
FYDP to ensure that their programs are fully funded to the approved
baseline. To receive Milestone B approval, DOD is required to certify
to Congress[Footnote 8] that funding is available to support the
portion of the program that falls within the period covered by the FYDP
that is submitted during the fiscal year in which the certification is
made.[Footnote 9]
Our past work has consistently shown that DOD's major weapon system
programs do not meet their cost and schedule targets. Since the mid-
1990s, we have studied leading commercial companies in order to
identify best practices for developing and producing new products.
Taking into account the differences between commercial product
development and weapons acquisitions, we articulated a best practices
model that relies on increasing knowledge when developing new products,
separating technology development from product development, and
following an evolutionary or incremental approach to product
development.[Footnote 10] This knowledge-based approach requires
developers to make investment decisions on the basis of specific,
measurable levels of knowledge at critical junctures before investing
more money and before advancing to the next phase of acquisition. An
evolutionary product development process defines the individual
increments on the basis of mature technologies and a feasible design
that are matched with firm requirements. The knowledge-based,
evolutionary approach in our model is intended to help reduce
development risks and to achieve better program outcomes on a more
consistent basis.
In October 2000, DOD began to significantly revise the department's
acquisition policy, adopting a knowledge-based, evolutionary product
development approach. DOD's revised policy emphasizes the importance of
and provides a good framework for capturing knowledge about critical
technologies, product design, and manufacturing processes. In recent
years, we have reported that if properly implemented and enforced,
DOD's acquisition policy could reduce technical risk at the start of a
program and make cost and delivery estimates much more predictable.
Failure to Commit Full Funding to Weapon Systems Contributes to Poor
Acquisition Outcomes:
DOD often does not commit full funding to its major weapon system
acquisitions when they are initiated, despite the department's policy
to do so. For a majority of the programs we reviewed, costs exceeded
the funding levels initially planned for and reflected in the FYDP. To
make up for these funding shortfalls, DOD often shifts funds from one
program to pay for another, reduces system capabilities, cuts
procurement quantities, extends development and procurement schedules,
or in rare cases terminates programs. Such actions not only create
instability in DOD's weapon system portfolio, they also obscure the
true future costs of current commitments, making it difficult to make
informed investment decisions.
At the beginning of system development, the approved cost estimates for
most of the major acquisition programs we reviewed have exceeded the
funding initially established in the FYDP. First, our analysis
indicates that 14 of the 20 programs had development cycles that
extended beyond the FYDP. As a result, the FYDP could not capture all
of the estimated funding needed to complete development. Second, the
funding established in the FYDP at the beginning of system development
for 15 of the 20 programs was less than what they required for that
time period.[Footnote 11] For example, at the start of system
development in 2003, the initial approved cost estimate for the
development portion of the FCS program was about $20 billion, of which
$13 billion was needed within the 2003 FYDP time frame--that is for
fiscal years 2003 through 2007. However, at that time the 2003 FYDP
only contained $5 billion for FCS development. This means that the
department had only committed to fund 25 percent of FCS's estimated
development cost when the program started, and only 39 percent of the
funding needed in its first 5 years. Other programs we reviewed
experienced similar shortfalls (see fig. 2).
Figure 2: Shortfalls at the Start of Development for Five Major Weapon
System Programs:
[See PDF for image]
Program: Multi-mission Maritime Aircraft;
Level of funding established in the FYDP in the year the program was
initiated: 32%;
Level of funding the program needed to be fully funded in the initial
FYDP: 35%;
Funding required beyond the initial FYDP to complete development: 33%;
Total: 100%.
Program: Warfighter Information Network-Tactical;
Level of funding established in the FYDP in the year the program was
initiated: 21%;
Level of funding the program needed to be fully funded in the initial
FYDP: 49%;
Funding required beyond the initial FYDP to complete development: 30%;
Total: 100%.
Program: Future Combat System;
Level of funding established in the FYDP in the year the program was
initiated: 26%;
Level of funding the program needed to be fully funded in the initial
FYDP: 41%;
Funding required beyond the initial FYDP to complete development: 33%;
Total: 100%.
Program: Joint Strike Fighter;
Level of funding established in the FYDP in the year the program was
initiated: 64%;
Level of funding the program needed to be fully funded in the initial
FYDP: 13%;
Funding required beyond the initial FYDP to complete development: 23%;
Total: 100%.
Program: Global Hawk;
Level of funding established in the FYDP in the year the program was
initiated: 31%;
Level of funding the program needed to be fully funded in the initial
FYDP: 48%;
Funding required beyond the initial FYDP to complete development:
Total: 21%.
Source: DOD (data). GAO (analysis and presentation).
[End of figure]
To bring funding commitments more in line with original program
baselines, DOD typically makes adjustments in subsequent FYDPs. By that
time, however, program cost baselines have often increased--as more
knowledge about technologies, design, and manufacturing needs is
gained--creating new funding shortfalls. For example, by the third year
of the WIN-T program, the program's cost estimate, and therefore
funding needs, had increased 72 percent because requirements were
unstable and critical technologies were not ready, creating a 26
percent shortfall in the FYDP. Similarly, a 57 percent increase in
FCS's cost estimate created an additional 12 percent--or $2 billion--
funding shortfall.
In reaction to funding shortfalls in individual programs, DOD often
moves funds from other programs,[Footnote 12] reduces capabilities by
scaling back requirements, cuts procurement quantities, or extends
development schedules. Although rare, programs have also been
terminated. Such reactive measures destabilize programs and in many
cases increase program costs. For example, as we previously reported,
funding for the Air Force's Space-Based Infrared System-High (SIBRS)
satellite system was cut in 1998 and 1999 to pay for higher budget
priorities, which contributed to a 2-year delay in the program and a
breach of the cost baseline. As a result, the Air Force changed SIBRS's
procurement strategy, which independent cost estimators calculated
would double program costs.[Footnote 13] For the F-22A Raptor program,
our past work has noted that the Air Force drastically reduced its
planned buys from 648 to 183 as program costs escalated. Similarly, the
number of requirements for the Joint Tactical Radio System (JTRS) was
reduced or deferred by about one-third as the program encountered
development problems. This change had a reverberating effect on several
JTRS-dependent efforts--such as the Army's modernization of radios for
its helicopters--as those programs had to make adjustments and go
forward with alternative, less capable solutions.[Footnote 14] Making
these types of adjustments to compensate for inadequate funding
obscures future resource requirements and limits the Congress's
visibility over DOD's true funding needs.
Unrealistic Cost Estimates and a Failure to Balance Needs with
Available Resources Underlie DOD's Flawed Funding Approach:
For most of the 20 major acquisition programs that we reviewed, initial
funding was based on cost estimates that proved to be too low. DOD's
funding approach is further compromised by the department's failure to
balance requirements with available resources. Because DOD commits to
more programs than resources can support, programs often have to
compete for funding by overpromising capabilities and by providing low
cost estimates as inputs to the funding process. We have previously
reported that when such trends go unchecked, Congress is consistently
faced with a difficult choice: pull funds from other federal programs
to support DOD's acquisitions or accept less warfighting capability
than promised.[Footnote 15]
Unrealistic Cost Estimates Drive Inaccurate Funding Commitments:
The foundation of an accurate funding commitment for a weapon system
program should be a realistic cost estimate that is based on a high
degree of knowledge about requirements, technology, design, and
manufacturing. Realistic estimates also provide a sound basis for
setting priorities by allowing decision makers to compare the relative
value of one program to another and to make adjustments accordingly.
These adjustments could include a decision about whether to proceed
with a program, reduce requirements, or defer requirements to a future
increment. Most of the 20 programs we reviewed underestimated costs.
Unrealistic cost estimates are largely the result of a lack of
knowledge, failure to adequately account for risk and uncertainty, and
overly optimistic assumptions about the time and resources needed to
develop weapon systems. By repeatedly relying on unrealistically low
cost estimates, DOD has initiated more programs than its budget can
support.
Our assessment of cost data for 20 major acquisition programs in DOD's
current portfolio found that the majority of these programs were
initiated with cost estimates for system development that were too low-
-a finding consistent with our prior work and with cost growth patterns
reported by RAND, the Institute for Defense Analysis (IDA), and other
organizations that conduct defense analyses.[Footnote 16] For 19 of the
20 programs, the independent CAIG estimates were higher than the
service estimates--by as much as 139 percent in one case--yet the CAIG
estimates for 5 of those programs were still understated by billions of
dollars (see table 1). In addition, while 5 of the 20 programs had not
reported cost growth as of December 2007, the remaining 15 programs
had. For example, the initial CAIG estimate for the Expeditionary
Fighting Vehicle (EFV) program was about $1.4 billion compared to a
service estimate of about $1.1 billion, but development costs for the
EFV system is now expected to be close to $3.6 billion. Similarly, the
Army initially estimated that WIN-T development would cost $338
million, but the development program is now expected to cost over $2.0
billion, or $1.7 billion more than initially estimated. Eight of the 20
programs have reported development cost growth of more than 35 percent,
resulting in the need for nearly $19 billion in additional funding.
Estimates that are this far off the mark do not provide the necessary
foundation for sufficient funding commitments.
Table 1: Development Cost Estimates and Baselines for 20 Major Weapon
System Programs:
Program: Global Hawk:
Development cost estimate: Service: $905;
Development cost estimate: CAIG: $992;
Percent difference: 10%;
Development cost baselines: Initial: $967;
Development cost baselines: Current: $3,515;
Percent change: 264%.
Program: UH-60M helicopter upgrade;
Development cost estimate: Service: $311;
Development cost estimate: CAIG: $379;
Percent difference: 22%;
Development cost baselines: Initial: $311;
Development cost baselines: Current: $838;
Percent change: 169%.
Program: WIN-T;
Development cost estimate: Service: $338[A];
Development cost estimate: CAIG: $807;
Percent difference: 139%;
Development cost baselines: Initial: $796;
Development cost baselines: Current: $2,088;
Percent change: 162%.
Program: C-130 Avionics Modernization;
Development cost estimate: Service: $1,020;
Development cost estimate: CAIG: $1,175;
Percent difference: 15%;
Development cost baselines: Initial: $720;
Development cost baselines: Current: $1,844;
Percent change: 156%.
Program: EFV;
Development cost estimate: Service: $1,056;
Development cost estimate: CAIG: $1,438;
Percent difference: 36%;
Development cost baselines: Initial: $1,472;
Development cost baselines: Current: $3,556;
Percent change: 142%.
Program: Advanced Extremely High Frequency Satellites;
Development cost estimate: Service: $3,031;
Development cost estimate: CAIG: $3,175;
Percent difference: 5%;
Development cost baselines: Initial: $2,923;
Development cost baselines: Current: $6,008;
Percent change: 105%.
Program: Wideband Global SATCOM;
Development cost estimate: Service: $296;
Development cost estimate: CAIG: $414;
Percent difference: 40%;
Development cost baselines: Initial: $199;
Development cost baselines: Current: $323;
Percent change: 62%.
Program: Future Combat Systems;
Development cost estimate: Service: $20,248;
Development cost estimate: CAIG: $27,184;
Percent difference: 34%;
Development cost baselines: Initial: $20,248;
Development cost baselines: Current: $27,955;
Percent change: 38%.
Program: Joint Strike Fighter[B];
Development cost estimate: Service: $30,500;
Development cost estimate: CAIG: $31,476;
Percent difference: 3%;
Development cost baselines: Initial: $33,939;
Development cost baselines: Current: $40,210;
Percent change: 18%.
Program: COBRA JUDY replacement;
Development cost estimate: Service: $1,398;
Development cost estimate: CAIG: $1,521;
Percent difference: 9%;
Development cost baselines: Initial: $1,527;
Development cost baselines: Current: $1,626;
Percent change: 6%.
Program: E-2 Advanced Hawkeye;
Development cost estimate: Service: $3,495;
Development cost estimate: CAIG: $3,720;
Percent difference: 6%;
Development cost baselines: Initial: $3,589;
Development cost baselines: Current: $3,796;
Percent change: 6%.
Program: EA-18G;
Development cost estimate: Service: $1,707;
Development cost estimate: CAIG: $1,795;
Percent difference: 5%;
Development cost baselines: Initial: 1,797;
Development cost baselines: Current: $1,865;
Percent change: 4%.
Program: VH-71 Presidential Helicopter Replacement Program;
Development cost estimate: Service: $3,378;
Development cost estimate: CAIG: $3,569;
Percent difference: 6%;
Development cost baselines: Initial: $3,771;
Development cost baselines: Current: $3,859;
Percent change: 2%.
Program: Joint Land Attack Cruise Missile Defense Elevated Netted
Sensor;
Development cost estimate: Service: $1,781;
Development cost estimate: CAIG: $1,926;
Percent difference: 8%;
Development cost baselines: Initial: $1,894;
Development cost baselines: Current: $1,922;
Percent change: 1%.
Program: C-5 RERP;
Development cost estimate: Service: $1,454;
Development cost estimate: CAIG: $1,583;
Percent difference: 9%;
Development cost baselines: Initial: $1,627;
Development cost baselines: Current: $1,630;
Percent change: 0[C].
Program: CH-53K Heavy Lift Replacement;
Development cost estimate: Service: $3,970;
Development cost estimate: CAIG: $4,293;
Percent difference: 8%;
Development cost baselines: Initial: $4,149;
Development cost baselines: Current: $4,095;
Percent change: -1%.
Program: Longbow Apache III;
Development cost estimate: Service: $1,155;
Development cost estimate: CAIG: $1,382;
Percent difference: 20%;
Development cost baselines: Initial: $1,095;
Development cost baselines: Current: $1,087;
Percent change: -1%.
Program: MMA;
Development cost estimate: Service: $6,100;
Development cost estimate: CAIG: $6,970;
Percent difference: 14%;
Development cost baselines: Initial: $7,080;
Development cost baselines: Current: $6,804;
Percent change: -4%.
Program: Small Diameter Bomb;
Development cost estimate: Service: $416;
Development cost estimate: CAIG:$427;
Percent difference: 3%;
Development cost baselines: Initial: $415;
Development cost baselines: Current: $395;
Percent change: -5%.
Program: Standard Missile 6;
Development cost estimate: Service: $1,000;
Development cost estimate: CAIG: $992;
Percent difference: -1%;
Development cost baselines: Initial: $1,009;
Development cost baselines: Current: $932;
Percent change: -8%.
Source: GAO analysis of DOD data.
[A] The service's original estimate for the WIN-T program of $338
million was revised prior to Milestone B to align with the CAIG's
estimate.
[B] JSF data include Air Force and Navy portions of the program only.
[C] C-5 RERP costs have increased by almost $4 million. However, due to
rounding, the table indicates a 0 percent increase.
[End of table]
DOD has found similar problems in other acquisition programs. For
example, in 2003, a DOD study found that space programs are strongly
biased to produce unrealistically low cost estimates throughout the
acquisition process.[Footnote 17] The study found that most programs at
the time of contract initiation had a predictable cost growth of 50 to
100 percent. The study also found that the unrealistically low
projections of program cost and lack of provisions for management
reserve seriously distorted management decisions and program content,
increased risks to mission success, and virtually guaranteed program
delays.
Inaccurate cost estimates are often the result of limited knowledge
about requirements and technologies. Our best practices work has shown
that conducting early disciplined analysis, such as systems
engineering,[Footnote 18] builds knowledge that enables a developer to
identify and resolve gaps between requirements and available resources
before beginning product development. DOD's acquisition policy and
guidance emphasize the importance of obtaining knowledge prior to
Milestone B--the start of system development--through key analyses and
activities--such as conducting an analysis of alternatives (AOA),
[Footnote 19] refining requirements, and reducing technology risks.
Knowledge gained from these analyses and activities should inform a
Cost Analysis Requirements Description (CARD)--a key document that
quantifies the program's technical, physical, programmatic, and
performance characteristics for developing cost estimates. However, we
have frequently reported that DOD programs do not adequately define
requirements, mature technologies, or develop an effective acquisition
strategy before Milestone B--the point at which cost estimates are
approved.[Footnote 20] According to several CAIG analysts, CARDs often
lack sufficient detail about planned program content to develop sound
cost estimates. One analyst noted that the CARD for the FCS program had
to be sent back to the program office because it was too vague.
At the same time, programs are expected to deliver the CARD to the CAIG
at least 180 days prior to Milestone B, while other supporting
documents that contain critical program information--such as the
program requirements document--are not required until about 90 days in
advance of a milestone review. Many programs conduct analyses of
alternatives and other analyses at the same time program cost estimates
are being developed. According to a senior CAIG official, such
concurrency limits their ability to develop a realistic independent
estimate. One senior DOD official also noted that the department's
acquisition policy may have encouraged programs to rush to Milestone B,
limiting the quality of the data available to develop a program cost
estimate. Although DOD's acquisition policy has included an early
formal review at Milestone A, this review was not mandatory, and as we
have reported in the past, most major acquisition programs have not
gone through this early review.[Footnote 21]
Cost analysts have also indicated that the lack of comparable products
and good historical program data limits knowledge, making it difficult
to develop realistic cost estimates. This is true for the majority of
DOD's larger and more complex transformational programs that are
seeking to provide capabilities that require advanced technologies and
equipment that have no historical precedent, like JSF and FCS.
In the absence of knowledge, cost estimators must rely heavily on
assumptions about system requirements, technology, and design maturity
as well as the time and funding needed. However, the level of resources
needed for product development is often understated, as we found in
several of the programs we reviewed. For example:
* FCS: The estimated lines of code needed to support FCS's software
development are almost three times original assumptions--from 32
million to 95 million lines of code--leading to an increase in software
development costs that now approaches $8 billion. A 2007 IDA report
also identified significant additional unquantifiable cost risk due to
immature technologies, dependencies on complementary programs,
concurrent experimentation and development, and the overall complexity
and synchronization of FCS development activities.[Footnote 22]
* JSF: JSF assumed that the commonality between the three variants of
aircraft and the use of a joint development program, instead of three
separate programs, could cut development costs by about 40 percent.
However, after development started, significant design issues forced
the program to delay development approximately 18 months to conduct
unexpected design work. In addition, the assumed commonality between
the variants decreased. As we reported in 2005, these two factors
contributed to cost increases that nearly eroded all of the assumed
cost savings.[Footnote 23]
* WIN-T: The Army assumed that the radios and software needed to
support the WIN-T system would be commercially available. However, once
system development started, the Army learned that the radios and
software would require significantly more development and integration
than initially anticipated. Further, the Army assumed that WIN-T would
be able to meet its portion of the FCS program requirements. However,
subsequent changes in the FCS requirements contributed to the need to
restructure the WIN-T program.
* Global Hawk: The Air Force assumed that validated warfighter
requirements could be met with minor additional development to a
smaller version, subsequently designated the RQ-4A. However, 1 year
after initiating both system development and low-rate initial
production, it was determined that a larger airframe and additional,
unproven technologies would be needed. As a result, DOD restructured
the acquisition strategy to include a second model--designated RQ-4B--
tripling development costs and extending the development cycle from 7
years to 12 years.
These examples are consistent with our work over the past several
decades, which has raised concerns about the cost and schedule
implications of DOD's cost estimating assumptions in its major
acquisitions. Our work on DOD's space acquisition programs has found
that these programs regularly made overly optimistic cost-estimating
assumptions, including assumptions about their ability to define
requirements, mature technologies, and secure the funding and other
resources needed to develop the system within a specified time frame.
[Footnote 24] Similarly, we recently testified that the Navy tends to
underestimate the costs needed to construct ships, resulting in
unrealistic budgets and large cost increases after ship construction
has begun. [Footnote 25] We noted that for two major ship programs, the
Navy assumed significant savings based on efficiencies that did not
materialize. We linked these optimistic assumptions to cost growth and
schedule delays in a number of these programs.
Cost estimates that lack knowledge and rely heavily on assumptions have
inherently high levels of risk and uncertainty. Conducting quantitative
risk and uncertainty analysis provides a way to assess the variability
in an estimate. Using this type of analysis, cost estimators can model
such effects as a schedule slipping or a key technology failing to
materialize, thereby identifying a range of likely costs around an
estimate. Presenting decision makers with the range of likely costs
around an estimate provides insight into the amount of cost, schedule,
and technical risks they are being asked to accept and conveys a level
of confidence associated with achieving the proposed estimate. A range
of costs also provides a basis for deciding how much funding a program
needs to be successful.
In developing cost estimates, the services and the CAIG often do not
present decision makers with the range of costs around an estimate.
[Footnote 26] Instead, they present a single, or point, estimate as the
most probable cost, which OSD expects to be a 50 percent chance that
actual program costs will be at or below the estimated value. Several
recent studies have questioned DOD's approach and recommended
establishing estimates with higher levels of confidence under the
assumption that the result will be more realistic estimates.[Footnote
27] For example, the Defense Acquisition Performance Assessment Panel
recommended adjusting program cost estimates to reflect "high
confidence"--defined as an 80 percent chance of completing development
at or below the estimated amount. Requiring a higher confidence level
could provide a better basis for determining program funding, but only
if the quality of the cost estimate is sound and the underlying risk
and uncertainty associated with the estimate are accurately captured.
If the quality of the estimate is poor to begin with, simply applying a
higher confidence level to the estimate will not make it any more
realistic.
DOD's Failure to Balance Needs with Resources Promotes Unhealthy
Competition for Funding:
In prior years, we have reported that DOD commits to more programs than
its resources can support.[Footnote 28] DOD's failure to balance
requirements with available resources promotes unhealthy competition
among programs for funding. Ultimately, programs tend to push the need
for funding to the future rather than limit program length or adjust
requirements.
DOD's portfolio of weapon system programs has grown over the past
several years at a pace that far exceeds available resources. From 1992
to 2007, the estimated acquisition costs remaining for major weapons
programs increased almost 120 percent, while the annual funding
provided for these programs only increased 57 percent, creating a
fiscal bow wave that may be unsustainable (see fig. 3). If this trend
goes unchecked and fiscal pressures to reduce spending continue to grow
as expected, Congress will be faced with a difficult choice to either
pull funds from other federal programs to support DOD's acquisitions or
accept less warfighting capability than promised.
Figure 3: Costs Remaining versus Annual Appropriations for Major
Defense Acquisitions:
[See PDF for image]
This figure is a multiple line graph depicting the following
information:
Fiscal year: 1992;
Annual RDTE and Procurement Appropriations: $100;
Costs Remaining for Major Defense Acquisitions: $324.
Fiscal year: 1993;
Annual RDTE and Procurement Appropriations: $91;
Costs Remaining for Major Defense Acquisitions: $271.
Fiscal year: 1994;
Annual RDTE and Procurement Appropriations: $79;
Costs Remaining for Major Defense Acquisitions: $253.
Fiscal year: 1995;
Annual RDTE and Procurement Appropriations: $78;
Costs Remaining for Major Defense Acquisitions: $341.
Fiscal year: 1996;
Annual RDTE and Procurement Appropriations: $78;
Costs Remaining for Major Defense Acquisitions: $345.
Fiscal year: 1997;
Annual RDTE and Procurement Appropriations: $79;
Costs Remaining for Major Defense Acquisitions: $331.
Fiscal year: 1998;
Annual RDTE and Procurement Appropriations: $82;
Costs Remaining for Major Defense Acquisitions: $280.
Fiscal year: 1999;
Annual RDTE and Procurement Appropriations: $89;
Costs Remaining for Major Defense Acquisitions: $233.
Fiscal year: 2000;
Annual RDTE and Procurement Appropriations: $94;
Costs Remaining for Major Defense Acquisitions: $309.
Fiscal year: 2001;
Annual RDTE and Procurement Appropriations: $104;
Costs Remaining for Major Defense Acquisitions: $234.
Fiscal year: 2002;
Annual RDTE and Procurement Appropriations: $111;
Costs Remaining for Major Defense Acquisitions: $586.
Fiscal year: 2003;
Annual RDTE and Procurement Appropriations: $137;
Costs Remaining for Major Defense Acquisitions: $579.
Fiscal year: 2004;
Annual RDTE and Procurement Appropriations: $148;
Costs Remaining for Major Defense Acquisitions: $722.
Fiscal year: 2005;
Annual RDTE and Procurement Appropriations: $165;
Costs Remaining for Major Defense Acquisitions: $745.
Fiscal year: 2006;
Annual RDTE and Procurement Appropriations: %147;
Costs Remaining for Major Defense Acquisitions: $841.
Fiscal year: 2007;
Annual RDTE and Procurement Appropriations: $160.
Costs Remaining for Major Defense Acquisitions: $767.
Source: DOD (data). GAO (analysis and presentation).
[End of figure]
DOD's ability to prioritize needs within available resources is
hampered because its processes and management structures for
determining capability needs and allocating resources are not
effectively integrated. Capability needs are formally identified and
validated through JCIDS, which does not account for the resources that
will be needed to meet those needs. Instead, resources are assessed and
allocated through the PPBE process, which as we have reported in the
past, is service-centric and does not effectively link resources to
capabilities. In addition, the PPBE and JCIDS processes are led by
different organizations within DOD, as is the process for executing
acquisitions, making it difficult to hold any one person or
organization accountable for saying no to a proposed program or for
program outcomes.
DOD's failure to balance its needs with available resources promotes
unhealthy competition among and within the services to get funding for
new programs and to sustain funding for existing ones. This competition
for funding creates strong incentives for service and program officials
to establish requirements that make their particular weapon systems
stand out from others, with less consideration given to the resources
that will be needed to develop them. Because DOD's overall funding must
fit within fiscal constraints of the FYDP that are established by OMB,
services place a high priority on the appearance of affordability. To
maintain a competitive edge, services develop cost estimates that will
fit within established funding levels. Also, because of the length of
the PPBE process, initial funding for a program is often established in
the FYDP before the program's initial cost estimated is developed and
approved. In addition, program officials are motivated to initiate a
weapon system development program because once initiated a program is
in a more competitive position to attract high levels of funding and
management support. Many of the weapon system program managers who
responded to a GAO survey in 2005 cited this competitive funding
environment as a key obstacle to their ability to effectively manage
their programs.[Footnote 29] For example, program managers provided
comments such as the following on competition for funding:
* "OSD has reduced funding without any understanding or appreciation
for program impacts. Funding cuts appear arbitrary."
* "OSD's near-term execution year focus results in great instability.
In reality, it should provide more strategic vectors for the department
instead of short-term adjustments to fix more tactical-level funding
needs."
* "The service and OSD typically cut programs to pay top down bills."
* "There is no such thing as funding stability in DOD. Funding
reductions and program stretch-outs are the norm due to top down fiscal
bills that occur during the execution year."
* "Unstable funding results in pressure to do aggressive things in
order to minimize the impact of budget cuts on schedule and
performance. I believe this has been a major factor in recent — program
execution problems."
* "When funding gets tight, we have been considered a bill payer for
others, even if it has "broken" our program."
OSD reviews program budgets and makes adjustments toward the end of the
PPBE cycle--often only 2 or 3 months before the budget is submitted to
Congress. For example, in December 2004, OSD cut $30 billion from the
2006 FYDP for many of its major acquisition programs, including
prominent aircraft programs like the F-22A Raptor, the V-22 Osprey, and
the C-130J Hercules. Attempting to balance investments this late in the
process often leads to additional churn in programs and encumbers
efforts to meet strategic objectives and joint needs.
Instead of rethinking performance requirements and making needed trade-
offs within and among programs, DOD and the services often pursue
ambitious solutions and technologies that result in cost growth and the
need for unplanned funding. Ultimately, DOD often defers costs into the
future--beyond the FDYP--expecting that additional funding will become
available when it is needed. According to DOD and service officials,
what often ends up happening is that programs have to extend their
schedules to spread costs out over time or reduce requirements to hold
costs down. These strategies ultimately result in delaying the delivery
of programs or providing the warfighter with less capability than
promised.
Proven Practices Help Ensure Accurate Cost Estimates and Adequate
Program Funding:
Successful commercial companies that we have previously reviewed
achieved adequate and stable funding for product development programs
by following practices that we have recognized as best practices for
estimating and managing program costs.[Footnote 30] In contrast to DOD,
these companies followed a disciplined, knowledge-based approach to
estimating program costs. Prior to initiating product development,
these companies expected cost estimates to be refined based on
increasing product knowledge and assessed the estimates at multiple
gated reviews. At the same time, the companies expected proposed
programs to have a manageable development cycle, which increases the
predictability of funding needs and the likelihood of program success.
Understanding that resources are limited, the companies used portfolio
management practices to make tough decisions about which programs to
pursue. Once programs were approved, these companies firmly committed
to fully fund them--which they often achieved by committing resources
incrementally to successive phases of a program. By making these
incremental commitments, these companies were able to limit the
disruptions that can be caused by poorly performing programs, and thus
maintained a high degree of stability within and among other programs.
A Disciplined, Knowledge-Based Approach and Manageable Development
Cycles Are Key to Realistic Cost Estimates:
As part of our best practices work on successful product development,
we have found that following a disciplined, knowledge-based approach
that reduces risk and uncertainty over time is integral to developing
realistic cost estimates. The successful companies we have reviewed
conducted multiple management reviews prior to initiating product
development to assess the business case of each proposed product,
including its cost estimate. They typically expected cost estimates to
be developed early on and refined over time as knowledge is gained. In
addition, they expected estimates to be transparent because it allowed
them to make informed decisions about the risks and uncertainty
associated with proposed programs and whether they should pursue or
cancel them. While these companies acknowledged that early in a
program's life cycle less is known about the product's design and
technologies, they still expected to receive rough cost estimates at
that stage. They expected the estimates to be revised and more precise
as knowledge was gained. Before committing to product development, they
expected the level of uncertainty in the estimates to be low.
Our Cost Assessment Guide similarly emphasizes the need to refine cost
estimates based on knowledge gained over time and account for risk and
uncertainty in the estimates. Early cost estimates are more uncertain
because less is known about requirements and the opportunity for change
is greater. As more knowledge is gained, programs can retire some risk
and reduce the potential for unexpected cost and schedule growth. The
best practice is not to commit to product development until the level
of knowledge is high and the level of uncertainty is low. However, our
past work has found that DOD typically commits to starting programs
with low levels of knowledge and high levels of uncertainty (see fig.
4).
Figure 4: Notional Comparison of Cost Estimating Uncertainty and Levels
of Knowledge at Program Start:
[See PDF for image]
This figure is a matrix depicting a notional comparison of cost
estimating uncertainty and levels of knowledge at program start.
Knowledge moves from low to high on the horizontal axis, and cost
estimating uncertainty moves from low to high on the vertical axis.
Noted on the matrix:
Typical DOD program start: Occurs with high cost estimating uncertainty
and low knowledge.
Best practice program start: Occurs with high knowledge and low cost
estimating uncertainty.
Source: GAO.
[End of figure]
Despite having high levels of uncertainty, DOD commits to development
programs based on point estimates that are expected to represent most
likely costs. The Air Force's Cost Risk and Uncertainty Analysis
Handbook notes that decision makers need point estimates when preparing
and managing a budget because programs are funded and executed using
discrete dollars, not ranges of dollars. However, DOD's estimates often
prove to be understated by as much as 30 to 40 percent. To make more
informed investment decisions, cost estimating best practices call for
estimating a range of possible costs around a point estimate to provide
information about the levels of uncertainty and confidence. As proposed
programs gain more knowledge and progress through the phases leading up
to the start of product development, these ranges should narrow (see
fig. 5). According to cost-estimating experts and representatives of
one of the commercial companies we spoke with, it is better to
overestimate than underestimate costs.
Figure 5: Range of Possible Costs Narrows as Knowledge Is Gained:
[See PDF for image]
This figure s an illustration of how the range of possible costs
narrows as knowledge is gained.
Prior to the concept refinement phase, the margin of overestimating is
extremely high, and the margin for underestimating is at its' highest.
During the concept refinement state and the concept definition phase,
these margins narrow significantly. During the product development
phase, bot margins continue to narrow and approach cost estimating
certainty.
Source: GAO.
[End of figure]
Officials in Motorola's Government and Enterprise Mobility Solutions
business unit have established a rigorous review process, within which
proposed products are expected to provide cost estimates that fall
within an established range at successive review gates. At the first
review gate, the range is generous, allowing for estimates to be as
much as 75 percent too high and 25 percent too low. As the proposed
program moves through review gates and more becomes known about
requirements, technologies, and design, the established range narrows.
At product development initiation, the cost estimate is expected to be
no more than 10 percent higher and 5 percent lower than what actual
costs will be. IBM similarly allows products to deviate from their
original estimates as long as the deviation is within agreed-upon
limits, which are established in a contract between senior management
and project managers.[Footnote 31] Product development teams are
expected to execute according to the contract. According to IBM
officials, program cost growth of more than 5 or 10 percent is
generally not acceptable.
Successful companies have found that a relatively short, manageable
development cycle is a hallmark of an executable program. Officials at
Motorola told us that cycle time is one of the key metrics they use
when making decisions about what programs to pursue. They noted that
development programs longer than 2 or 3 years are not likely to be
initiated because the increased uncertainty would make it too difficult
to accurately plan for and execute the programs. This is consistent
with what we found at other successful commercial companies as well. By
constraining development cycles, it is easier to more accurately
estimate costs, and with more accurate cost estimates companies are
able to more precisely predict the future funding needs and effectively
allocate resources.
In 1998, the Under Secretary of Defense for Acquisition, Technology and
Logistics stated that the department's objective must and will be to
achieve acquisition cycle times no longer than 5 to 7 years, noting
that long acquisition cycle times for major defense programs lead to
higher costs and diminished military effectiveness. DOD's acquisition
policy, revised in 2003, suggests that system development should be
limited to a manageable time frame--about 5 years. An assessment of
DOD's acquisition system commissioned by the Deputy Secretary of
Defense in 2006 similarly recommended that programs should be time-
constrained with development cycles no longer than 6 years from
Milestone A to low-rate initial production.[Footnote 32] According to
the assessment, a time-constrained development cycle can reduce
pressure on investment accounts and increase funding stability for all
programs. With development cycles of 6 years or less, programs could be
fully funded within the FYDP time frame. In addition, constrained cycle
times would force programs to conduct more detailed systems engineering
analyses, increasing the likelihood that their requirements can be met
with available resources. We recently reported that while there are
isolated examples of DOD programs with cycle times shorter than 5
years, the majority of programs were initiated with much longer cycle
times. In some cases, these longer cycle times have been extended.
[Footnote 33] For example, FCS was expected to be a 7.5-year
development effort but is now a 12-year program and will likely be
extended again. Unconstrained and lengthy cycle times promote program
funding instability--especially when considering DOD's tendency to
change requirements and funding as well as frequent changes in
leadership.
A Portfolio-Based Investment Approach with Incremental Commitments
Supports Adequate and Stable Program Funding:
To ensure that resources are available to fully support their programs,
successful commercial companies we reviewed used an integrated
portfolio management approach to make incremental investment decisions.
In making these decisions, the companies looked across their entire
product portfolio within the context of available resources to ensure
that they were pursuing a balanced mix of products that can optimize
the return on their investment. Once a decision was made to pursue a
development program, the companies committed to fully funding the first
phase of that program, and made similar commitments prior to initiating
each successive phase. Before programs were approved to enter the next
phase of product development, Eli Lilly and IBM required program
officials and management to sign a contract that represented the
program's promise to deliver a given product that would meet the
customers' needs within established cost and time constraints. These
contracts also represented the ongoing commitment of management to
provide the necessary funding and other resources needed to ensure that
the program could be successfully executed.
By making incremental commitments companies are better positioned to
effectively manage and maintain stability in programs. For companies
like Eli Lilly--which tends to have development cycles of 10 to 15
years on average--the practice of making short-term, incremental
commitments mitigates many of the cost, schedule, and performance risks
inherent in long development programs. When Eli Lilly decides to pursue
a new drug development program, management and the project team enter
into a contract that identifies deliverables, time frames, and the
costs to get to the next milestone. The contract represents
management's commitment to fund the entire phase. IBM similarly
allocates funding in increments. For example, at the initial review
gate, funding is allocated to the product development team to support
the development of a sound business plan, which is expected to be
presented at the next review gate before any further funding commitment
is made.
Making incremental commitments also allows companies the flexibility to
effectively manage their portfolio. As they assess the value and
progress of each investment at multiple points throughout product
development, companies are able to make tough decisions to defer or
terminate programs and rebalance their portfolio. Terminating low-value
or poor-performing programs is important to the successful companies we
spoke with, because each dollar spent on a failing program is one less
dollar available for use elsewhere in the company. As a result, they
emphasize the need to make these tough decisions early. For example, at
Eli Lilly projects are terminated at early points in the review process
when it is determined that their critical success factors cannot be
achieved. Because Eli Lilly's projects typically have a high degree of
technical risk, only about 1 percent of those that start early
development actually make it to the marketplace.
We have found that successful portfolio management requires a strong
governance structure with committed leadership that empowers portfolio
managers to make decisions about the best way to invest resources and
holds those managers accountable for the outcomes they achieve. Because
portfolio managers are on the front line, several of the companies we
have reviewed empower these managers to make product investment
decisions and then hold them accountable for outcomes, not just for
individual products but also for the overall performance of the
portfolio. These organizations underscore the importance of holding
individuals accountable, aligning performance expectations with
organizational goals, and cascading those expectations down to lower
levels.
In contrast, DOD approves proposed programs without adequately
considering its overall portfolio and commits to programs with less
knowledge of cost and feasibility. Moreover, DOD lacks the
accountability that commercial companies emphasize is necessary to
ensure successful outcomes. Consequently, DOD starts more programs than
current and likely future resources can support and has less assurance
that its investment decisions address the right mix of warfighting
needs.
Conclusions:
At a time when the federal budget is strained by spending needs for a
growing number of national priorities, it is imperative that DOD get
the best value for every dollar of its significant investments. Yet DOD
has more major weapons system programs in its portfolio than it can
afford. All too often, these programs incur cost increases well beyond
original funding levels and significant delays in delivering weapon
systems to the warfighter. These outcomes are the direct result of
DOD's failure to prioritize its needs within resource constraints and a
funding process that allows programs to go forward with unpredictable
cost estimates and lengthy development cycles--not a sound basis for
allocating resources and ensuring program stability. Successful
commercial companies we have reviewed recognize the importance of
having adequate knowledge about requirements and available resources
before initiating a development program. These companies have learned
that disciplined processes with early management review points,
knowledge-based cost estimates, manageable product development cycles,
and a portfolio approach to funding product development efforts are key
to achieving program success. While DOD has recognized that more
accurate cost estimates coupled with manageable development cycles
could improve program outcomes, it has yet to take action. Until DOD
revises its policies and processes to address funding imbalances and
reform its funding approach, programs will continue to experience
instability and delayed delivery of needed capabilities to the
warfighter.
Recommendations for Executive Action:
To better ensure adequate funding for DOD's major weapon system
acquisition programs and to increase the likelihood of achieving
successful outcomes, we recommend that the Secretary of Defense take
the following three actions:
* Develop and implement a strategy to bring the department's current
portfolio into balance by aligning the number of programs and the cost
and schedule of those programs with available resources. In developing
and implementing a strategy, the department should determine ways to
prioritize needs and identify whether the budget and the FYDP should be
increased to more accurately reflect the actual costs of current
programs or whether the portfolio of current programs should be reduced
and lower-priority programs terminated to match available resources.
* Require that all new programs have manageable development cycles,
realistic cost estimates, and have planned and programmed full funding
for the entire development cycle.
* Require all cost estimates submitted for funding a program at
milestone decisions to be reported as a range of likely costs and
reflect the associated levels of risk and uncertainty. At Milestone A,
require estimates that allow for a wide range of likely costs. At
Milestone B, require estimates that, based on knowledge gained, are
more precise--in line with best practice standards.
Agency Comments and Our Evaluation:
In written comments on a draft of this report, DOD partially concurred
with our first and second recommendations and non-concurred with the
third. DOD's partial concurrences are rooted in the belief that its
current policies and initiatives address our recommendations. We agree
that aspects of DOD's current policies appear consistent with our
recommendations, and the initiatives could contribute to better program
outcomes if implemented appropriately. However, we have found no
evidence of widespread adoption of these policies or any other process
that would better ensure adequate funding for DOD's major weapon system
acquisition programs and increase the likelihood of achieving
successful outcomes.
DOD partially concurred with our first recommendation--that the
Secretary of Defense develop and implement a strategy to bring the
department's current portfolio into balance by aligning the number of
programs and the cost and schedule of those programs with available
resources. In its written response, DOD identified the 2003
restructuring of the PPBE process as one initiative it has taken to
address the mismatch between program commitments and available
resources. Yet, since the restructuring, more than 5 years--and several
budget cycles--have lapsed, and the department is still committed to
more programs than it can support. While DOD notes that seeing the
effects of process changes will take years, the department does not
indicate how many additional years are needed to see positive results
or what incremental changes it should be held accountable for. In the
meantime, the department continues to risk tax dollars and delaying the
delivery of needed capabilities to the warfighter. The department also
cites several other, more recent initiatives like Capability Portfolio
Management, Capital Accounts, and Configuration Steering Boards that
are intended to balance needs with resources; drive more realistic,
cost-effective plans and budgets; limit requirements growth; and
improve oversight. While we believe that these initiatives, like many
before them, are well intentioned, DOD has not established indicators
to measure their success, and it is unclear how the initiatives will
help bring DOD's current portfolio into balance. Further, we are
concerned that they do not go far enough to address the systemic
cultural and structural problems identified in this report.
DOD states that "external influences" can cause turbulence in program
execution. Specifically, the department notes that it does not control
the amount of Total Obligation Authority (TOA) available in the FYDP or
how much funding is appropriated by Congress. While we acknowledge that
decisions made by OMB and Congress can directly affect program funding,
we believe that our recommendation is sound regardless of the total
amount of funding provided to the department. As we have recently
reported, substantially more funding has been committed to develop new
weapon systems since 2000, yet cost overruns and schedule delays have
increased even more. By focusing on external influences DOD misses the
main point of our recommendation--to align the number of programs in
the current portfolio and the cost and schedule of those programs with
available resources.
DOD also partially concurred with our second recommendation--to require
all new programs to have manageable development cycles, realistic cost
estimates, and full funding for the entire development cycle. Again,
the department's partial concurrence is based on its belief that
certain DOD policies and initiatives respond to this recommendation. We
agree that some DOD policies and initiatives emphasize a knowledge-
based approach to acquiring weapon systems, but acquisition officials
do not effectively implement these policies. DOD programs are often
initiated with development cycle times that are much longer than the 5
years the department's policy suggests. As we note in this report,
programs with development cycles of 6 years or less could be fully
funded in a single FYDP and would be less likely to experience funding
instability. DOD further commented that the current acquisition policy
requires cost estimates and, where required, independent cost estimates
to be completed at key program milestones and full funding to be
programmed prior to Milestone B. However, we found that because DOD's
cost estimates at key milestones are based on limited knowledge and
optimistic assumptions about requirements and technology, they often
significantly underestimate true costs and do not provide the necessary
foundation for making accurate funding commitments. In addition, our
analysis of DOD's selected acquisition reports and budget data found
that DOD often does not commit full funding to its major weapon system
acquisitions when they are initiated, despite the department's policy
to do so.
While the department generally agreed that product development cycles
should be "manageable," it questioned the comparability of DOD to the
private sector firms we reviewed with regard to industry goals,
incentives, products, and services. This misses the point. We are
recommending that development cycles for all new DOD programs be
manageable--a guiding principle that the successful commercial
companies we have reviewed follow to enable better program outcomes.
The department's own acquisition policy suggests development cycles of
about 5 years--an objective the department established a decade ago.
Similarly, a recent study commissioned by DOD recommended that programs
should be time-constrained with development cycles no longer than 6
years. In general, DOD indicated an interest in obtaining more insight
into our methodology. Much of this information, including information
about the companies and commodities that we have reviewed in the past,
can be found in most of the works cited in the related GAO products
section of this report.
DOD did not concur with our third recommendation. While DOD agreed that
risks should be considered at key milestones, it did not agree that
cost estimates submitted at milestone decisions should be reported as a
range of likely costs--a wide range at Milestone A based on limited
knowledge, narrowing to a more precise range at Milestone B as more
knowledge is gained. The CAIG accounts for cost risk by examining
program schedule durations, technical risks, contract vehicles,
incentives, and management structures. However, DOD's current practice
of presenting a single point estimate as the "most likely cost"
provides decision makers with limited insight into these risks. DOD
also cited concerns that a certain method for calculating cost ranges
can produce misleading results. While we do not advocate a specific
method for calculating a range of potential costs, we maintain that
presenting such a range would provide decision makers with additional
knowledge about the level of risk in a proposed program.
Finally, DOD states that our report does not consider certain causal
factors bearing on the estimates we cite in table 1. It is unclear how
DOD came to this conclusion. Our report states that inaccurate cost
estimates are often the result of limited knowledge about requirements
and technologies. We also note that conducting early disciplined
analyses, such as systems engineering, builds knowledge that enables a
developer to identify and resolve gaps between requirements and
available resources before beginning product development. As knowledge
increases, uncertainty and associated risks in the cost estimate
decrease. Therefore, we believe that following a disciplined knowledge-
based process--a recommendation we have made repeatedly in our reviews
of DOD's major weapon system programs--would obviate most of the causal
factors DOD cites, including "incomplete, error-full, or volatile
requirements or program specification" and "engineering change
proposals."
DOD's written comments are reprinted in appendix II. The Department
also provided technical comments, which were incorporated as
appropriate.
We are sending copies of this report to the Secretary of Defense; the
Secretaries of the Air Force, Army, and Navy; and the Director of the
Office of Management and Budget. We will provide copies to others on
request. This report will also be available at no charge on GAO's Web
site at [hyperlink, http://www.gao.gov].
If you have any questions about this report or need additional
information, please contact me at (202) 512-4841 or sullivanm@gao.gov.
Key contributors to this report were John Oppenheim, Assistant
Director; Travis Masters; Keith Hudson; Victoria Klepacz; Karen Sloan;
Karen Richey; and John Krump.
Signed by:
Michael J. Sullivan:
Director, Acquisition and Sourcing Management:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
This report assesses the extent to which the Department of Defense's
(DOD) resource allocation approach supports stability within and across
its major weapon system acquisition programs. Specifically, our
objectives were to (1) assess how DOD budgets for and funds its major
weapon system acquisition programs, (2) identify key factors that
influence the effectiveness of this approach, and (3) identify proven
processes and practices that could help improve DOD's ability to
effectively allocate resources to its acquisition programs.
To assess the effectiveness of DOD process for funding for its major
weapon system acquisition programs, we compared initial approved cost
estimates for 20 of the 95 major weapons programs in DOD's current
portfolio to the amount of development funding contained in each
program's budget documentation from the year in which development
started--which covers the initial Future Years Defense Program (FYDP)
time frame. The 20 programs we reviewed initiated system development
from 1999 through 2006 and represent almost 35 percent of the total
expected cost of DOD's current portfolio of major weapon system
programs. Due to data limitations, we were not able to assess a number
of programs, including several major ship acquisitions, in our cost
estimate and funding analysis. While our findings may not be
generalized to all of DOD's acquisition programs, we believe that
capturing more than one-third of DOD's planned investment is
significant and provides insights into DOD's resource allocation
approach for its major weapon system programs. To gain further insights
into the impact of DOD's funding process on individual programs, we
conducted more detailed analysis for 5 high-profile programs, which
represent the nearly 75 percent of DOD's planned investment in the 20
programs: Global Hawk, Joint Strike Fighter (JSF), Future Combat System
(FCS), Warfighter Information Network-Tactical (WIN-T), and Multi-
mission Maritime Aircraft (MMA).[Footnote 34] We obtained cost and
baseline data from DOD's selected acquisition reports (SAR) submitted
through December 2007, FYDP and program funding data from the
President's Budget exhibits through fiscal year 2008, and program cost
estimates from the Office of the Secretary of Defense's (OSD) Cost
Analysis Improvement Group (CAIG) and the service cost analysis centers
developed to support each program's Milestone B review. Our analysis
began with cost and funding data from the fiscal year that each program
entered system development--received Milestone B approval--and ended
with the data reported in the December 2007 SARs. We also examined and
compared individual program cost estimates from the CAIG, service cost
analysis centers, and program offices to determine how they differed
from current program costs.
To better understand how DOD funds its weapon system acquisitions and
to identify the key factors that influence its effectiveness, we
reviewed DOD Acquisition Regulations, DOD Financial Management
Regulations, Title 10 of the United States Code, Planning, Programming,
Budgeting, and Execution guidance from Management Information Directive-
913, as well as individual military service budgeting policies and
guidance. We also met with knowledgeable officials from the OSD CAIG;
OSD Program Analysis and Evaluation; the Office of the Under Secretary
of Defense for Acquisition, Technology and Logistics; the military
service cost analysis centers; military service planning and
programming organizations; the Defense Acquisition University; the
Brookings Institution; the Institute for Defense Analysis; and the
Naval Postgraduate School.
To identify proven cost estimating and budgeting processes and
practices that could be used by DOD to improve its resource allocation
process, we relied on our prior work in best practices at successful
commercial companies and reviewed documentation and interviews from
previous GAO best practices work. We also conducted follow-up
interviews with key officials from Eli Lilly, IBM, and Motorola
[Footnote 35] to gain a better understanding of how they prioritize
projects for funding and ensure that each product receives and
maintains adequate funding. We also relied on our Cost Assessment
Guide--which provides a cost-estimating methodology based on best
practices--and numerous other GAO products, including our extensive
body of best practices work.
We conducted this performance audit from June 2007 to May 2008 in
accordance with generally accepted government auditing standards. Those
standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.
[End of section]
Appendix II: Comments from the Department of Defense:
Office Of The Under Secretary Of Defense:
Acquisition, Technology And Logistics:
3000 Defense Pentagon
Washington, DC 20301-3000
June 13, 2008:
Mr. Michael J. Sullivan:
Director, Acquisition and Sourcing Management:
U.S. Government Accountability Office:
441 G Street, N.W.
Washington, DC 20548:
Dear Mr. Sullivan:
This is the Department of Defense (DoD) response to the GAO draft
report GAO-08-619, "Defense Acquisitions: A Knowledge-Based Funding
Approach Could Improve Major Weapon System Program Outcomes," dated May
15, 2008 (GAO Code 120665).
The Department partially concurs with two recommendations and
nonconcurs with the other draft report recommendation. The rationale
for DoD's position is included in the enclosure.
The Department appreciates the opportunity to comment on the draft
report. Technical comments were provided separately. My point of
contact for this effort is Mr. Joseph A. Alfano, 703-697-3343.
Sincerely,
Signed by:
Dr. Nancy L. Spruill:
Director:
Acquisition Resources & Analysis:
Enclosure: As stated:
GAO Draft Report Dated May 15, 2008:
GAO-08-619 (GAO Code 120665):
"Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve
Major Weapon System Program Outcomes"
Department Of Defense Comments To The GAO Recommendations:
Recommendation 1: The GAO recommends that the Secretary of Defense
develop and implement a strategy to bring the department's current
portfolio into balance by aligning the number of programs and the cost
and schedule of those programs with available resources. In developing
and implementing a strategy, the Department should determine ways to
prioritize needs and identify whether the budget and the Future Years
Defense Program should be increased to more accurately reflect the
actual costs of current programs or whether the portfolio of current
programs should be reduced and lower priority programs terminated to
match available resources. (p. 28/GAO Draft Report)
DOD Response: Partially concur. The Department goes to great lengths in
order to develop overall investment strategy and guidance, then flows
that guidance to the appropriate levels, and then prioritizes
requirements in order to "align the number of programs and the cost and
schedule of those programs with available resources." The restructure
of the Planning, Programming, and Budgeting System into the Planning,
Programming, Budgeting, and Execution process was one of several
initiatives the Department has taken towards this end; however,
positive results from these types of process changes will take years
and experience across several programs in order to validate their
positive effects. Yet while the Department has control over its
programming and budgeting processes (within limits set by the Office of
Management and Budget) it does not control the amount of Total
Obligation Authority available over the Future Years Defense Program,
nor how much funding is appropriated by Congress. At times, these
external influences will also cause turbulence in program execution.
Over the last several years the Department has taken on several
initiatives to address this GAO recommendation. A recent Defense
initiative known as Capability Portfolio Management is intended to
provide an enterprise-level, horizontal (cross-component) view of the
Department to better balance and harmonize the Departments capability
needs with existing and planned force management and development
efforts and produce strategically aligned outcomes optimized for the
enterprise. As of February 2008, Department leadership has established
nine Capability Portfolio Managers aligned to the nine Tier 1 Joint
Capability Areas (the Department's capabilities language), formalized
four and is testing the other five. Capability Portfolio Managers are
focused on highlighting opportunities to better integrate, coordinate
and synchronize programs to strategic intent and capability priorities
within resource and time constraints.
With respect to budgeting, more realistic, cost-effective plans and
budgets are goals for the integrated Planning, Programming, Budgeting
and Execution process. Establishing Capital Accounts will result in
more stable, predictable acquisition lifecycle management program
execution, and the process would benefit from this guaranteed funding
stream. Pilot programs provide valuable lessons learned to apply
capital accounts to a wide variety of acquisition and sustainment
programs. Current initiatives include coordination with the Office of
Management and Budget Program Assessment Rating Tool and establishment
of an authoritative financial information source by integrating
transactional-level accounting data. Further, Configuration Steering
Boards have been implemented across all major programs to limit cost
and requirements growth through senior level evaluation of trade-offs
and opportunities to descope programs and seek savings.
In the oversight of Major Defense Acquisition Programs and the Defense
Acquisition Board process, multiple initiatives are being implemented
to improve and standardize the milestone decision process. The Defense
Acquisition Executive Summaries system is being reengineered to ensure
visibility for senior leaders. The system will help facilitate
information sharing, accurate input of data, and tracking of negative
trends to focus on problem areas. Systems to model time and new
technology factors are being developed to budget effectively to
changing needs in a rapidly evolving technological development
environment.
Recommendation 2: The GAO recommends that the Secretary of Defense
require that all new programs have manageable development cycles,
realistic cost estimates, and have planned and programmed full funding
for the entire development cycle. (p. 28/GAO Draft Report)
DOD Response: Partially concur. Current DoD acquisition policy, as
articulated in DoDI 5000.2, establishes an evolutionary, event-based
acquisition approach as the Department's preferred strategy for rapid
acquisition of mature technology for the user. Additionally, current
policy requires cost estimates and, where required, independent cost
estimates to be completed at key program milestones and full funding to
be programmed prior to Milestone B.
These existing policies are consistent with the GAO's recommendation.
Additionally, the Department has recently introduced a number of new
policies that we believe will substantively contribute to more
predictable program outcomes:
* Our prototyping and competition policy requires program managers to
plan for two or more competing teams to produce prototypes of key
system elements up to or through Milestone B. The intent is to reduce
technical risk, validate cost estimates, evaluate manufacturing
processes, and refine requirements.
* We have established Configuration Steering Boards for all ACAT I
programs to review proposed requirements changes, a significant
contributor to increased cost and extended development cycles, and any
technical configuration changes that have the potential to result in
increases to cost and or lengthier schedules. Such changes will
generally be rejected, deferring them to future development increments.
Finally, the Department is considering policy initiatives that are
designed to reduce program instability and improve schedule and cost
predictability. These initiatives include: (1) moving the Preliminary
Design Review before Milestone B, to improve our understanding of cost
and technical requirements before program initiation; and (2) requiring
a Milestone Decision Authority-conducted post-Critical Design Review
Assessment in order to ensure that programs are proceeding through the
System Development and Demonstration phase consistent with sound
engineering practice and the approved acquisition program baseline.
Together we believe these new or pending policies will enhance existing
policy, and substantively contribute to improved cost estimates, better
informed decision making and shorter cycle times.
Recommendation 2 includes a provision that all new development programs
have "manageable development cycles." This conclusion is based on the
commercial product development models reviewed by GAO and mentioned in
the report, specifically those from Eli Lilly, IBM, and Motorola. While
we generally agree that DoD product development cycles should be
"manageable," these development cycles may bear little relation to
those of the three commercial firms cited by GAO in the report.
The use by GAO of three commercial firm product development models as a
basis for determining `best practice' in DoD has several flaws. First,
DoD as a public sector institution, in contrast to a private sector
firm, has key differences in goals and incentives. All of the DoD
weapon system programs cited in the report are developed through
contracts from the public to the private sector, which contain
significantly different incentives and provisions than the product
development programs typically undertaken internally by firms in the
private sector. It may be more appropriate that instead of benchmarking
DoD against private sector firms, GAO should be benchmarking DoD
against best practices in the public sector throughout the world.
Further the commodity classes for comparison to DoD development
programs for major weapon systems that GAO selected, we believe, are
not a good comparison. The three firms cited in the report are in the
pharmaceutical, computer products and services, and the radio and
cellular telephony sectors. Yet the implication of the report is that
the development cycles in these product sectors can be directly applied
to systems like the Joint Strike Fighter program. At a minimum, GAO
might better examine commercial product developments in comparable
commodity sectors, such as the commercial aircraft sector, to avoid
drawing conclusions that may be invalid based on analogies to
dissimilar commodity sectors.
In order to more thoroughly review the study, the DoD requested that
the GAO share its methodology, raw data, interview notes, detailed
analyses, working papers, etc. Although to date the GAO has not
provided the aforementioned materials, the Department would be pleased
to work with GAO to better understand the details of such a comparison.
Recommendation 3: The GAO recommends that the Secretary of Defense
require all cost estimates submitted for funding a program at Milestone
decisions to be reported as a range of likely costs and the associated
levels of risk and uncertainty. At Milestone A, require estimates that
allow for a wide range of likely costs. At Milestone B, require
estimates that, based on knowledge gained, are more precise--in line
with best practice standards. (p. 28/GAO Draft Report)
DOD Response: Non-concur. While we and the DoD Cost Analysis
Improvement Group (CRAIG) agree with the spirit of the recommendation
to consider risks at key milestones in development programs; however
the GAO recommendation implies that the DoD should calculate a range of
costs and associated confidence levels using Monte Carlo analysis
techniques. The CAIG has observed limitations to this analysis
methodology when used in practice: Most mathematical implementations of
this methodology assume no correlation of work breakdown structure
elements in developing distributions of potential cost outcomes. In
most cases this simplifying assumption is incorrect, particularly for
cost estimates developed for product-oriented work breakdown
structures, as are commonly used in the DoD. This assumption results in
calculated distributions of potential cost outcomes that are "too
narrow" relative to the actual distribution of cost outcomes, providing
misleading results.
The CAIG does routinely calculate cost risks for major milestone
reviews by examining the key structural elements of cost estimates,
including schedule durations, technical risks, contract vehicles,
incentives, and management structures. In certain situations, the CAIG
has reported a range of life-cycle costs because of the poor quality of
the definition of the program to be undertaken in the Cost Analysis
Requirements Description document. For example, the CAIG reported a
likely range of Future Combat Systems (FCS) development costs on
several occasions, a fact that GAO did not include in its report. In
summary, the CAIG tailors the risk analysis to the specific program and
milestone at hand, calculating and presenting ranges of costs when it
is appropriate to do so. The specification. of a single methodology and
practice to be used at each and every milestone review of a Major
Defense Acquisition Program is not appropriate.
In the GAO report, Table 1, Development Costs Estimates and Baselines
for 20 Major Weapon System Programs, contains information on
development cost estimates and cost baselines. Unfortunately, the
report does not consider causal factors bearing on the estimates: (1)
methodological issues, (2) incomplete, errorful, or volatile
requirements or program specification, (3) engineering change
proposals, (4) Congressionally mandated changes in program funding
levels or unit quantities, etc. Additionally, the report does not
specify the metrics whereby a cost estimate is considered as
acceptable.
In order to assist in maintaining an awareness of overall risk with its
acquisition programs, the Air Force is evaluating the idea of setting a
goal of annual updates to program cost estimates for major defense
acquisition programs, versus the DoD 5000 series requirement to update
for each milestone review. This should bring newly-acquired knowledge
to bear more quickly and bring problems to light in a more timely
manner.
[End of section]
Related GAO Products:
Defense Acquisitions: Assessments of Selected Weapon Programs.
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-467SP].
Washington, D.C.: March 31, 2008.
Best Practices: Increased Focus on Requirements and Oversight Needed to
Improve DOD's Acquisition Environment and Weapon System Quality.
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-294], Washington,
D.C.: February 1, 2008.
Cost Assessment Guide: Best Practices for Estimating and Managing
Program Costs. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-
1134SP], Washington, D.C.: July 2007.
Defense Acquisitions: Realistic Business Cases Needed to Execute Navy
Shipbuilding Programs. [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-07-943T], Washington, D.C.: July 24, 2007.
Best Practices: An Integrated Portfolio Management Approach to Weapon
System Investments Could Improve DOD's Acquisition Outcomes.
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-388], Washington,
D.C.: March 30, 2007.
Space Acquisitions: DOD Needs to Take More Action to Address
Unrealistic Initial Cost Estimates of Space Systems. [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-07-96], Washington, D.C.:
November 17, 2006.
Defense Acquisitions: Major Weapon Systems Continue to Experience Cost
and Schedule Problems under DOD's Revised Policy. [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-06-368]. Washington, D.C.: April
13, 2006.
Defense Acquisitions: Actions Needed to Get Better Results on Weapons
Systems Investments. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-
06-585T]. Washington, D.C.: April 5, 2006.
Best Practices: Better Support of Weapon System Program Managers Needed
to Improve Outcomes. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-
06-110]. Washington, D.C.: November 30, 2005.
Future Years Defense Program: Actions Needed to Improve Transparency of
DOD's Projected Resource Needs. [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-04-514]. Washington, D.C.: May 7, 2004.
Best Practices: Capturing Design and Manufacturing Knowledge Early
Improves Acquisition Outcomes. [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-02-701]. Washington, D.C.: July 15, 2002.
Best Practices: Better Matching of Needs and Resources Will Lead to
Better Weapon System Outcomes. [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-01-288]. Washington, D.C.: March 8, 2001.
Footnotes:
[1] GAO, Best Practices: An Integrated Portfolio Management Approach to
Weapon System Investments Could Improve DOD's Acquisition Outcomes.
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-388], Washington,
D.C.: March 30, 2007.
[2] Global Hawk, an Air Force unmanned aircraft system, is intended to
provide intelligence, surveillance, and reconnaissance capabilities;
JSF is a joint Air Force, Navy, and Marine Corps program to develop and
field stealthy fighter aircraft to replace DOD's aging fighter and
attack aircraft; FCS is an Army program intended to provide advanced,
networked combat and sustainment systems, unmanned ground and air
vehicles, and unattended sensors and munitions; WIN-T is intended to
provide the Army with a high-speed, high-capacity communications
network; MMA is a Navy program intended to provide persistent
antisubmarine and antisurface warfare, and intelligence, surveillance,
and reconnaissance capabilities.
[3] GAO, Cost Assessment Guide: Best Practices for Estimating and
Managing Program Costs. [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-07-1134SP] (Washington, D.C.: July 2007).
[4] Full funding is a DOD 5000.2 requirement for formal program
initiation of an acquisition program. In this sense, full funding means
having an approved current (and projected) resource stream to execute
the acquisition program; that is, program funding is included both in
the budget and in the out-years of the Future Years Defense Program
sufficient to cover the current and future efforts described in the
acquisition strategy.
[5] DOD modified the PPBS process, renamed the PPBE, in 2003. PPBE in
its entirety is not implemented every year even though DOD must request
funding from congress annually. Full-scale planning and programming
activities are conducted in even-numbered years (called on-years) while
programming adjustments are made in odd-numbered years (called off-
years). As a result of this modification, the number of years covered
by the FYDP now alternates between 5 years (the President's budget plus
4 years) and 6 years (the President's budget plus 5 years).
[6] The Milestone Decision Authority for all Major Defense Acquisition
Programs is the Under Secretary of Defense for Acquisition, Technology
& Logistics unless delegated to the head of a DOD component.
[7] Although DOD policy requires full funding within the FYDP, most
major weapon system acquisition programs have development cycles that
extend beyond the FYDP time period. However, there is no requirement to
constrain development cycles to fit within that same time period.
[8] U.S.C. Title 10 § 2366(a), which was enacted in 2006, specifies
that a major defense acquisition program may not receive Milestone B
approval, or Key Decision Point B approval in the case of a space
program, until the Milestone decision authority certifies that--(1) the
program technology has been demonstrated in a relevant environment; (2)
the program demonstrates a high likelihood of accomplishing its
mission; (3) the program is affordable when considering the per unit
cost and the total acquisition cost in the context of the total
resources available during the period covered by the future-years
defense program submitted during the fiscal year in which the
certification is made; (4) the department has completed an analysis of
alternatives; (5) the program is affordable when considering the
ability of DOD to accomplish the program's mission using alternative
systems; (6) the Joint Requirements Oversight Council has accomplished
its duties with respect to the program pursuant to section 181 (b) of
this title, including an analysis of the operational requirements for
the program; and (7) the program complies with all relevant policies,
regulations, and directives of DOD.
[9] Programs are not required to hold a Milestone A review, the point
at which concept refinement ends and technology development begins.
However, in 2007, Congress enacted legislation, which specifies that a
major defense acquisition program beginning after March 2008 may not
receive Milestone A approval, to begin a technology development
program, until the Milestone Decision Authority certifies to Congress
that (1) the system fulfills an approved initial capabilities document;
(2) the system is being executed by an entity with a relevant core
competency as identified by the Secretary of Defense; (3) if the system
duplicates a capability already provided by an existing system, the
duplication provided by such system is necessary and appropriate; and
(4) a cost estimate for the system has been submitted. DOD is currently
revising its policy and guidance for conducting and certifying
Milestone A reviews.
[10] See Related GAO Products at the end of this report.
[11] The funding needed within the FYDP for each program was calculated
using the funding data contained in the RDT&E Annual Funding Summary of
the first SAR issued after system development began. The amount of
funding established in the FYDP was calculated using each program's
RDT&E budget justification documentation for the year in which system
development was initiated.
[12] DOD has a formal process for reprogramming appropriated funds,
including if necessary, congressional notification and approval.
[13] GAO, Space Acquisitions: DOD Needs to Take More Action to Address
Unrealistic Initial Cost Estimates for Space Systems. [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-07-96], Washington, D.C.:
November 17, 2006.
[14] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-388].
[15] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-388].
[16] Assessment Panel of the Defense Acquisition Performance Assessment
Project for the Deputy Secretary of Defense, Defense Acquisition
Performance Assessment Report (January 2006); Defense Science Board
Summer Study on Transformation: A Progress Assessment (February 2006);
RAND, Historical Cost Growth of Completed Weapon System Programs
(2006); Institute for Defense Analysis, Costs Growth in Major Weapon
Procurement Programs, Presentation (2005).
[17] Defense Science Board, Report of the Defense Science Board /Air
Force Advisory Board, Joint Task Force on Acquisition of National
Security Space Programs, May 2003
[18] Systems engineering is a technical management tool that provides
the knowledge necessary to translate requirements into specific,
achievable capabilities. By using the tools of systems engineering
during the early phases of concept refinement and technology
development, acquisition decision makers and developers can work
together to close gaps between requirements and available resources
well before system development starts.
[19] An AOA should compare the costs and benefits of alternative
solutions for meeting a validated need. As such an AOA should assess
the life-cycle cost, schedule, and operational effectiveness of each
possible solution and identify a preferred alternative.
[20] GAO, Defense Acquisitions: Assessments of Selected Major Weapon
Programs. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-467SP].
Washington, D.C.: March 31, 2008; Defense Acquisitions: Major Weapon
Systems Continue to Experience Cost and Schedule Problems under DOD's
Revised Policy. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-
368]. Washington, D.C.: April 13, 2006.
[21] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-368] and
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-388].
[22] Institute for Defense Analysis, Future Combat Systems (FCS) Cost
Review: Summary of Findings, P-4212, April 2007, p. S-2.
[23] GAO, Tactical Aircraft: Opportunity to Reduce Risks in the Joint
Strike Fighter Program with Different Acquisition Strategy. [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-05-271] (Washington, D.C.: March
15, 2005).
[24] GAO, Space Acquisitions: DOD Needs to Take More Action to Address
Unrealistic Initial Cost Estimates for Space Systems. [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-07-96] (Washington, D.C.: Nov.
17, 2006).
[25] GAO, Defense Acquisitions: Realistic Business Cases Needed to
Execute Navy Shipbuilding Programs. [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-07-943T], Washington, D.C.: July 24, 2007.
[26] For management to make good decisions about programs, our Cost
Assessment Guide calls for a quantitative risk and uncertainty analysis
to assess variability in an estimate.
[27] A Defense Science Board report recommended funding space
acquisition programs at the 80-percent confidence level (Report of the
Defense Science Board/Air Force Advisory Board, Joint Task Force on
Acquisition of National Security Space Programs, May 2003); and
subsequently the Air Force issued guidance in March 2007 stating that
an 80-percent confidence level is the objective for space programs, but
it is too soon to assess whether this policy has resulted in more
realistic estimates.
[28] GAO, Future Years Defense Program: Risks in Operation and
Maintenance and Procurement Programs. [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-01-33] (Washington, D.C.: October
5, 2000); Best Practices: An Integrated Portfolio Management Approach
to Weapon System Investments Could Improve DOD's Acquisition Outcomes.
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-388] (Washington,
D.C.: March 30, 2007).
[29] GAO, Best Practices: Better Support of Weapon System Program
Managers Needed to Improve Outcomes, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-06-110] (Washington, D.C.: November 30, 2005).
[30] See Related GAO Products at the end of this report.
[31] IBM's contracts are not focused solely on program costs; they also
establish goals, objectives, and allowable deviations for other program
measures such as schedule and revenue.
[32] Assessment Panel of the Defense Acquisition Performance Assessment
Project for the Deputy Secretary of Defense, Defense Acquisition
Performance Assessment Report (Jan. 2006).
[33] Defense Acquisitions: Assessments of Selected Major Weapon
Programs. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-467SP].
Washington, D.C.: March 31, 2008.
[34] Global Hawk, an Air Force unmanned aircraft system, is intended to
provide intelligence, surveillance, and reconnaissance capabilities;
JSF is a joint Air Force, Navy, and Marine Corps program to develop and
field stealthy fighter aircraft to replace DOD's aging fighter and
attack aircraft; FCS is an Army program intended to provide advanced,
networked combat and sustainment systems, unmanned ground and air
vehicles, and unattended sensors and munitions; WIN-T is intended to
provide the Army with a high-speed, high-capacity communications
network; MMA is a Navy program intended to provide persistent
antisubmarine and antisurface warfare, and intelligence, surveillance,
and reconnaissance capabilities.
[35] Eli Lilly is one of the largest corporations in the world and
engages in pharmaceutical research and development around the world;
IBM is the largest supplier of hardware, software, and information
technology services; and Motorola is a Fortune 100 global
communications leader. We also contacted the other two companies that
provided input to our prior review--Proctor & Gamble and Caterpillar--
but relevant officials were not available to meet with us for follow-up
discussions during the timeframes of our review.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
