Defense Acquisitions
Observations on Weapon Program Performance and Acquisition Reforms Gao ID: GAO-10-706T May 19, 2010The past two years have seen the Congress and DOD take meaningful steps towards addressing long-standing weapon acquisition issues--an area that has been on GAO's high risk list since 1990. This testimony focuses on the progress DOD has made in improving the planning and execution of its weapon acquisition programs and the potential for recent acquisition reforms to improve program outcomes. The testimony includes observations about (1) DOD's efforts to manage its portfolio of major defense acquisition programs, (2) the knowledge attained at key junctures of a subset of 42 weapon programs from the 2009 portfolio, (3) other factors that can affect program execution, and (4) DOD's implementation of recent acquisition reforms. The testimony is based on the results of our annual assessment of weapon programs. To conduct the assessment, GAO analyzed data on the composition of DOD's portfolio of major defense acquisition programs. GAO also collected data from program offices on technology, design, and manufacturing knowledge, as well as on other factors that can affect program execution. GAO has made numerous recommendations on weapon system acquisition in prior work but is not making any new recommendations in this testimony.
While DOD still faces significant challenges in managing its weapon system programs, the current acquisition reform environment provides an opportunity to leverage the lessons of the past and manage risks differently. This environment is shaped by significant acquisition reform legislation, constructive changes in DOD's acquisition policy, and initiatives by the administration, including making difficult decisions to terminate or trim numerous weapon systems. To sustain momentum and make the most of this opportunity, it will be essential that decisions to approve and fund acquisitions be consistent with the reforms and policies aimed at getting better outcomes. DOD has started to reprioritize and rebalance its weapon system investments. In 2009, the Secretary of Defense proposed canceling or significantly curtailing weapon programs with a projected cost of at least $126 billion that he characterized as too costly or no longer relevant for current operations, while increasing funding for others that he assessed as higher priorities. Congress supported several of the recommended terminations. DOD plans to replace several of the canceled programs in fiscal years 2010 and 2011, hopefully with new, knowledge-based acquisition strategies, because the warfighter need remains. The most significant of these will be the effort to restructure the Army's terminated Future Combat System program. At the same time, however, DOD's portfolio of major defense acquisition programs continues to grow. Between December 2007 and July 2009, the number of major defense acquisition programs grew from 96 to 102 programs. GAO has previously reported that DOD should continue to work to balance its weapon system portfolio with available funding, which includes reducing the number or size of weapon system programs, or both, and assessing the affordability of new programs and capabilities in the context of overall defense spending. At the program level, our recent observations present a mixed picture of DOD's adherence to a knowledge-based acquisition approach, which is a key for improving acquisition outcomes. For 42 programs GAO assessed in depth in 2010, there has been continued improvement in the technology, design, and manufacturing knowledge programs had at key points in the acquisition process. However, most programs are still proceeding with less knowledge than best practices suggest, putting them at higher risk for cost growth and schedule delays. A majority of programs have also experienced requirements changes, software development challenges, or workforce issues, or a combination, which can affect program stability and execution. DOD has begun to implement a revised acquisition policy and congressional reforms that address many of these areas. For example, eight programs we examined in the technology development phase plan to test competitive prototypes before starting system development and seven programs plan to hold early systems engineering reviews. If DOD consistently applies this policy, the number of programs adhering to a knowledge-based acquisition should increase and the outcomes for DOD programs should improve.
GAO-10-706T, Defense Acquisitions: Observations on Weapon Program Performance and Acquisition Reforms
This is the accessible text file for GAO report number GAO-10-706T
entitled 'Defense Acquisitions: Observations on Weapon Program
Performance and Acquisition Reforms' which was released on May 19,
2010.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Testimony:
Before the Subcommittee on National Security and Foreign Affairs,
Committee on Oversight and Government Reform, House of Representatives:
United States Government Accountability Office:
GAO:
For Release on Delivery:
Expected at 10:00 a.m. EDT:
Wednesday, May 19, 2010:
Defense Acquisitions:
Observations on Weapon Program Performance and Acquisition Reforms:
Statement of Michael J. Sullivan:
Director, Acquisition and Sourcing Management:
GAO-10-706T:
GAO Highlights:
Highlights of GAO-10-706T, a testimony to Subcommittee on National
Security and Foreign Affairs, Committee on Oversight and Government
Reform, House of Representatives.
Why GAO Did This Study:
The past two years have seen the Congress and DOD take meaningful
steps towards addressing long-standing weapon acquisition issues”an
area that has been on GAO‘s high risk list since 1990. This testimony
focuses on the progress DOD has made in improving the planning and
execution of its weapon acquisition programs and the potential for
recent acquisition reforms to improve program outcomes.
The testimony includes observations about (1) DOD‘s efforts to manage
its portfolio of major defense acquisition programs, (2) the knowledge
attained at key junctures of a subset of 42 weapon programs from the
2009 portfolio, (3) other factors that can affect program execution,
and (4) DOD‘s implementation of recent acquisition reforms. The
testimony is based on the results of our annual assessment of weapon
programs. To conduct the assessment, GAO analyzed data on the
composition of DOD‘s portfolio of major defense acquisition programs.
GAO also collected data from program offices on technology, design,
and manufacturing knowledge, as well as on other factors that can
affect program execution.
GAO has made numerous recommendations on weapon system acquisition in
prior work but is not making any new recommendations in this testimony.
What GAO Found:
While DOD still faces significant challenges in managing its weapon
system programs, the current acquisition reform environment provides
an opportunity to leverage the lessons of the past and manage risks
differently. This environment is shaped by significant acquisition
reform legislation, constructive changes in DOD‘s acquisition policy,
and initiatives by the administration, including making difficult
decisions to terminate or trim numerous weapon systems. To sustain
momentum and make the most of this opportunity, it will be essential
that decisions to approve and fund acquisitions be consistent with the
reforms and policies aimed at getting better outcomes.
DOD has started to reprioritize and rebalance its weapon system
investments. In 2009, the Secretary of Defense proposed canceling or
significantly curtailing weapon programs with a projected cost of at
least $126 billion that he characterized as too costly or no longer
relevant for current operations, while increasing funding for others
that he assessed as higher priorities. Congress supported several of
the recommended terminations. DOD plans to replace several of the
canceled programs in fiscal years 2010 and 2011, hopefully with new,
knowledge-based acquisition strategies, because the warfighter need
remains. The most significant of these will be the effort to
restructure the Army‘s terminated Future Combat System program. At the
same time, however, DOD‘s portfolio of major defense acquisition
programs continues to grow. Between December 2007 and July 2009, the
number of major defense acquisition programs grew from 96 to 102
programs. GAO has previously reported that DOD should continue to work
to balance its weapon system portfolio with available funding, which
includes reducing the number or size of weapon system programs, or
both, and assessing the affordability of new programs and capabilities
in the context of overall defense spending.
At the program level, our recent observations present a mixed picture
of DOD‘s adherence to a knowledge-based acquisition approach, which is
a key for improving acquisition outcomes. For 42 programs GAO assessed
in depth in 2010, there has been continued improvement in the
technology, design, and manufacturing knowledge programs had at key
points in the acquisition process. However, most programs are still
proceeding with less knowledge than best practices suggest, putting
them at higher risk for cost growth and schedule delays. A majority of
programs have also experienced requirements changes, software
development challenges, or workforce issues, or a combination, which
can affect program stability and execution. DOD has begun to implement
a revised acquisition policy and congressional reforms that address
many of these areas. For example, eight programs we examined in the
technology development phase plan to test competitive prototypes
before starting system development and seven programs plan to hold
early systems engineering reviews. If DOD consistently applies this
policy, the number of programs adhering to a knowledge-based
acquisition should increase and the outcomes for DOD programs should
improve.
View [hyperlink, http://www.gao.gov/products/GAO-10-706T] or key
components. For more information, contact Michael J. Sullivan at (202)
512-4841 or sullivanm@gao.gov.
[End of section]
Mr. Chairman and Members of the Subcommittee:
I am pleased to be here today to discuss the Department of Defense's
(DOD) management of its acquisition of major weapons systems--an area
that has been a part of GAO's high-risk list since 1990--and the
potential for recent acquisition reforms to improve program outcomes.
While DOD still faces significant challenges in managing its weapon
system programs, the past two years have seen DOD and the Congress
take meaningful steps towards addressing long-standing weapon
acquisition issues. DOD made major revisions to its acquisition
policies to place more emphasis on acquiring knowledge about
requirements, technology, and design before programs start--thus
putting them in a better position to field capabilities on-time and at
the estimated cost. Congress strengthened DOD's acquisition policies
and processes by passing the Weapon Systems Acquisition Reform Act of
2009,[Footnote 1] which includes provisions to ensure programs are
based on realistic cost estimates and to terminate programs that
experience high levels of cost growth. The House Armed Services
Committee Panel on Defense Acquisition Reform issued its final report
and made recommendations on areas, such as assessing the performance
of the defense acquisition system, that were incorporated into the
proposed Implementing Management for Performance and Related Reforms
to Obtain Value in Every Acquisition (IMPROVE) Act of 2010.[Footnote
2] In addition, DOD has started to reprioritize and rebalance its
weapon system investments. In DOD's fiscal year 2010 and 2011 budget
requests, the Secretary of Defense proposed ending all or part of at
least a half dozen major defense acquisition programs that were over
cost, behind schedule, or no longer suited to meet the warfighters'
current needs. Congress's support for several of the recommended
terminations signaled a willingness to make difficult choices on
individual weapon systems and DOD's weapon system investments as a
whole.
While DOD's acquisition policies and process may be headed in the
right direction, fiscal pressures continue to build. Notwithstanding
the federal government's long-term fiscal challenges, the Pentagon
faces its own near-term and long-term fiscal pressures as it attempts
to balance competing demands, including ongoing operations in
Afghanistan and Iraq, initiatives to grow and modernize the force, and
increasing personnel and health care costs. While DOD's fiscal year
2010 budget request started the process of reprioritizing acquisition
dollars to meet warfighters' most pressing needs, the department must
still address the overall affordability of its weapon system
investments. DOD should continue to work to balance its weapon system
portfolio with available funding, which includes reducing the number
or size of weapon system programs, or both, and assessing the
affordability of new programs and capabilities in the context of
overall defense spending.
My statement focuses on the progress DOD has made in improving the
planning and execution of its weapon acquisition programs and the
potential for recent acquisition reforms to improve program outcomes.
It includes observations about (1) DOD's efforts to manage its
portfolio of major defense acquisition programs,[Footnote 3] (2) the
knowledge attained at key junctures of a subset of 42 weapon programs
from the 2009 portfolio, (3) other factors that can affect program
execution, and (4) DOD's implementation of recent acquisition reforms.
The testimony is based on the results of our recently issued annual
assessment of weapon programs.[Footnote 4] To conduct the assessment,
GAO analyzed data on the composition of DOD's portfolio of major
defense acquisition programs. GAO also collected data from program
offices on technology, design, and manufacturing knowledge, as well as
on other factors that can affect program execution. That work was
conducted in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objectives.
We believe that the evidence obtained provides a reasonable basis for
our findings and conclusions based on our audit objectives.
Observations on DOD's 2009 Major Defense Acquisition Program Portfolio:
In our 2010 assessment of weapon programs, we made several
observations concerning DOD's management of its major defense
acquisition portfolio. First, in DOD's fiscal year 2010 budget, the
Secretary of Defense proposed canceling or significantly curtailing
programs with projected total costs of at least $126 billion that he
characterized as too costly or no longer relevant for current
operations, while increasing funding for others that he assessed as
higher priorities. Congress supported several of the recommended
terminations (see table 1).
Table 1: Secretary of Defense's Fiscal Year 2010 Budget
Recommendations:
Recommended termination:
System: VH-71 Presidential Helicopter;
Total estimated cost: $13;
Secretary's comments: Plan to develop options for a new program;
Congressional action: Conferees recommended $100 million for
technology capture that DOD has budgeted for the VH-71 program.
System: Combat Search and Rescue Helicopter;
Total estimated cost (dollars in billions): Unspecified;
Secretary's comments: Plan to reexamine requirements;
Congressional action: Did not authorize appropriations for the program.
System: Next-Generation Bomber;
Total estimated cost: Unspecified;
Secretary's comments: Will not initiate new development program
without better understanding of the requirement and technology;
Congressional action: Supported development of a Next-Generation
Bomber Aircraft, but did not authorize appropriations.
System: Future Combat System-Manned Ground Vehicles;
Total estimated cost: $87 billion;
Secretary's comments: Plan to reevaluate requirements, technology, and
approach before relaunching and recompeting program;
Congressional action: Directed Army to develop, test, and field an
operationally effective and affordable next generation ground combat
vehicle. Conferees recommended rescission of $26 million in existing
funding.
System: Transformational Satellite;
Total estimated cost: $26 billion;
Secretary's comments: Plan to buy two more AEHF satellites as
alternative;
Congressional action: Did not authorize appropriations for the program.
System: Ballistic Missile Defense-Multiple Kill Vehicle;
Total estimated cost: Unspecified;
Secretary's comments: Plan to reexamine requirements;
no mention of new program;
Congressional action: Did not authorize appropriations for the program.
Recommended end of production:
System: C-17;
Total estimated cost: Unspecified;
Secretary's comments: Recommended ending production at 205 aircraft;
Congressional action: Conferees recommended $2.5 billion for the
procurement of 10 C-17 aircraft, associated spares, support equipment,
and training equipment.
System: DDG-1000;
Total estimated cost: Unspecified;
Secretary's comments: Recommended ending production at 3 ships;
Congressional action: Did not fund additional ships. Appropriated $1.4
billion for completion of third DDG-1000.
System: F-22;
Total estimated cost: Unspecified;
Secretary's comments: Recommended ending production at 187 aircraft;
Congressional action: Did not fund additional aircraft. Conferees
recommended rescission of $383 million in existing funding.
Total;
Total estimated cost: $126 billion.
Source: GAO analysis of Secretary's April 2009 statement on fiscal
year 2010 budget and fiscal year 2010 DOD authorization and
appropriations acts.
[End of table]
Second, DOD plans to replace several of the canceled programs in
fiscal years 2010 and 2011, hopefully with new, knowledge-based
acquisition strategies, because the warfighter need remains. The most
significant of these new programs will be the effort to restructure
the Army's Future Combat System program into several smaller,
integrated programs. Third, DOD's portfolio of major defense
acquisition programs grew to 102 programs in 2009--a net increase of 6
since December 2007. Eighteen programs with an estimated cost of over
$72 billion entered the portfolio.[Footnote 5] Not all of these
programs entering the portfolio are new starts. For instance, the
Airborne Signals Intelligence Payload, and the Reaper Unmanned
Aircraft System are two programs that began as acquisition category II
programs,[Footnote 6] but their total research and development or
procurement costs now exceed the threshold for major defense
acquisition programs. Twelve programs with an estimated cost of $48
billion, including over $7 billion in cost growth, left the portfolio.
[Footnote 7] These programs left the portfolio for a variety of
reasons, including program restructure, termination, or completion.
When the Future Combat System is added to the programs leaving the
portfolio, the total cost of these programs increases to $179 billion,
including over $47 billion in cost growth.
Our 2010 assessment did not include an analysis of the cost and
schedule performance of DOD's major defense acquisition program
portfolio as a whole. In recent years, this analysis showed that the
cumulative cost growth on DOD programs had reached $300 billion (in
fiscal year 2010 dollars) and the average delay in delivering initial
capabilities was 22 months. DOD did not issue timely or complete
Selected Acquisition Reports for its major defense acquisition
programs in fiscal year 2009 for the second consecutive presidential
transition, which precluded an analysis of the performance of DOD's
portfolio. We will resume our portfolio analysis in next year's
assessment.
Observations from Our Assessment of Knowledge Attained by Key
Junctures in the Acquisition Process:
At the program level, our recent observations present a mixed picture
of DOD's adherence to a knowledge-based acquisition approach, which is
key for improving acquisition outcomes. In our 2010 assessment of
weapon programs, we assessed the knowledge attained by key junctures
in the acquisition process for 42 individual weapon programs in DOD's
2009 portfolio. While program knowledge is increasing, as in the past,
none of the 42 programs we assessed have attained or are on track to
attain all of the requisite amounts of technology, design, and
production knowledge by each of the key junctures in the acquisition
process.[Footnote 8] However, if DOD consistently implements its
December 2008 policy revisions on new and ongoing programs, then DOD's
performance in these areas, as well as its cost and schedule outcomes,
should improve. Our analysis allows us to make five observations about
DOD's management of technology, design, and manufacturing risks and
its use of testing and early systems engineering to reduce these risks.
* Newer programs are beginning with higher levels of technology
maturity, but they are not taking other steps, such as holding early
systems engineering reviews, to ensure there is a match between
requirements and resources. Achieving a high level of technology
maturity by the start of system development is an important indicator
of whether a match between the warfighter's requirements and the
available resources--knowledge, time, and money--has been
made.[Footnote 9] Since 2006, there has been a significant increase in
the percentage of technologies demonstrated in a relevant or realistic
environment by the start of system development. This increase
coincided with a change in statute. In 2006, the National Defense
Authorization Act included a provision requiring all major defense
acquisition programs seeking milestone B approval--entry into system
development--to get a certification stating the program's technologies
have been demonstrated in a relevant environment.[Footnote 10] While
only one of the six programs that entered system development since
2006 and provided data had fully mature critical technologies--that
is, demonstrated in a realistic environment, according to our
criteria--all the programs had critical technologies that had been at
least demonstrated in a relevant environment. Overall, only 4 of the
29 programs in our assessment that provided data on technical maturity
at development start did so with fully mature critical technologies.
While the technology levels of DOD programs entering system
development have increased, these programs are still not regularly
conducting early systems engineering reviews, which help ensure there
is a match between requirements and resources. We have previously
reported that before starting development, programs should hold
systems engineering events, such as the preliminary design review, to
ensure that requirements are defined and feasible and that the
proposed design can meet those requirements within cost, schedule, and
other system constraints. We have also found that programs conducting
these events prior to development start experienced less research and
development cost growth and shorter delays in the delivery of initial
operational capabilities than programs that conducted these reviews
after development start.[Footnote 11] Almost all nonship programs (37
of 40 that provided data) in our latest assessment have held at least
one of three key systems engineering reviews (system requirements
review, system functional review, and preliminary design review).
However, only 1 of 37 programs that held a preliminary design review
did so before the start of system development. The remaining programs
held the review, on average, 30 months after development start. The
Weapon Systems Acquisition Reform Act of 2009 established a statutory
requirement for programs to conduct a preliminary design review before
milestone B, so we expect improvements in this area.[Footnote 12]
* Programs that have held critical design reviews in recent years
reported higher levels of design knowledge; however, few programs are
demonstrating that the design is capable of meeting performance
requirements by testing an integrated prototype. Knowing a product's
design is stable before system demonstration reduces the risk of
costly design changes occurring during the manufacturing of
production--representative prototypes--when investments in
acquisitions become more significant. The overall design knowledge
that programs have demonstrated at their critical design reviews has
increased since 2003. Programs in our assessment that held a critical
design review between 2006 and 2009 had, on average, almost 70 percent
of their design drawings releasable at the time of the review, which
is a consistent upward trend since 2003. However, most designs are
still not stable at this point. Of the 28 programs in our latest
assessment that have held a system-level critical design review, only
8 reported having a stable design. Only 2 of the 5 programs that held
a critical design review in 2009 had a stable design at that point.
The 5 programs reported that, on average, 83 percent of the total
expected drawings were releasable.
While the design knowledge of DOD programs at the system-level
critical design review has increased since 2003, these programs are
still not regularly demonstrating that these designs can meet
performance requirements by testing integrated prototypes before the
critical design review--a best practice. None of the 5 programs in our
latest assessment that held their critical design review in 2009 and
planned to test a prototype did so before the review. Of the 33
programs that reported that they either had tested or were going to
test an early system prototype and provided a critical design review
date, only 4 did so before their critical design review.[Footnote 13]
The Weapon Systems Acquisition Reform Act of 2009 requires that DOD
policy ensure that the acquisition strategy for each major defense
acquisition program provides for competitive prototypes before
milestone B approval, unless a waiver is properly granted.[Footnote
14] This requirement should increase the percentage of programs
demonstrating that the system's design works as intended before the
critical design review.
* Some programs are taking steps to bring critical manufacturing
processes into control, however many programs still rely on "after the
fact" metrics. Capturing critical manufacturing knowledge before
entering production helps ensure that a weapon system will work as
intended and can be manufactured efficiently to meet cost, schedule,
and quality targets. Identifying key product characteristics and the
associated critical manufacturing processes is a key initial step to
ensuring production elements are stable and in control. Seven programs
in our latest assessment have identified their critical manufacturing
processes, including four of the programs that entered production in
2009. Three of those seven programs reported that their critical
manufacturing processes were in control.[Footnote 15] It is generally
less costly--in terms of time and money--to eliminate product
variation by controlling manufacturing processes than to perform
extensive inspection after a product is built. However, many DOD
programs rely on inspecting produced components instead of using
statistical process control data in order to assess the maturity of
their production processes. For example, 12 programs in our assessment
reported tracking defects in delivered units, nonconformances, or
scrap/rework as a way to measure production process maturity. The use
of "after the fact" metrics is a reactive approach towards managing
manufacturing quality as opposed to a prevention-based approach.
* Programs are still not regularly testing production representative
prototypes before committing to production. We have previously
reported that in addition to demonstrating that the system can be
built efficiently, production and postproduction costs are minimized
when a fully integrated, capable prototype is demonstrated to show
that the system will work as intended and in a reliable manner. The
benefits of testing are maximized when the tests are completed prior
to a production decision because making design changes after
production begins can be both costly and inefficient. However, of the
32 programs in our assessment that could have tested a prototype
before production, only 17 either tested or expect to test a fully
configured, integrated, production-representative prototype before
holding their production decision. In December 2008, DOD changed its
policy to require programs to test production-representative articles
before entering production.
* More programs are using reliability growth curves before beginning
production. Reliability growth testing provides visibility over how
reliability is improving and uncovers design problems so fixes can be
incorporated before production begins. According to DOD's acquisition
policy, a major defense acquisition program may not proceed beyond low-
rate initial production until it has demonstrated acceptable
reliability. Over half--22 of 40 programs that responded to our
questionnaire--reported that they use a reliability growth curve, with
18 of these programs reporting they are currently meeting their
established goals. In addition, 12 of 19 programs that expect to hold
their production decision in 2010 and beyond reported using
reliability growth curves and most stated they are currently meeting
their goals. This practice should help these programs begin production
with a reliable product design.
Observations on Other Factors That Can Affect Program Execution:
Our 2010 assessment of weapon programs also included three
observations on other areas related to DOD's management of its weapons
programs, including requirements, software management, and program
office staffing. We have previously identified requirements changes
and increases in software lines of code as sources of program
instability that can contribute to cost growth and schedule delays. We
have also reported that workforce challenges can hinder program
execution and negatively affect program management and oversight.
* A majority of programs changed key systems requirements after
development start. Of the 42 programs in our 2010 assessment that
reported tracking requirements changes, 23 programs reported having
had at least one change (addition, reduction, enhancement, or
deferment) to a key performance parameter--a top-level requirement--
since development start. Further, nine programs experienced at least
one change to a key system attribute--a lower level, but still a
crucial requirement of the system. Eight programs reported major
effects on the program as a result of these requirements changes, such
as not meeting acquisition program baseline cost, schedule, and
performance thresholds. DOD's revised December 2008 acquisition policy
attempts to reduce potentially disruptive requirements changes by
requiring programs to hold annual configuration steering board
meetings to ensure that significant technical changes are not approved
without considering their effect on cost and schedule.
* Many programs are at risk for cost growth and schedule delays
because of software development issues. Seventeen of the 28 programs
in our 2010 assessment that reported data on software lines of code
estimated that the number of lines of code required for the system to
function has grown or will grow by 25 percent or more--a predictor of
future cost and schedule growth. Overall, the average growth or
expected growth in lines of code for the 28 programs was about 92
percent. In addition to measuring growth in software lines of code, we
have previously reported that collecting earned value management data
for software development and tracking and containing software defects
in phase are good management practices. Overall, 30 programs in our
assessment reported collecting earned value management data to help
manage software development. Thirty-two programs in our latest
assessment also reported collecting some type of software defect data.
For the 22 programs that responded a more specific question about
defect correction, on average, only 69 percent of the defects were
corrected in the phase of software development in which they occurred.
Capturing software defects in phase is important because discovering
defects out of phase can cause expensive rework later in programs.
* Programs' reliance on nongovernment personnel continues to increase
in order to make up for shortfalls in government personnel and
capabilities. In recent years, Congress and DOD have taken steps to
ensure the acquisition workforce has the capacity, personnel, and
skills needed to properly perform its mission; however, programs
continue to struggle to fill all staff positions authorized. Only 19
of the 50 programs in our 2010 assessment that responded to our
questions on staffing were able to fill all the positions they had
been authorized. A commonly cited reason for not being able to fill
positions was difficulty finding qualified candidates. As a result of
staff shortfalls, program offices reported that program management and
oversight has been degraded, contracting activities have been delayed,
and program management costs have increased as contractors are used to
fill the gap. Overall, 43 programs or 86 percent of those providing
data reported utilizing support contractors to make up for shortfalls
in government personnel and capabilities.
In addition, for the first time since we began reporting on program
office staffing in 2008, programs reported having more nongovernment
than government staff working in program offices (see table 2). The
greatest numbers of support contractors are in engineering and
technical positions, but their participation has increased in all
areas, from program management and contracting to administrative
support and other business functions.
Table 2: Program Office Composition for 50 DOD Programs:
Percentage of staff:
Military:
Program management: 28%;
Engineering and technical: 7%;
Contracting: 6%;
Other business functions: 3%;
Administrative support: 2%;
Other: 5%;
Total: 8%.
Civilian government:
Program management: 40%;
Engineering and technical: 41%;
Contracting: 74%;
Other business functions: 45%;
Administrative support: 18%;
Other: 24%;
Total: 40%.
Total government:
Program management: 67%;
Engineering and technical: 47%;
Contracting: 80%;
Other business functions: 48%;
Administrative support: 20%;
Other: 29%;
Total: 49%.
Support contractors:
Program management: 32%;
Engineering and technical: 43%;
Contracting: 20%;
Other business functions: 50%;
Administrative support: 78%;
Other: 70%;
Total: 45%.
Other nongovernment[A]:
Program management: 0%;
Engineering and technical: 9%;
Contracting: 0%;
Other business functions: 3%;
Administrative support: 2%;
Other: 1%;
Total: 6%.
Total nongovernment:
Program management: 33%;
Engineering and technical: 53%;
Contracting: 20%;
Other business functions: 52%;
Administrative support: 80%;
Other: 71%;
Total: 51%.
Source: GAO analysis of DOD data.
Notes: Totals may not add due to rounding.
[A] Other nongovernment includes federally funded research and
development centers, universities, and affiliates.
[End of table]
Observations about DOD's Implementation of Acquisition Reforms:
DOD has begun to incorporate acquisition reforms into the acquisition
strategies for new programs. Both DOD's December 2008 acquisition
policy revisions and the Weapon Systems Acquisition Reform Act of 2009
require programs to invest more time and resources in the front end of
the acquisition process--refining concepts through early systems
engineering, developing technologies, and building prototypes before
starting system development. In addition, DOD policy requires
establishment of configuration steering boards that meet annually to
review all program requirements changes as well as to make
recommendations on proposed descoping options that could help keep a
program within its established cost and schedule targets. These steps
could provide a foundation for establishing sound, knowledge-based
business cases for individual weapon programs and are consistent with
many of our past recommendations; however, if reform is to succeed and
weapon program outcomes are to improve, they must continue to be
reinforced in practice through decisions on individual programs.
Our analysis of the programs in our 2010 assessment allowed us to make
two observations about the extent to which DOD is implementing recent
acquisition reforms:
* Most of the ten programs in our 2010 assessment that had not yet
entered system development reported having acquisitions strategies
consistent with both DOD's revised acquisition policy and the
provisions of the Weapon Systems Acquisition Reform Act of 2009.
Specifically, 8 programs in our assessment planned to develop
competitive prototypes before milestone B.[Footnote 16] In addition, 7
programs have already scheduled a preliminary design review before
milestone B.[Footnote 17]
* Only a few programs reported holding configuration steering boards
to review requirements changes, significant technical changes, or de-
scoping options in 2009. Seven programs in our assessment reported
holding configuration steering boards in 2009. Under DOD's revised
acquisition policy, ongoing acquisition category I and IA programs in
development are required to conduct annual configuration steering
boards to review requirements changes and significant technical
configuration changes that have the potential to result in cost and
schedule effects on the program. In addition, the program manager is
expected to present de-scoping options to the board that could reduce
program costs or moderate requirements. None of the programs reported
that the boards that were held approved requirements changes or
significant technical changes. One program--the P-8A Poseidon--
reported that it presented de-scoping options to decrease cost and
schedule risk on the program and had those options approved.
Concluding Observations on the Challenges to Achieving Lasting Reform:
I would like to offer a few thoughts about other factors that should
be considered so that we make the most out of today's opportunity for
meaningful change. First, I think it is useful to think of the
processes that affect weapon system outcomes (requirements, funding,
and acquisition) as being in a state of equilibrium. Poor outcomes--
delays, cost growth, and reduced quantities--have been persistent for
decades. If we think of these processes as merely "broken", then some
targeted repairs should fix them. I think the challenge is greater
than that. If we think of these processes as being in equilibrium,
where their inefficiencies are implicitly accepted as the cost of
doing business, then the challenge for getting better outcomes is
greater. Seen in this light, it will take considerable and sustained
effort to change the incentives and inertia that reinforce the status
quo.
Second, while actions taken and proposed by DOD and Congress are
constructive and will serve to improve acquisition outcomes, one has
to ask the question why extraordinary actions are needed to force
practices that should occur normally. The answer to this question will
shed light on the cultural or environmental forces that operate
against sound management practices. For reforms to work, they will
have to address these forces as well. For example, there have been a
number of changes to make cost estimates more rigorous and realistic,
but do these address all of the reasons why estimates are not already
realistic? Clearly, more independence, methodological rigor, and
better information about risk areas like technology will make
estimates more realistic. On the other hand, realism is compromised as
the competition for funding encourages programs to appear affordable.
Also, when program sponsors present a program as more than a weapon
system, but rather as essential to new fighting concepts, pressures
exist to accept less than rigorous cost estimates. Reform must
recognize and counteract these pressures as well.
Third, decisions on individual systems must reinforce good practices.
Programs that have pursued risky and unexecutable acquisition
strategies have succeeded in winning approval and funding. If reform
is to succeed, then programs that present realistic strategies and
resource estimates must succeed in winning approval and funding. Those
programs that continue past practices of pushing unexecutable
strategies must be denied funding before they begin. This will require
sustained leadership from the Secretary of Defense, the Under
Secretary of Defense for Acquisition, Technology and Logistics, and
the military services, and the cooperation and support of Congress.
Fourth, consideration should be given to setting some limits on what
is a reasonable length of time for developing a system. For example,
if a program has to complete development within 5 or 6 years, this
could serve as a basis to constrain requirements and exotic programs.
It would also serve to get capability in the hands of the warfighter
sooner.
Fifth, the institutional resources we have must match the outcomes we
desire. For example, if more work must be done to reduce technical
risk before development start--milestone B--DOD needs to have the
organizational, people, and financial resources to do so. Once a
program is approved for development, program offices and testing
organizations must have the workforce with the requisite skills to
manage and oversee the effort. Contracting instruments must be used
that match the needs of the acquisition and protect the government's
interests. Finally, DOD must be judicious and consistent in how it
relies on contractors.
Mr. Chairman, this completes my prepared statement. I would be happy
to respond to any questions you or other Members of the Subcommittee
may have at this time.
Contacts and Acknowledgments:
For further information about this statement, please contact Michael
J. Sullivan (202) 512-4841 or sullivanm@gao.gov. Contact points for
our Offices of Congressional Relations and Public Affairs may be found
on the last page of this statement. Individuals who made key
contributions to this statement include Ronald E. Schwenn, Assistant
Director, Kristine R. Hassinger, Carol T. Mebane, and Kenneth E.
Patton.
[End of section]
Footnotes:
[1] Pub. L. No. 111-23.
[2] H.R. 5013, 111th Cong. (as received from the House and referred to
the S. Comm. on Armed Serv., Apr. 29, 2010).
[3] Major defense acquisition programs (MDAP) are those identified by
DOD that require eventual total research, development, test and
evaluation (RDT&E), including all planned increments, expenditures of
more than $365 million or procurement expenditures, including all
planned increments , of more than $2.19 billion in fiscal year 2000
constant dollars.
[4] GAO, Defense Acquisitions: Assessments of Selected Weapon
Programs, [hyperlink, http://www.gao.gov/products/GAO-10-388SP],
Washington, D.C.: March 30, 2010.
[5] Cost data was only available for 13 of the 18 newly designated
major defense acquisition programs.
[6] An acquisition category II program is defined as a program that
does not meet the criteria for an acquisition category I program and
is estimated to require eventual total RDT&E expenditures of more than
$140 million or procurement expenditures of more than $660 million in
fiscal year 2000 constant dollars.
[7] The estimated cost for these 12 programs is based on DOD's
December 2007 Selected Acquisition Reports. Cost growth was calculated
from the programs' first cost estimate.
[8] Not all programs provided information for every knowledge point or
had reached all of the knowledge points--development start, design
review, and production start.
[9] The start of system development, as used here, indicates the point
at which significant financial commitment is made to design,
integrate, and demonstrate that the product will meet the user's
requirements and can be manufactured on time, with high quality, and
at a cost that provides an acceptable return on investment. Under the
revised Department of Defense Instruction 5000.02, Operation of the
Defense Acquisition System (Dec. 8, 2008), system development is now
called engineering and manufacturing development. Engineering and
manufacturing development follows materiel solution analysis and
technology development. For shipbuilding programs, this point occurs
when a program awards a detailed design and construction contract.
[10] A major defense acquisition program may not receive milestone B
approval until the milestone decision authority certifies that the
technology in the program has been demonstrated in a relevant
environment. National Defense Authorization Act for Fiscal Year 2006,
Pub. L. No. 109-163, § 801 (codified at 10 U.S.C. § 2366b (a)(3)(D)).
[11] GAO, Defense Acquisitions: Assessments of Selected Weapon
Programs, [hyperlink, http://www.gao.gov/products/GAO-09-326SP],
Washington, D.C.: March 30, 2009.
[12] Under the Weapon Systems Acquisition Reform Act of 2009, a major
defense acquisition program may not receive milestone B approval until
the program has held a preliminary design review and the milestone
decision authority has conducted a formal post-preliminary design
review assessment and certified on the basis of such assessment that
the program demonstrates a high likelihood of accomplishing its
intended mission. Pub. L. No. 111-23, § 205(a)(3) (codified as amended
at 10 U.S.C. § 2366b(a)(2)).
[13] One program that held a critical design review in 2009 did not
plan to test an early systems prototype.
[14] Pub. L. No. 111-23, § 203.
[15] DOD policy states that the knowledge required for a major defense
acquisition program to proceed beyond low-rate initial production
shall include demonstrated control of the manufacturing process and
acceptable reliability, the collection of statistical process control
data, and demonstrated control and capability of critical processes.
Department of Defense Instruction 5000.02, Operation of the Defense
Acquisition System, enclosure 2, paragraph 7.c.(2) (Dec. 8, 2008). We
did not specifically assess compliance with this requirement.
[16] The Weapon Systems Acquisition Reform Act of 2009 requires that
DOD policy ensure that the acquisition strategy for each major defense
acquisition program provides for competitive prototypes before
milestone B approval, unless a waiver is properly granted. Pub. L. No.
111-23, § 203(a).
[17] The Weapon Systems Acquisition Reform Act of 2009 establishes a
statutory requirement that a major defense acquisition program may not
receive milestone B approval until the milestone decision authority
has received a preliminary design review, conducted a formal post-
preliminary design review assessment, and certified on the basis of
such assessment that the program demonstrates a high likelihood of
accomplishing its intended mission. Pub. L. No. 111-23, § 205(a)(3)
(codified as amended at 10 U.S.C. § 2366b(a)(2)).
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
