Defense Acquisitions
Risks Posed by DOD's New Space Systems Acquisition Policy Gao ID: GAO-04-379R January 29, 2004On November 18, 2003, we testified before Congress on the Department of Defense's (DOD) new acquisition policy for space systems. The new acquisition policy, issued in October 2003, sets the stage for decision making for DOD's investment in space systems, which currently stands at more than $18 billion annually and is expected to grow considerably over the next decade. Congress requested that we provide additional comments on several issues relating to the new policy and other space acquisition issues.
Our concern with the Defense Space Acquisition Board is not with earlier identification of problems or the added senior level attention the new process calls for, but with earlier investment decisions, which are also called for. Under the new process, the DSAB may approve product development to begin before DOD knows whether technologies can work as intended. As a result, it will make major investment commitments without really knowing what resources will be required to deliver promised capability. We do not believe the new space acquisition policy will provide better cost estimates. Although some process changes will be made, the underlying causes of underestimating costs remain. DOD's acquisition policy for other weapon systems requires a commitment for full funding at milestone B--the start of product development and the point at which DOD should have knowledge that technologies can work as intended. Competition can provide natural incentives for an organization to be more efficient and more innovative. These incentives work in DOD's favor. However, it is also important to recognize that competition can take various forms. For example, DOD can increase competition by using shadow contractors, pursuing alternative sensor designs, and breaking acquisitions into smaller blocks. DOD can also optimize its investment in weapon systems by competing air, land, sea, and space-based capabilities. While there are only a few contractors currently capable of implementing large and complex space programs, there are many more capable of building specific satellite components and technologies. Thus, by increasing competition at the mission payload or sensor level and breaking acquisitions into smaller pieces, DOD can expand the universe of contractors competing for work. Over the long run, this could enable more contractors to build the expertise and knowledge needed to manage large space programs. It would also require DOD to have significant insight into the lower tiers of the industry. Managing the industrial base is one of the most critical determinants of acquisition success. According to DOD studies, this not only means injecting competition early on to ensure that the highest performing and most cost-effective technologies and designs are being pursued, but adequately defining work; establishing shorter, more manageable contract periods; and providing the right incentives for contractors. GAO does not believe that space programs will be less schedule driven under the new policy. In the past, DOD has taken a schedule-driven versus a knowledge-driven approach to the acquisition process for space and other weapons systems with the justification that capabilities were urgently needed. Prior to the restructuring, the SBIRS High program office exerted no control over requirements changes, leaving many decisions on requirements to its contractors or within lower management levels of the program office. As part of the SBIRS High program restructuring, the Air Force established an advisory program management board to oversee requirements changes. The board's role is to ensure that new requirements are urgent and compelling, that they reflect an appropriate use of funds, and that decisions about requirements are more transparent. Air Force leadership, not the Space-Based Infrared System High (SBIRS) program office, made the decision that the new requirements were urgent and compelling enough to address. Problems with software development in DOD weapons systems are well known. The fundamental problem with the SBIRS High program has been the failure to develop key knowledge at critical junctures early in the development of the system, that is, before major investments were made. The program is now paying the price for this lack of knowledge development. We recently reported that the majority of satellite programs over the past couple decades, like SBIRS High, cost more than expected and took longer to develop than planned.
GAO-04-379R, Defense Acquisitions: Risks Posed by DOD's New Space Systems Acquisition Policy
This is the accessible text file for GAO report number GAO-04-379R
entitled 'Defense Acquisitions: Risks Posed by DOD's New Space Systems
Acquisition Policy' which was released on January 29, 2004.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
January 29, 2004:
The Honorable Wayne Allard:
Chairman:
The Honorable Bill Nelson:
Ranking Minority Member:
Subcommittee on Strategic Forces:
Committee on Armed Services:
United States Senate:
Subject: Defense Acquisitions: Risks Posed by DOD's New Space Systems
Acquisition Policy:
On November 18, 2003, we testified before the Subcommittee on the
Department of Defense's (DOD) new acquisition policy for space systems.
The new acquisition policy, issued in October 2003, sets the stage for
decision making for DOD's investment in space systems, which currently
stands at more than $18 billion annually and is expected to grow
considerably over the next decade. You requested that we provide
additional comments on several issues relating to the new policy and
other space acquisition issues. Your specific questions and our answers
are discussed below.
1. The Air Force maintains that its Defense Space Acquisition Board
(DSAB) process allows earlier identification of problems and senior
level attention, which will improve management and lower risk. Does
GAO have any concerns with the DSAB process?
Our concern is not with earlier identification of problems or the added
senior level attention the new process calls for, but with earlier
investment decisions, which are also called for. Under the new process,
the DSAB may approve product development to begin before DOD knows
whether technologies can work as intended. As a result, it will make
major investment commitments without really knowing what resources will
be required to deliver promised capability. Moreover, the policy
encourages development of leading-edge technology within product
development, that is, at the same time the program manager is designing
the system and undertaking other product development activities. DOD
believes this approach will allow space systems to better incorporate
leading-edge technologies. But as our work has repeatedly shown, such
concurrency within space and other weapon system programs increases the
risk that significant problems will be discovered as the system is
integrated and built, when it is more costly and time-consuming to fix
them.
Moreover, as we testified, the knowledge-building approach for space
stands in sharp contrast to that followed by successful programs and
the approach recommended by DOD's revised acquisition policy for weapon
systems. Successful programs will not commit to undertaking product
development unless they have high confidence that they have achieved a
match between what the customer wants and what the program can deliver.
Technologies that are not mature continue to be developed in an
environment that is focused solely on technology development. This
system puts programs in a better position to succeed because they can
focus on design, system integration, and manufacturing. Further, our
work has shown that taking an evolutionary approach to improving
capability increases the likelihood of delivering that capability to
the war fighter sooner than the revolutionary approach the Air Force
continues to support in the new space policy.
2. Does GAO believe that the process put into place in the new space
acquisition policy by which cost estimates are derived will provide
better cost estimates?
No. Although some process changes will be made, the underlying causes
of underestimating costs remain.
DOD is adopting new methodologies and tools to enhance cost estimates,
and it is enlisting assistance from DOD's Cost Analysis Improvement
Group (CAIG) to conduct independent cost estimates using cost
estimating teams drawn from a broad spectrum of the cost-estimating
community. Moreover, programs are now required to resolve differences
between their cost estimates and estimates produced by the independent
teams. In the past, cost-estimating groups have developed estimates
that were different, leaving decision makers to select one estimate or
combine a few.
However, under the new space acquisition policy, cost estimates do not
have to be based on the knowledge that technologies can work as
intended. History has shown that cost estimates not based on such
knowledge are significantly understated. Moreover, incentives that work
against providing good estimates have not changed. Unlike the
commercial world where the focus is on delivering a product to market,
DOD's system focuses on competing for resources from oversubscribed
budgets. In the competition for funding, managers are encouraged to
launch product developments before technologies are mature. Because
funding is competitive and DOD's forecasts of costs, schedules, and
performance are largely based on immature technologies and other
unknowns, estimates tend to be squeezed into insufficient profiles of
available funding. In fact, pressures to underestimate costs may
increase over the next decade as DOD plans to undertake a number of
new, challenging space programs--which are expected to require an
additional $4 billion in the next 4 years alone. Costs beyond that
period are as yet unknown but are likely to be considerably higher.
3. What is GAO's view on the Air Force policy related to full funding?
DOD's acquisition policy for other weapon systems requires a commitment
for full funding at milestone B--the start of product development and
the point at which DOD should have knowledge that technologies can work
as intended. However, the new space acquisition policy does not require
DOD to commit to fully fund a space program either when this knowledge
has been obtained or at any point in the development process. Hence,
there is no guarantee that the resources needed to meet requirements on
any individual program will be there when needed--particularly as DOD
moves forward with its new programs.
This represents another important departure from the development
approach followed by successful programs. Our prior work[Footnote 1]
has found that if a product's business case measures up, that is a
company is assured that there is a market or need for the product and
that it has the right knowledge in hand to develop the product with
firm cost and schedule estimates, the company then commits to the
entire development of the product, including the financial investment.
In other words, corporate resources are made available to the
development team so that product success is not compromised. As noted
earlier, because DOD begins too many programs, its resources are always
oversubscribed. By requiring program managers to continually justify
funding, DOD runs a risk of foreclosing the ability for sound planning
and execution.
4. The Young Panel was not convinced of the merits of competition in
some circumstances, particularly when the incumbent has performed well
and "owns" the expertise and the government would incur significant
cost in choosing another contractor for follow-on systems. Does GAO
have a view on the merits or demerits of competition in space programs?
Competition can provide natural incentives for an organization to be
more efficient and more innovative. These incentives work in DOD's
favor. However, it is also important to recognize that competition can
take various forms. For example, DOD can increase competition by using
shadow contractors, pursuing alternative sensor designs, and breaking
acquisitions into smaller blocks. DOD can also optimize its investment
in weapon systems by competing air, land, sea, and space-based
capabilities. By pursuing these various options, DOD would have greater
assurance that it is obtaining the best value when it must select a
prime contractor for follow-on systems.
5. How effective can competition be when we have so few major
contractors capable of executing large and complex space programs?
While there are only a few contractors currently capable of
implementing large and complex space programs, there are many more
capable of building specific satellite components and technologies.
Thus, by increasing competition at the mission payload or sensor level
and breaking acquisitions into smaller pieces, DOD can expand the
universe of contractors competing for work. Over the long run, this
could enable more contractors to build the expertise and knowledge
needed to manage large space programs. It would also require DOD to
have significant insight into the lower tiers of the industry.
6. Is there a path to making competition a useful element in healthy
programs?
Managing the industrial base is one of the most critical determinants
of acquisition success. According to DOD studies, this not only means
injecting competition early on to ensure that the highest performing
and most cost-effective technologies and designs are being pursued, but
adequately defining work; establishing shorter, more manageable
contract periods; and providing the right incentives for contractors.
Following an evolutionary development path would better enable programs
to take these kinds of actions. It would also foster a healthier
industrial base because it would get programs into production sooner.
Also important is ensuring that programs have the right capability to
evaluate contractor proposals and to manage the contracts once they are
in place. As DOD's studies of space programs show, the government will
invariably encounter problems when too much responsibility is handed
over to contractors and too little oversight is provided.
We have also found that the path to healthier programs is characterized
by having an open systems design. Such a design is characterized by
(1) well defined, widely used, preferably nonproprietary interfaces and
protocols between systems, subsystems, and components and (2) an
explicit provision for system expansion or upgrade through
incorporation of additional higher performance subsystems and
components with minimal negative impact on the existing system. Open
systems design allows competing developers to offer additional features
and capabilities. With this approach, the government might be able to
minimize dependence on a specific contractor. Also, upgrades can be
added without replacing the entire system. Costs across the board--
development, production, operations, and support--can thereby be
reduced.
7. Does GAO believe that space programs will be less schedule driven
under the new acquisition policy?
No. In the past, DOD has taken a schedule-driven versus a knowledge-
driven approach to the acquisition process for space and other weapons
systems with the justification that capabilities were urgently needed.
In other words, commitments were made to achieving certain capabilities
without knowing whether technologies being pursued could really work as
intended. As a result, time and costs estimates were consistently
exceeded, and steps essential to containing costs, maximizing
competition among contractors, and testing technologies were
shortchanged. Perversely, programs actually took longer when rushed at
the start. Moreover, DOD often lacked assurance that it was even
pursuing the best technical solution because alternatives were not
analyzed or they were eliminated in order to meet schedule pressures.
When technology did not perform as planned, assigning additional
resources in terms of time and money became the primary option for
solving problems, since customer expectations about the products'
performance already became hardened.
The new space acquisition process does not change this approach or the
incentives that drive it. Rather, it encourages programs to enter into
product development without knowledge that technologies can work as
intended. Moreover, for new programs like the Transformational
Satellite (TSAT) and Space Based Radar (SBR), DOD is still setting
initial satellite launch dates before this knowledge has been obtained.
By contrast, DOD's acquisition policy for non-space systems establishes
mature technologies--that is, technologies demonstrated in a relevant
environment--as critical before entering product development. By
encouraging programs to do so, the policy for non-space systems puts
programs in a better position to deliver capability to the war fighter
in a timely fashion and within funding estimates because program
managers can focus on the design, system integration, and manufacturing
tasks needed to produce a product.
8. The requirements for the Space-Based Infrared System High system
(SBIRS High) still continue to change. In GAO's report, you highlight
several examples, including batteries and solar cell panels. From the
report, I gather that GAO finds that the Air Force's efforts to limit
requirements changes to only those that are "urgent and compelling" are
better, but that they are not successfully eliminating the growth of
requirements. You mention at least $203 million in new requirements.
Is this a correct interpretation of the new "urgent and compelling
approach?":
Prior to the restructuring, the SBIRS High program office exerted no
control over requirements changes, leaving many decisions on
requirements to its contractors or within lower management levels of
the program office. As part of the SBIRS High program restructuring,
the Air Force established an advisory program management board to
oversee requirements changes. The board's role is to ensure that new
requirements are urgent and compelling, that they reflect an
appropriate use of funds, and that decisions about requirements are
more transparent. Air Force leadership, not the SBIRS High program
office, made the decision that the new requirements were urgent and
compelling enough to address.
We believe that establishing the board is a positive step and should
help manage requirements changes more effectively. Nevertheless, the
board will still be challenged to ensure some discipline in
requirements setting, since there is a diverse group of Air Force and
other DOD users that have an interest in SBIRS High and there are
increasing demands for surveillance capabilities. Currently, there are
several proposed requirements changes on the table that could have a
significant impact on the program.
9. The GAO report also indicates that software development problems
continue to be a problem. This problem is not limited to SBIRS High,
however. What recommendations can you make to address this continuing
problem?
Problems with software development in DOD weapons systems are well
known. For example, the Defense Science Board reviewed selected DOD
software intensive systems and found that programs lacked well thought-
out, disciplined program management and/or software development
processes. The programs lacked meaningful cost, schedule, and
requirements baselines, making it difficult to track progress against
them. These findings are echoed by the work of DOD's Tri-Service
initiative. Because weapon systems are becoming increasing dependent on
software, lax management and oversight over software development can be
detrimental to a program, as it was for SBIRS High.
There are steps we have identified in an ongoing review for the Senate
Committee on Armed Services that DOD could take to address this
problem. Chief among them is to require programs to apply best
practices for software development and acquisition, many of which have
been identified by the Software Engineering Institute at Carnegie
Mellon University and packaged into continuous improvement models and
guidance. In adopting these models, organizations would take a more
disciplined and rigorous approach toward managing or overseeing
software development. At the same time, organizations need to provide
the right environment to reduce software development risk. This means
establishing an environment comprised of an evolutionary software
development approach that relies on well-understood, manageable
requirements and a desire to continuously improve development
processes. It also means adopting and using a host of metrics to track
cost and scheduling deviations; requirements changes and their impact
on software development efforts; testing efforts; as well as efforts to
detect and fix defects. Also important is to integrate these practices
into existing acquisition policies and improvement plans as well as to
enforce the use of these practices within individual programs.
10. The GAO conclusion is that SBIRS High is still a program in
trouble. To remedy this problem, GAO recommends that the Secretary
reconvene the independent review team, or a similar body, to provide
an assessment of the restructured program and concrete guidance for
addressing the program's underlying problems. To play devil's advocate
for a moment, how will another review of this program improve its
chances of technical, budget, and schedule success?
The fundamental problem with the SBIRS High program has been the
failure to develop key knowledge at critical junctures early in the
development of the system, that is, before major investments were made.
The program is now paying the price for this lack of knowledge
development. Although the restructuring of the program in 2002 improved
management and oversight capabilities, it did not go far enough in
addressing the underlying problems with system design, integration, and
software development. Another independent and in-depth technical review
of the program is important to ensure that these problems are more
clearly understood and that there are no other hidden problems lurking.
At the same time, such a review will keep attention focused and
heighten oversight of the program. Moreover, until it becomes standard
to make knowledge-based decisions on DOD programs, ad hoc reviews such
as the one we call for may be the only way to bring transparency to the
decision making process.
11. SBIRS High is clearly a highly visible troubled program. How
representative is it of space programs in general? Is it unique or are
the problems identified present throughout the space acquisition
effort?
We recently reported[Footnote 2] that the majority of satellite
programs over the past couple decades, like SBIRS High, cost more than
expected and took longer to develop than planned. SBIRS High is one of
the few weapon systems programs to exceed the 25 percent cost threshold
established in 10 U.S.C. 2433, but the problems affecting other
programs have been equally dramatic. For example, cost estimates for
the Advanced Extremely High Frequency (AEHF) communications satellite
program grew by $1.2 billion from 1999 through 2001, while the program
experienced a 2-year delay in the launch of the first satellite. And
while DOD has spent several billion dollars over the past 2 decades to
develop low-orbiting satellites that can track ballistic missiles
throughout their flight, it has not launched a single satellite to
perform this capability.
A key underlying problem with many programs has been the desire to
achieve revolutionary advancements in capability instead of
evolutionary advancements. Such an approach meant that requirements
exceeded resources (time, money, and technology) at the time of product
development, setting the stage for costly and time-consuming rework
later in the program. More specifically, in reviewing our past reports,
we found that: (1) requirements for what the satellite needed to do and
how well it must perform were not adequately defined at the beginning
of a program or were changed significantly once the program had already
begun; (2) investment practices were weak, e.g., cost estimates were
optimistic or potentially more cost-effective approaches were not
examined; (3) acquisition strategies were poorly executed, e.g.,
competition was reduced for the sake of schedule or DOD did not
adequately oversee contractors; and (4) technologies were not mature
enough to be included in product development. All of these problems
affected SBIRS High and AEHF. One or more affected the Space Tracking
and Surveillance System (STSS) and the predecessor SBIRS programs as
well as Milstar, the Global Positioning System (GPS), and the National
Polar-orbiting Operational Environmental Satellite System (NPOESS).
Because DOD took a schedule-driven approach instead of a knowledge-
driven approach to the acquisition process, activities essential to
containing costs, maximizing competition among contractors, and testing
technologies were compressed or not done. Like SBIRS High, many
programs also encountered problems in setting requirements due to the
diverse array of organizations with competing interests involved in
overall satellite development--from the individual military services,
to testing organizations, contractors, civilian agencies, and in some
cases international partners. Requirements setting for SBIRS High was
particularly problematic because the government put too much
responsibility on its contractors to balance these competing interests-
-a problem recognized in DOD's own study of SBIRS High and other
studies of space acquisition problems.
In our view, new programs like the Transformational Satellite (TSAT)
will likewise be unable to make a match between needs and resources at
the onset of product development because DOD's new space acquisition
policy encourages product development to begin without knowing that
technologies can work as intended to meet capability needs.
In preparing answers to your questions, we relied on our prior work on
DOD's new space acquisition policy,[Footnote 3] best practices in
weapon system acquisitions, and our reviews of specific space
acquisitions. Because we relied on previously issued work, we did not
obtain comments from DOD on a draft of this letter. We conducted our
work from December 2003 through January 2004 in accordance with
generally accepted government auditing standards.
We are sending copies of this letter to the Secretaries of Defense and
the Air Force and interested congressional committees. We will also
make copies available to others upon request. In addition, the report
will be available at no charge on the GAO Web site at http://
www.gao.gov.
If you or your staff have any questions concerning this report, please
contact either me or Katherine Schinasi at (202) 512-4841. Key
contributors to this letter were Cristina Chaplain, Sigrid McGinty, Art
Gallegos, Maricela Cherveny, John Oppenheim, and Mike Hazard.
Robert E. Levin:
Director, Acquisition and Sourcing Management:
Signed by Robert E. Levin:
(120318):
FOOTNOTES
[1] U.S. General Accounting Office, Best Practices: A More Constructive
Test Approach Is Key to Better Weapon System Outcomes, GAO/NSIAD-00-199
(Washington, D.C., July 31, 2000).
[2] U.S. General Accounting Office, Military Space Operations: Common
Problems and Their Effects on Satellite and Related Acquisitions, GAO-
03-825R (Washington, D.C.: June 2, 2003).
[3] U.S. General Accounting Office, Defense Acquisitions: Improvements
Needed in Space Systems Acquisition Management Policy, GAO-03-1073
(Washington, D.C.: September 15, 2003) and Defense Acquisitions:
Improvements Needed in Space Systems Acquisition Policy to Optimize
Growing Investment in Space, GAO-04-253T (Washington, D.C.: November
18, 2003).
