Space Acquisitions
Committing Prematurely to the Transformational Satellite Program Elevates Risks for Poor Cost, Schedule, and Performance Outcomes Gao ID: GAO-04-71R December 4, 2003In a multibillion-dollar effort, the Department of Defense (DOD) plans to build a space-based communications system that leverages technologies never before used in space. Such a system would enable DOD to transform how information is collected on potential U.S. adversaries and how military forces are warned of hostile action. The backbone of this system will be the Transformational Satellite (TSAT), which is expected to play a pivotal role in connecting communications networks on the ground, in the air, on ships, and in space. TSAT represents a potential leap forward in communications speed, security, and availability. The Air Force, which heads up DOD's space programs, intends for TSAT to be interoperable with similar systems being acquired for the National Aeronautics and Space Administration (NASA) and the intelligence agencies. The initial TSAT program is expected to cost about $12 billion from 2003 to 2015 for development and production. Several billions more are to be spent acquiring and supporting the associated ground infrastructure, including thousands of user terminals. The Air Force intends to start the acquisition program in December 2003 and expects to launch the first TSAT in 2011. To help pay for TSAT, the Air Force has scaled back its acquisition of the Advanced Extremely High Frequency (AEHF) satellites currently under development. However, because of senior military commanders' concerns about TSAT's risks and the potential delay in delivering improved space communications, the Air Force plans to reassess the need for future AEHF funding in November 2004. If TSAT is considered too high a risk to meet the warfighter's expectations, the contingency plan is to take TSAT's funding--thereby delaying TSAT's development--and use it to buy another AEHF satellite. The Air Force has targeted November 2004 as the latest date such a decision could be made and still include funds for AEHF in the DOD budget submission for fiscal year 2006. We conducted this assessment in response to the large investment planned and the importance of the communications capabilities promised by TSAT and AEHF. Specifically, we assessed the Air Force's readiness to (1) initiate a TSAT acquisition program in December 2003 and (2) make a decision in November 2004 about whether to take TSAT funding and use it to buy another AEHF satellite.
Air Force officials have set two imminent deadlines: starting the TSAT program in December 2003, and deciding whether to shift funding from TSAT to AEHF in November 2004. The Air Force is currently not prepared to make an informed decision in either case. Air Force officials are not ready to initiate the TSAT program in December 2003 because they do not have the knowledge to reliably establish cost, schedule, and performance goals. At program start, program managers are required by law to establish such goals. Our past work on successful acquisition programs has found that these goals cannot be set reliably unless the critical technologies and design have been determined to meet minimum performance requirements. Programs that do not have this knowledge at program start have a much greater risk of resorting to costly design changes later in the development process, asking the warfighter to compromise on desired capabilities, or incurring schedule overruns to correct problems. Realizing that TSAT's schedule is ambitious, the Air Force added 2 years to the acquisition program. However, the extra time was mostly allocated to the latter part of the development process, not to the front end, when program managers typically need the time to become reasonably certain that technologies and early designs will work as envisioned. We are concerned about the Air Force's readiness to make the planned decision in November 2004 to take TSAT funding to buy another AEHF satellite in case the TSAT program falters. Air Force officials have not defined what evaluation criteria they intend to use in making this decision. Senior military commanders want assurance that they will get at least the level of capabilities promised by AEHF early in the next decade. However, senior DOD and Air Force officials told us that if funds were shifted from TSAT back to AEHF, then TSAT--the linchpin of its plan to transform military communications--would be substantially delayed. To promote well-informed and objective investment decisions, our past work has found that decision makers establish and use measurable criteria for evaluating the costs, benefits, and risks of various alternatives.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-04-71R, Space Acquisitions: Committing Prematurely to the Transformational Satellite Program Elevates Risks for Poor Cost, Schedule, and Performance Outcomes
This is the accessible text file for GAO report number GAO-04-71R
entitled 'Space Acquisitions: Committing Prematurely to the
Transformational Satellite Program Elevates Risks for Poor Cost,
Schedule, and Performance Outcomes' which was released on December 04,
2003.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
December 4, 2003:
The Honorable Donald H. Rumsfeld:
The Secretary of Defense:
Subject: Space Acquisitions: Committing Prematurely to the
Transformational Satellite Program Elevates Risks for Poor Cost,
Schedule, and Performance Outcomes:
Dear Mr. Secretary,
In a multibillion-dollar effort, the Department of Defense (DOD) plans
to build a space-based communications system that leverages
technologies never before used in space. Such a system would enable DOD
to transform how information is collected on potential U.S. adversaries
and how military forces are warned of hostile action. The backbone of
this system will be the Transformational Satellite (TSAT),[Footnote 1]
which is expected to play a pivotal role in connecting communications
networks on the ground, in the air, on ships, and in space. TSAT
represents a potential leap forward in communications speed, security,
and availability. The Air Force, which heads up DOD's space programs,
intends for TSAT to be interoperable with similar systems being
acquired for the National Aeronautics and Space Administration (NASA)
and the intelligence agencies.
The initial TSAT program is expected to cost about $12 billion from
2003 to 2015 for development and production. Several billions more are
to be spent acquiring and supporting the associated ground
infrastructure, including thousands of user terminals. The Air Force
intends to start the acquisition program in December 2003 and expects
to launch the first TSAT in 2011.
To help pay for TSAT, the Air Force has scaled back its acquisition of
the Advanced Extremely High Frequency (AEHF) satellites currently under
development. However, because of senior military commanders' concerns
about TSAT's risks and the potential delay in delivering improved space
communications, the Air Force plans to reassess the need for future
AEHF funding in November 2004. If TSAT is considered too high a risk to
meet the warfighter's expectations, the contingency plan is to take
TSAT's funding--thereby delaying TSAT's development--and use it to buy
another AEHF satellite. The Air Force has targeted November 2004 as the
latest date such a decision could be made and still include funds for
AEHF in the DOD budget submission for fiscal year 2006.
We conducted this assessment in response to the large investment
planned and the importance of the communications capabilities promised
by TSAT and AEHF. Specifically, we assessed the Air Force's readiness
to (1) initiate a TSAT acquisition program in December 2003 and (2)
make a decision in November 2004 about whether to take TSAT funding and
use it to buy another AEHF satellite.
RESULTS IN BRIEF:
Air Force officials have set two imminent deadlines: starting the TSAT
program in December 2003, and deciding whether to shift funding from
TSAT to AEHF in November 2004. The Air Force is currently not prepared
to make an informed decision in either case.
Air Force officials are not ready to initiate the TSAT program in
December 2003 because they do not have the knowledge to reliably
establish cost, schedule, and performance goals. At program start,
program managers are required by law to establish such goals.[Footnote
2] Our past work on successful acquisition programs has found that
these goals cannot be set reliably unless the critical technologies and
design have been determined to meet minimum performance requirements.
Programs that do not have this knowledge at program start have a much
greater risk of resorting to costly design changes later in the
development process, asking the warfighter to compromise on desired
capabilities, or incurring schedule overruns to correct problems.
Realizing that TSAT's schedule is ambitious, the Air Force added 2
years to the acquisition program. However, the extra time was mostly
allocated to the latter part of the development process, not to the
front end, when program managers typically need the time to become
reasonably certain that technologies and early designs will work as
envisioned.
We are concerned about the Air Force's readiness to make the planned
decision in November 2004 to take TSAT funding to buy another AEHF
satellite in case the TSAT program falters. Air Force officials have
not defined what evaluation criteria they intend to use in making this
decision. Senior military commanders want assurance that they will get
at least the level of capabilities promised by AEHF early in the next
decade. However, senior DOD and Air Force officials told us that if
funds were shifted from TSAT back to AEHF, then TSAT--the linchpin of
its plan to transform military communications--would be substantially
delayed. To promote well-informed and objective investment decisions,
our past work has found that decision makers establish and use
measurable criteria for evaluating the costs, benefits, and risks of
various alternatives.
We are recommending that you direct the Secretary of Air Force to
develop critical technologies more fully and to conduct early design
studies before starting the TSAT acquisition program. We are also
recommending that you direct the Secretary to establish and use
measurable evaluation criteria for the planned November 2004 funding
decision. Although DOD agrees to adopt such criteria, it believes the
acquisition program can be started because sufficient controls are in
place to allow concurrent development of technology and product design.
BACKGROUND:
DOD intends to develop a new generation of space communications
systems, taking advantage of rapidly advancing technologies. This
reflects an increasing demand and reliance on satellite communications
systems to move larger volumes of information to more users. The Air
Force reports that the demand for communications bandwidth increased
473 percent between Operation Desert Storm in 1991 and Operation
Enduring Freedom in 2001. To help meet this demand, DOD has augmented
its own satellite communications capability with commercial satellites.
However, in each major conflict in the past decade, senior military
commanders still reported shortfalls in communications capacity,
particularly for rapid transmission of large digital files, such as
those created by imagery sensors. DOD's communications studies indicate
the shortfall will continue to grow, despite major improvements in
communications satellites currently in development.
Investment Strategy for Satellite Communications Revised:
In 1996, DOD developed and began to implement a space investment
strategy that proposed a new mix of improved communication satellites
for use in 2010 and beyond. Among the proposed systems were the AEHF
satellite, the Wideband Gapfiller Satellite (WGS), Advanced Polar
System (APS), and Advanced Wideband Satellite (AWS), a less capable and
earlier version of TSAT. At that time, DOD believed that AWS, AEHF, and
WGS would provide a significant increase in communications capacity and
would meet the warfighters' needs in 2010 and beyond. (More information
about these satellite systems and their associated acquisition programs
is included in enc. I.):
In 2001, DOD developed a new Transformational Communications
Architecture that uses emerging communications technologies. The
architecture is expected to transform future combat and intelligence
operations, with TSAT playing a critical role. The concept is to use
laser-based and improved radio frequency transmission systems and high-
speed, Internet-like networks that will link communications systems on
the ground, in the air, on ships, and in space. Instead of circuit-
based systems, such as those used today to link specific sending-and-
receiving devices, future systems are expected to connect multiple
sending-and-receiving devices at the same time. The ultimate goal is to
remove the existing constraints to communication and enable
transmissions regardless of location, size, or message.
Knowledge-based Acquisition Strategy Results in Better Outcomes:
Historically, DOD has had difficulty meeting the cost, schedule, and
performance goals that were established at the start of its major
defense acquisition programs such as TSAT. DOD's investments in money
and time have far exceeded initial estimates for developing and
acquiring communications satellites and other weapon systems. In
addition, weapon systems have frequently been saddled with performance
shortfalls.[Footnote 3] To address some of these difficulties, DOD
recently implemented a new space systems acquisition policy, which
intends to provide decision makers in the Air Force with more
consistent and robust information on costs, technologies, and
requirements. The new acquisition policy also promotes rapid
introduction of emerging technologies into space systems and allows
technology, design, and system development to occur concurrently in an
effort to speed the acquisition process. A recent GAO report identified
some positive aspects of the policy; however, the report stated that
any benefits will be limited because the policy permits major
investments in new programs before managers know what resources are
really required to deliver a promised capability.[Footnote 4]
Our work on best practices in weapon system acquisitions has shown that
program managers have a much higher probability of meeting cost,
schedule, and performance objectives if the needed technologies are
mature and the developing contractor has completed early design studies
before starting the acquisition program. Having this knowledge in hand
means managers can build a strong business case and ensure their
products can be successfully developed. A business case provides the
necessary structure for managers to identify the best product solution
based on knowledge of performance, constraints and assumptions, and a
risk-adjusted cost-benefit analysis. In the past several years, GAO has
developed a knowledge-based acquisition model based on best practices
by leading companies. The best practices model has three knowledge
points. Each knowledge point builds on the preceding one. The acquired
knowledge is used to identify and reduce any risks before moving a
product to the next stage of development. Figure 1 shows when the three
knowledge points occur on the best practices model.
The first knowledge point sets the stage for the eventual outcome of an
acquisition program--desirable or problematic. When the customer's
needs match the developer's resources (which include technology, design
knowledge, time, and money) before program start, successful outcomes
are much more likely to occur. If a match occurs after program start,
managers often make additional, unanticipated investments in money and
time because gaps between requirements and resources are discovered
later in the process.
AIR FORCE IS SETTING COST, SCHEDULE, AND PERFORMANCE GOALS AND STARTING
TSAT BEFORE CRITICAL KNOWLEDGE IS ATTAINED:
By December 2003, when the TSAT program is scheduled to start, Air
Force officials are required by law to establish cost, schedule, and
performance goals, but the knowledge they need to set reliable goals is
still not available. Critical technologies are underdeveloped and early
design studies have not been started. Without this essential knowledge,
the Air Force is likely to have difficulty developing a sound business
case for starting the TSAT program. If the Air Force proceeds without a
sound business case, the program is at risk of higher costs, lower
performance, and delays in providing capabilities to the warfighters.
Our work has found that successful commercial and DOD development
programs insist on having mature technologies and early design studies
to support the business case.[Footnote 5]
Critical Technologies Are Immature:
Critical technologies are necessary building blocks for a system to
meet its minimum performance requirements. If these technologies are
not available when needed, the system cannot be completed as planned.
And because technology development does not happen on a planned or
predictable schedule, it is difficult to predict when or if a critical
technology will mature. This is why leading commercial companies demand
that critical technologies are mature before the commitment to a new
system is made. Within the federal government, NASA leveraged this best
practice by developing an analytical tool to assess technology
maturity. This tool--adopted by many DOD programs--establishes
Technology Readiness Levels (TRL) for demonstrated performance, with a
higher value indicating a greater maturity level. (The various levels
are defined in enc. II.):
According to best commercial practices and DOD guidance, the minimum
acceptable level for a technology to be included in an acquisition is
TRL 6. At this level, the technology is considered sufficiently mature
and has been engineered into a subsystem or prototype that closely
resembles the final design. Also, the technology has been successfully
demonstrated to work in a relevant environment. DOD policy[Footnote 6]
prefers the maturity to reach TRL 7--a prototype demonstration in an
operational environment. A TRL 7 for a satellite would mean the
technology prototype has achieved form, fit, and function and has been
demonstrated in space. Commercial satellite companies frequently meet
these criteria by including a new technology on an existing satellite
design for demonstration purposes. Also, NASA usually requires all
mission-critical technologies to be demonstrated in space before being
placed on a new system. In some cases, demonstrating space technologies
in an operational environment is important because operating a system
in the harsh temperatures and radiation environment of space--where a
satellite must last essentially maintenance-free for 10 to 15 years--is
much more challenging than land-based operations. The new space
acquisition policy does not require a minimum threshold for including
new technology on a space acquisition program.
Critical technologies for TSAT include laser optics that can transport
information over long distances in much larger quantities than radio
waves; high-speed routers that enable multi-user networks,
sophisticated data packaging; security algorithms and management
utilities; multi-beam antennas; and software reprogrammable terminals.
Table 1 shows that most of these technologies were at a TRL 3 or 4 in
October 2003. When a technology is classified as a TRL 3, it means most
of the work performed so far has been based on analytical studies and a
few laboratory tests may have been conducted. A TRL 4 means some of the
key components have been wired and integrated and have been
demonstrated to work together in a laboratory environment. Significant
effort is required to move from these TRL levels to a TRL 6, the
minimum needed to effectively begin a new acquisition program. As shown
below, the program office estimates that most of these technologies
will have reached a TRL 6 threshold by fiscal year 2006.
Table 1: Current and Expected Technical Maturity Levels of TSAT
Technologies:
Critical technology: Information protection; TRL as of October 2003: 3-
4; When TRL 6 is expected: FY2006.
Critical technology: Laser communication; TRL as of October 2003: 4-5;
When TRL 6 is expected: FY2006.
Critical technology: Information packet processing; TRL as of October
2003: 6; When TRL 6 is expected: FY2003.
Critical technology: Antenna for communications on the move; TRL as of
October 2003: 4-5; When TRL 6 is expected: FY2006.
Critical technology: Information transmission management; TRL as of
October 2003: 3-4; When TRL 6 is expected: FY2006.
Critical technology: Protected bandwidth efficient modulation; TRL as
of October 2003: 3-4; When TRL 6 is expected: FY2006.
Source: MILSATCOM Joint Program Office.
[End of table]
If one or more of TSAT's critical technologies encounters development
problems, a backup technology should be available for insertion into
the program. The laser communications technology does not have a backup
provided by another satellite program. Typically, a backup technology
does not meet all of the user's requirements and/or can negatively
affect other design requirements of the new system, such as weight and
power. For example, the alternative for TSAT's communications antenna
is the current AEHF antenna, which does not provide the essential
communications-on-the-move capability. Reverting to alternative
technologies late in a development program results in a series of
costly design changes and a need to go back to the warfighter to
determine if the changes are acceptable.
Early Design Studies Have Not Been Started Yet:
As of October 2003, 2 months before TSAT's scheduled start, the Air
Force had not awarded contracts for early design studies. In the case
of successful programs, we have found that the developing contractor
evaluates the early designs according to system engineering principles
to assure that designs are technically feasible, match the user's
needs, and can be accomplished within the time frame and funds
available. Without this disciplined engineering process, programs can
learn too late that designs needed to achieve the warfighter's
requirements are not feasible. Program managers then have little choice
but to ask for more time and money to develop better designs, or they
must compromise by asking the warfighter to accept a less capable
backup design or technology. When discovered late in a development
program, these changes can be costly. Our prior work has shown that the
cost to change the design increases significantly as a program
progresses through the key decision points of an acquisition program.
For this reason, most commercial companies want greater assurance early
in a program that the design is feasible and producible.
The Air Force plans to competitively award contracts for early design
studies in December 2003, which is when the TSAT program is scheduled
to start. These studies are to be completed in 2006, when contractors
are expected to deliver a design specification in preparation for final
design efforts. To prepare for the next step--critical design review--
in 2007, the Air Force plans to assess the preliminary designs and
select one or both contractors to continue with detailed design studies
and development activities. Figure 2 shows key dates in TSAT's
acquisition schedule.
After hearing senior warfighters express concerns about the ambitious
schedule, the Air Force recently extended the launch date for the first
TSAT from 2009 to 2011. However, the additional 2 years was mostly
allocated to the build-and-test phase prior to launch. The front end of
the acquisition schedule--technology development and design--remains
much as it was before the extension. The technology development phase
was not extended and the preliminary design and critical design review
dates did not change. Based on our past reviews, the importance of
technology development and design to the success of a program is
critical and TSAT's current status shows significant immaturity to be
overcome.
LACK OF EVALUATION CRITERIA RAISES CONCERNS ABOUT PENDING DECISION TO
SHIFT FUNDS FROM TSAT TO AEHF:
Despite intense interest across DOD in the November 2004 decision, Air
Force officials have not defined what evaluation criteria they intend
to use to assess alternatives if the TSAT program should falter. Senior
military commanders have asked for assurances that promised
communications capabilities will be delivered early in the next decade.
If TSAT is likely to miss its promised launch date of 2011, they want
funding to be allocated to complete the AEHF constellation of
satellites. However, senior DOD and Air Force officials told us that if
a fourth AEHF were acquired and a full AEHF constellation were
delivered to the warfighter as originally planned, decision makers and
funding organizations within DOD may want to wait until AEHF has
reached the end of its useful life before replacing it with a next-
generation satellite, such as TSAT. If the fourth AEHF is acquired,
officials believe TSAT will be delayed by at least a decade. To these
officials, this is not a tenable scenario because they see TSAT as the
linchpin in DOD's plan to transform military communications and related
combat systems.
To promote well-informed and objective investment decisions, our past
work has found that decision makers establish and use measurable
criteria for evaluating the costs, benefits, and risks of various
alternatives. Although senior DOD and Air Force officials told us that
they expect to have accomplished a number of tasks before making the
November 2004 decision, they have not established measurable evaluation
criteria for deciding whether to shift funds from TSAT back to AEHF.
CONCLUSIONS:
DOD has embarked on a new transformational communications architecture
to take advantage of emerging technologies and to remove communications
constraints from combat. The department has told the warfighter and
Congress that TSAT is a key system that is necessary to achieve this
architecture. Responding quickly, the Air Force has set an imminent
deadline of December 2003 to start the TSAT program. By starting the
program so soon, the Air Force is moving ahead without mature
technologies and early design studies--two pillars of knowledge that
would help program officials to reliably establish cost, schedule, and
performance goals. This knowledge is not expected to be available until
2006. Our work over the years has found that when programs have been
started without the requisite knowledge, program managers and
contractors are later burdened by unreasonable expectations about cost,
schedule, and performance. Problems usually arise later that lead to
cost increases, delays in delivering needed capability to the
warfighters, and performance shortfalls.
For the planned November 2004 decision about whether to fund TSAT or
AEHF, Air Force officials would be in a better position to make a well-
informed, objective decision if they establish and use specific
criteria for evaluating alternative investments. Reporting the Air
Force's decision-making criteria and rationale to Congress would
enhance transparency and provide Congress with better information for
its oversight and funding responsibilities.
RECOMMENDATIONS FOR EXECUTIVE ACTION:
To promote better cost, schedule, and performance outcomes, we
recommend that you direct the Secretary of Air Force to delay the start
of the TSAT acquisition program until technologies have been
demonstrated to be at an acceptable level of maturity (at least TRL 6)
and until the developing contractor has determined through systems
engineering that the design is feasible and producible. We also
recommend that you direct the Secretary to provide the appropriate
level of funding necessary to gain this knowledge, which is critical
for building a business case to start the TSAT program at a later time.
To promote a well-informed and objective decision--now scheduled for
November 2004--about whether to fund another AEHF satellite, we further
recommend that you direct the Secretary of Air Force to:
* establish measurable criteria for use when evaluating alternative
investments in TSAT and AEHF and report this criteria in the Air
Force's 2005 budget submission;
* consider the alternative investments in TSAT and AEHF against these
measurable criteria; and:
* provide the rationale for how these criteria were applied in the Air
Force's 2006 budget submission.
AGENCY COMMENTS AND OUR EVALUATION:
In commenting on a draft of this report, the Deputy Assistant Secretary
of Defense for Networks and Information Integration disagreed with our
primary recommendation to delay the start of the TSAT acquisition
program until technologies are sufficiently matured and until the
contractor determines through systems engineering principles that the
design is feasible. DOD contends that the new Air Force National
Security Space Acquisition Policy provides sufficient controls to allow
concurrent development of technology and product design. DOD states
that starting the TSAT program enables it to establish the funding and
program controls--such as managing to the acquisition program baseline-
-provided by the new space acquisition policy. DOD did, however, concur
or partially concur with the other recommendations to provide funding
to mature TSAT's critical technologies and early designs, to establish
criteria for making decisions, and to report these criteria and
decisions to Congress.
We believe the new space acquisition policy does not have sufficient
controls to reverse the higher costs and longer schedules that have
plagued a number of satellite programs. The added risks of concurrent
technology and product development have not helped improve the typical
outcome for satellite programs. In a series of best practices reports
issued over the years, we have identified problems resulting in
substantially different cost and schedule outcomes when compared with
initial expectations at the outset of a new acquisition program. We
have offered improved approaches based on the best commercial and
defense practices. DOD has endorsed the practices that call for a
disciplined acquisition approach, one that separates technology from
product development and bases decisions at key junctures on a set of
critical product knowledge captured by the decision point. DOD
incorporated this knowledge-based approach in its new acquisition
system policy.[Footnote 7]
DOD's new space acquisition policy, on the other hand, is a step
backward and is similar to an older acquisition policy that contributed
to many unsuccessful acquisition programs of the past. DOD's history is
filled with examples of programs that concurrently developed technology
and new products and made decisions based on risk mitigation plans
instead of knowledge about the new products. Our June 2003
report[Footnote 8] on common problems in satellite programs identified
Milstar, SBIRS-Low, SBIRS-High, AEHF, and others as suffering the
consequences of this earlier acquisition strategy. Additionally, we
have found that setting an acquisition program baseline that is not
rooted in key product knowledge is unreliable and not useful as a
management tool. In fact, starting the program before technologies are
mature and a feasible design study is completed reduces accountability
and straps the program manager and the contractor with unreasonable
expectations in the baseline. Therefore, we believe that because DOD's
new space acquisition policy does not require a knowledge-based
acquisition strategy, it is destined to repeat the problems of the
past.
DOD stated that extensive studies done over the last two years provide
sufficient information for the Milestone Decision Authority to
determine if the TSAT program should be initiated. However, these
studies do not provide product-specific knowledge for building a
business case for TSAT. Instead, these studies were focused on
developing the overarching communications architecture rather than
detailed technology and design information needed to build and launch
TSAT.
While it is key to complete early design efforts before starting the
program, substantial investments in system design and development are
at risk if the Air Force cannot demonstrate TSAT's technologies, a
number of which were still in the early paper study phase without
hardware demonstrations to support that they would work. In its fiscal
year 2004 budget submission, the Air Force had budgeted over $800
million in fiscal years 2004 and 2005 for system design and
development.
To support its case for starting the TSAT program in December 2003, DOD
states backup technologies exist and are ready to fill any technology
void that might occur. They believe this will reduce the risk. However,
there are no backup technologies that will satisfy the two most
critical warfighter requirements--laser communications (critical to
transporting intelligence, surveillance, and reconnaissance data) and
communications on the move (critical to the Future Combat System).
These capabilities were the primary basis for persuading the warfighter
to favor the uncertain future of TSAT rather than to acquire the full
constellation of four AEHF satellites, which would have provided a 500
percent increase over the communications capability used in Operation
Iraqi Freedom.
If TSAT's investments were based on knowledge captured from mature
technologies and feasible design, then these informed decisions would
reduce the potential for major and costly changes as the program enters
the build-and-demonstration phase, when it is too late to consider
other options. We believe it is better to keep options open now, such
as AEHF, and decide at a later time when enough knowledge has been
gained to ensure TSAT is the right solution for the 2010 time frame.
Our past work shows the negative outcomes of the concurrent and risk
mitigation approach to acquisition. We also have shown the potential
for more successful outcomes if a knowledge-based approach is applied.
Therefore, we stand by our recommendation that TSAT's program start
should be delayed until technologies are mature and the developing
contractor has completed studies to demonstrate a feasible design.
To ensure that the warfighter is delivered an improved capability no
later than 2011, DOD intends to decide in November 2004, based on an
assessment of TSAT's progress, whether funding should be diverted back
to the AEHF program. In its comments, the Air Force suggests criteria
for this decision point that can only result in continuing the TSAT
program. For example, criteria for laser communications or
communications on the move do not apply to AEHF. These are capabilities
promised by TSAT, not AEHF. We believe that the criteria should be
based on the maturity of critical technologies and early design of
TSAT. To ensure the transparency and objectivity of the decision
process in November 2004, these criteria should be provided to Congress
in the fiscal year 2005 budget for TSAT, not--as DOD suggests--in the
2006 budget, when the decision will already have been made.
In response to DOD's detailed comments, we made changes to the report
where appropriate to correct technical inaccuracies. DOD's comments are
provided in enclosure III.
SCOPE AND METHODOLOGY:
In conducting our review, we analyzed the extent to which the TSAT and
APS programs have acquired the knowledge needed to set specific cost,
schedule, and performance goals. To do this, we compared the
acquisition strategy with GAO's knowledge-based acquisition model and
analyzed the differences between them. We specifically focused on the
portion of knowledge-based acquisition dealing with the necessity of
matching user's needs with developer's resources prior to making a
development commitment. We collected and analyzed information from the
Office of the Assistant Secretary of Defense for Command, Control,
Communication and Intelligence (ASDC3I), Defense Information Services
Agency (DISA), the National Security Agency (NSA), Air Force Space
Command (AFSPC), U.S. Strategic Command (USSTRATCOM), Military
Satellite Communication Joint Program Office (MJPO), Joint Forces
Command (JFCOM), Aerospace Corporation and RAND Corporation. We
conducted our review from February 2003 through November 2003 in
accordance with generally accepted government auditing standards.
As you know, 31 U.S.C. 720 requires the head of a federal agency to
submit a written statement of actions taken on our recommendations to
the Senate Committee on Governmental Affairs and the House Committee on
Government Reform not later than 60 days after the date of the report
and to the Senate and House Committees on Appropriations with the
agency's first request for appropriations made more than 60 days after
the date of this report.
We are sending copies of this report to interested congressional
committees. We will also make copies available to others upon request.
In addition, the report will be available at no charge on the GAO Web
site at http://www.gao.gov.
If you or your staff has any questions concerning this report, please
contact me at (202) 512-4841. Key contributors to this report were Lily
Chin, Mike Hazard, Dave Hubbell, Travis Masters, and Matt Mongin.
Sincerely yours,
Robert E. Levin:
Director, Acquisition and Sourcing Management:
Signed by: Robert E. Levin:
Enclosure I: Descriptions of Communication Satellites:
The Air Force is developing the following communication satellites.
Wideband Gapfiller Satellite:
The Wideband Gapfiller Satellite (WGS) system is a joint Air Force and
Army program intended to provide communications to U.S. warfighters,
allies, and coalition partners during all levels of conflict, short of
nuclear war. WGS will provide essential communications services for the
commanders in chief to command and control their tactical forces.
Tactical forces will rely on WGS to provide high-capacity links to the
terrestrial portion of the Defense Information Services Network. WGS is
the next generation wide-band component in the Department of Defense's
(DOD) future Military Satellite Communications architecture. WGS is
composed of three principal segments: Space Segment (satellites),
Terminal Segment (users), and Control Segment (operators). The WGS
program is leveraging commercial methods and technological advances in
the satellite industry to rapidly design, build, launch, and support a
constellation of highly capable military communications satellites.
The WGS program is being conducted as a DOD commercial acquisition and
as such is not subject to the same milestone and/or review processes
required in other space acquisition programs. The Air Force reports
that 95 percent of satellite content will be commercial off-the-shelf
products. The total budget for purchasing five WGSs is $1.5 billion.
The contract is firm fixed price over 10 years and was awarded to
Boeing Satellite Systems in January 2001. The Air Force purchased the
first two satellites in fiscal year 2002 and the third satellite in
fiscal year 2003. It plans to purchase satellites four and five in
fiscal years 2007 and 2008, respectively. The first two WGS satellites
are scheduled for launch in fiscal year 2005, with the third satellite
planned for launch in fiscal year 2006.
Upon first launch into geosynchronous orbit in 2005, WGS will be the
DOD's most capable and powerful communications satellite. Ultimately,
five WGSs will be in orbit, providing service in both the X-and Ka
band-radio frequencies. Each satellite is expected to have a capacity
of at least 2,100 megabits per second. WGS will augment X-band
communications now provided by the Defense Satellite Communications
System (DSCS) and one-way Ka-band service provided by the Global
Broadcast Service (GBS). Additionally, WGS will provide new two-way Ka-
band services. These satellites are not interconnected. They will,
however, provide communications capacity, connectivity, and
flexibility for U.S. military forces while maintaining full
interoperability with existing and programmed DSCS and GBS terminals.
Advanced Extremely High Frequency Satellite:
The Advanced Extremely High Frequency (AEHF) satellite system is to be
DOD's next generation of high-speed, secure communication satellites.
This satellite system is intended to replace the existing
communications satellites with improved, survivable, jam-resistant,
worldwide, secure communication capabilities at lower launch costs.
AEHF is to support the entire range of data rates to provide assured
communications across the entire spectrum of conflict, including
nuclear war. AEHF is also designed to be "backward compatible" with
existing satellites, that is, it will support both low and medium data
rates as necessary until an AEHF constellation with higher data rates
becomes available at initial operating capability (two satellites on
orbit). The first satellite is currently planned to launch in 2006 and
the second is scheduled to launch in 2007.
The Air Force is responsible for funding, developing, and producing the
AEHF satellites and the associated ground control systems. The Air
Force's budget for developing and acquiring the first three AEHF
satellites is $4.8 billion. Each service--Army, Navy, and Air Force--is
separately responsible for funding, developing, and producing its own
terminals to communicate with AEHF.
The AEHF program began in August 1998, and the final constellation will
be composed of satellites in geosynchronous orbit that can transmit
data to each other via radio frequency cross links, and communicate
with ground stations and communication terminals carried by air, sea,
and ground forces. Each satellite will have a capacity of about 250
megabits per second. Users communicate with the satellites through
their terminals. The mission control segment provides command and
control that directs the movements and other operations of satellites.
Transformational Satellite Communications:
The Transformational Satellite (TSAT) communications system is designed
to provide improved, survivable, jam-resistant, worldwide, secure and
general purpose communications as part of an independent but
interoperable set of space-based systems that will support the National
Aeronautics and Space Administration, DOD, and the intelligence
community. TSAT will replace the current satellite system and
supplement AEHF.
The TSAT architecture, requirements, and cost baselines are to be
approved in December 2003. Initial design contracts are to be awarded
in December 2003; therefore, the final configuration of the TSAT system
remains to be determined. Air Force budget documentation for TSAT
(funded under the Advanced Wideband Satellite budget line) shows a
total cost of $10.9 billion for purchasing the first five satellites
plus a spare.
The TSAT system will be the key transport mechanism of DOD's space-
based network communications system, which has individual satellites
operating as routers in space. The TSAT constellation of five
satellites will provide continuous communication coverage from 65
degrees south latitude to 65 degrees north. The satellites will support
communications in the EHF and Ka band radio frequency bands, in
addition to passing communications via lasers. The capacity of each
satellite is expected to be at least 10 times greater than the AEHF
satellites. The Air Force is currently conducting development
activities necessary in order to make a decision to start the program
in December 2003. The Air Force plans to launch the first TSAT in
fiscal year 2011. The first two satellites will have radio frequency
satellite cross links to engage the AEHF satellites as well as having
the laser cross links; the third through the fifth satellites will have
laser cross links only.
Advanced Polar Satellite:
The Advance Polar System (APS) is a part of the Air Force's
transformational communication architecture and is being developed and
acquired as part of the TSAT/APS acquisition program. APS will provide
the next generation protected EHF band, Ka band, and laser satellite
communications capability in the north polar region starting in fiscal
year 2012. APS will support strategic as well as tactical users who
require anti-jam and low probability of detection EHF satellite
communications. The results of the transformational communications
architecture definition will affect the APS program content.
Requirements are based on the July 1995 Polar Operational Requirements
Document. According to Air Force program officials, APS is to be a
"lighter" (i.e., lower capacity) version of the TSAT. The current APS
plan is to acquire three satellites (two funded with development funds
and one funded with procurement dollars) and associated ground
infrastructure for $1.2 billion. The three APS satellites will be
placed in highly inclined orbits and are expected to provide continuous
communication services to forces deployed from 65 degrees north to the
North Pole (90 degrees north).
Enclosure II: Descriptions of Technology Readiness Levels:
The Interim Defense Acquisition Guidebook (formerly DOD 5000.2-R)
directs that technology readiness assessments, using Technology
Readiness Levels (TRL) or some equivalent assessment methodology, for
critical technologies shall occur sufficiently before key decision
points B and C to provide useful technology maturity information to the
acquisition review process. TRLs, originally developed by the National
Aeronautics and Space Administration (NASA), are measured along a scale
of 1 to 9, starting with paper studies of the basic concept and ending
with a technology that has proven itself in actual usage on the
intended product. As the TRL scale increases, the risks associated with
uncertain technology decrease, because more is known about their
capabilities and performance. Unexpected problems can arise at every
level, and effort must be expended to overcome them. This effort takes
time and can delay the progress to the next readiness level. According
to our previous reviews of best commercial practices and DOD guidance,
a minimum level of TRL 6 should be reached before committing to a space
acquisition program. Table 1 provides a detailed explanation of each
TRL.
Table 2: TRL Scale for Assessing Critical Technologies:
Technology Readiness Levels: 1. Basic principles observed and
reported; Technology Readiness Level Description: Lowest level of
technology readiness. Scientific research begins to be translated into
technology's basic properties.
Technology Readiness Levels: 2. Technology concept and/or application
formulated; Technology Readiness Level Description: Invention begins.
Once basic principles are observed, practical applications can be
invented. The application is speculative and there is no proof or
detailed analysis to support the assumption. Examples are still
limited to paper studies.
Technology Readiness Levels: 3. Analytical and experimental critical
function and/or characteristic proof of concept; Technology Readiness
Level Description: Active research and development is initiated. This
includes analytical studies and laboratory studies to physically
validate analytical predictions of separate elements of the
technology. Examples include components that are not yet integrated or
representative.
Technology Readiness Levels: 4. Component and/or breadboard validation
in laboratory environment; Technology Readiness Level Description:
Basic technological components are integrated to establish that the
pieces will work together. This is relatively "low fidelity" compared
to the eventual system. Examples include integration of "ad hoc"
hardware in a laboratory.
Technology Readiness Levels: 5. Component and/or breadboard validation
in relevant environment; Technology Readiness Level Description:
Fidelity of breadboard technology increases significantly. The basic
technological components are integrated with reasonably realistic
supporting elements so that the technology can be tested in simulated
environment. Examples include "high fidelity" laboratory integration
of components.
Technology Readiness Levels: 6. System/subsystem model or prototype
demonstration in a relevant environment; Technology Readiness Level
Description: Representative model or prototype system, which is well
beyond the breadboard tested for level 5, is tested in a relevant
environment. Represents a major step up in a technology's demonstrated
readiness. Examples include testing a prototype in a high fidelity
laboratory environment or in simulated operational environment.
Technology Readiness Levels: 7. System prototype demonstration in an
operational environment; Technology Readiness Level Description:
Prototype near or at planned operational system. Represents a major
step up from level 6, requiring the demonstration of an actual system
prototype in an operational environment. Examples include testing the
prototype in a test bed aircraft.
Technology Readiness Levels: 8. Actual system completed and qualified
through test and demonstration; Technology Readiness Level
Description: Technology has been proven to work in its final form and
under expected conditions. In almost all cases, this level represents
the end of true system development. Examples include developmental
test and evaluation of the system in its intended weapon system to
determine if it meets design specifications.
Technology Readiness Levels: 9. Actual system proven through
successful mission operations; Technology Readiness Level Description:
Actual application of the technology in its final form and under
mission conditions, such as those encountered in operational test and
evaluation. Examples include using the system under operational
mission conditions.
Source: GAO based on NASA and DOD guidance.
[End of table]
Enclosure III: Comments From the Department of Defense:
OFFICE OF THE ASSISTANT SECRETARY OF DEFENSE:
6000 DEFENSE PENTAGON
WASHINGTON, DC 20301-6000:
NOV 21 2003:
NETWORKS AND INFORMATION INTEGRATION:
Mr. Robert E. Levin:
Director, Acquisition and Sourcing Management:
U. S. General Accounting Office:
Washington, D.C. 20548:
Dear Mr. Levin,
This is the Department of Defense (DoD) response to the General
Accounting Office (GAO) draft report, "Space Acquisitions: Committing
Prematurely to the Transformational Satellite Program Elevates Risks
for Poor Cost, Schedule, and Performance Outcomes," GAO-04-71R (GAO
Code 120225) dated October 29, 2003.
The report includes five recommendations to the SECDEF. The Department
of Defense non-concurs with the first recommendation, and partially
concurs or concurs with the remaining four. Specific DoD comments on
each of the recommendations are enclosed as Attachment 1.
The DoD does not agree with the key finding of the GAO report that the
Air Force is not currently prepared to make informed decisions
regarding either the starting of the TSAT program in the December 2003
timeframe or to shift funding from TSAT to AEHF in November 2004. The
DoD has conducted extensive studies at the architecture and systems
level over the past two years, by the government and by three
contractor teams. The government developed a reference architecture
which validated a point solution that satisfies a set of cost /
schedule / performance criteria. The users at a Senior Warfighter Forum
(SWarF) and JROC approved this architecture. The DoD believes that
contractor, government, and FFRDC studies are providing sufficient
information to the TSAT Milestone Decision Authority to determine if
the program is sufficiently mature to warrant transition into a formal
acquisition program at Phase B of the NSS Acquisition Policy 03-01.
The GAO report expresses concern that moving the TSAT program past KDP-
B before the relevant technologies are at Technology Readiness Level
(TRL)-6 could induce substantial cost, schedule, and performance risk.
The DoD has mitigated these risks by developing an evolutionary
acquisition strategy and technology investment plan structured to have
all relevant technologies reach TRL 6 prior to system PDR. This
technology investment plan will allow us to manage progress and, among
other things, plan in advance if we need to use already identified
fallback technologies to mitigate risk. We will manage this technology
investment to detailed milestones in order to be able to make decisions
as to if and when a fallback technology must be used if the primary
technology does not mature as planned. These fallback technologies are
all already at, or above, TRL-6. Even if the initial increment one
TSAAs have to use all of the fallback technologies, which is not
likely, they would still be much more capable than AEHF. Additionally,
even a reduced capability increment one TSAT puts DoD on the path to
transformation and net-centric capability, so critical to our future
operations.
While we agree with the GAO that it would be inappropriate to commit to
a detailed design with immature technology, we note that NSS
Acquisition Policy 03-01 states that Space Programs are to be baselined
at the beginning of the Risk Reduction and Technology Development
phase, which is KDP-B. The increase in reporting requirements, and the
program reviews associated with a baselined Phase B program facilitates
increased DoD oversight/insight. The Transformational Communications
MILSATCOM (TCM) program acquisition strategy has been developed in
accordance with this precept, which, again, is the reason our
technology investment plan is structured to have all relevant
technologies reach TRL 6 prior to system PDR. This approach allows
development of these maturation plans in synchronization with
development of the program requirements. This, in turn, allows the
products based on the technologies to be tailored to best meet and
refine user needs within cost, schedule and risk constraints. Program
management and DoD insight are aided by maintenance of an APB
(Acquisition Program Baseline), through periodic acquisition reports,
and through the program reviews associated with Phase B (e.g. SDR and
PDR). User needs are tracked through the transition of the Capabilities
Development Document (CDD) into the Capabilities Production Document
(CPD). To expend the funds that the GAO and the Air Force agree are
required without the reporting and review associated with a baselined
program may, in fact, limit the effectiveness of the technology
maturation.
It is also important to realize that the TSAT system will be procured
using a capabilities based approach. In order to maintain schedule and
cost, capabilities will be traded in a joint warfighter/ acquisition
forum. This "cost as an independent variable" (CAN) approach allows the
DoD to develop affordable increments of militarily useful capability.
We believe that the best way to provide the most transformational
capability to the warfighter at the earliest possible date is to pursue
this approach, and proceed to a KDP-B.
The GAO report itself contains a number of technical and other
inaccuracies. Detailed comments are provided in Attachment 2.
The DoD appreciates the opportunity to respond to the GAO draft report.
Signed by:
Michael S. Frankel:
Deputy Assistant Secretary of Defense:
Networks and Information Integration (C3, Space, and IT Programs):
Enclosures:
DoD Comments to GAO Recommendations Detailed Technical Comments on GAO
Report:
Attachment 1:
GAO DRAFT REPORT DATED OCTOBER 29, 2003 GAO-04-71R (GAO CODE 120225):
"Space Acquisitions: Committing Prematurely to the Transformational
Satellite Program Elevates Risks for Poor Cost, Schedule and
Performance Outcomes":
DEPARTMENT OF DEFENSE COMMENTS TO THE GAO RECOMMENDATIONS:
RECOMMENDATION 1: The GAO recommended that the Secretary of Defense
direct the Secretary of the Air Force to delay the start of the TSAT
acquisition program until technologies have been demonstrated to be at
an acceptable level of maturity (at least TRL 6) and until the
developing contractor has determined through systems engineering that
the design is feasible and producible. (p. 9/GAO Draft Report):
DOD RESPONSE: Non-Concur. Technologies selected for incorporation on
TSAT will be at or above Technology Readiness Level 6 by Preliminary
Design Review in FY06. The DoD has further mitigated technology risk by
developing an evolutionary acquisition strategy. This strategy allows
us to select from identified fallback technologies if required to
mitigate program risk. These fallback technologies are all already at,
or above, TRL-6. Even if the initial increment one TSATs had to use all
o^ the fallback technologies, which is not likely, they would still be
much more capable than AEHF. Additionally, even a reduced capability
increment one TSAT puts DoD on the path to transformation and net-
centric capability, so critical to our future operations. Program
initiation, governed by National Security Space (NSS) Acquisition
Policy 03-01, begins before PDR.
RECOMMENDATION 2: The GAO recommended that the Secretary of Defense
provide the appropriate level of funding necessary to gain this
knowledge, which is critical for building a business case to start the
TSAT program at a later time.
(p. 9/GAO Draft Report):
DOD RESPONSE: Partially Concur. The DoD is currently reviewing TSAT
RDT&E funding to ensure that it is adequate to do the technology and
non-recurring engineering work necessary to acquire the program with
reasonable risk. The DoD will re-visit funding each year to ensure
adequate funding is maintained to achieve TSAT program objectives. The
DoD position is that the Dec 03 - Jan 04 timeframe is the proper time
to start the TSAT program and establish a program baseline so that
these investments can be properly tracked and managed. Our evolutionary
strategy, fallback technologies, and CAIV approach will allow us to
manage risk while developing an affordable increment of militarily
useful capability.
RECOMMENDATION 3: The GAO recommended that the Secretary of Defense
direct the Secretary of the Air Force to establish measurable criteria
for use when evaluating alternative:
Attachment 1:
investments in TSAT and AEHF and report these criteria in the Air
Force's 2005 budget submission. (p. 10/GAO Draft Report):
DOD RESPONSE: Partially Concur. The decision will be based on known
risks balanced against warfrghter requirements. Measurable criteria are
being developed and will be used to make this decision in November
2004. For example, one of the criteria we are considering is the
ability to support communications on the move (COTM) at data rates
above 250 kbps to a one-foot antenna mounted on a vehicle traveling at
speeds above 25 mph. Another is the ability to support multiple high
date rate Airborne ISR missions with interoperable laser communications
links. The DoD will provide these criteria and resulting decision as
part of the FY06 President's Budget process.
RECOMMENDATION 4: The GAO recommended that the Secretary of Defense
direct the Secretary of the Air Force to consider the alternative
investments in TSAT and AEHF against these measurable criteria. (p. 10/
GAO Draft Report):
DOD RESPONSE: Concur. The investment strategy for TSAT will support an
evolutionary approach potentially using fallback technologies, this
approach will allow us to manage risk while developing an affordable
increment of militarily useful capability much more advanced than AEHF.
This approach will also start us on the transformational path to net-
centric operations that can be integrated with the DOD's other
investments in transformational capabilities. These TSAT results will
be compared to progress on the AEHF program, and against measurable
criteria to form the basis of our decision. The DoD will use all
appropriate criteria and consider alternative investments strategies
that satisfy the needs of the warfighters.
RECOMMENDATION 5: The GAO recommended that the Secretary of Defense
direct the Secretary of the Air Force to provide the rationale for how
these criteria were applied in the Air Force's 2006 budget submission.
(p. 10/GAO Draft Report):
DOD RESPONSE: Concur. We will use the results of our evolutionary
acquisition strategy as the basis of our FY06 budget submission to
Congress. When the 2006 DoD budget is submitted in February 2005, the
investment strategy will support the November 2004 decision to either
buy a fourth AEHF and potentially slow the TSAT program, or proceed
with TSAT as planned. The rationale and supporting data will be shared
with Congress.
[End of section]
(120225):
FOOTNOTES
[1] The TSAT program also includes development of another satellite,
the Advanced Polar System (APS). Because development efforts for TSAT
and APS are similar, we are referring to both programs as TSAT in this
report. More information about APS is included in enclosure I.
[2] 10 U.S.C. sections 2220 and 2435.
[3] U.S. General Accounting Office, Military Space Operations: Common
Problems and Their Effects on Satellite and Related Acquisitions, GAO-
03-825R (Washington, D.C.: June 2, 2003).
[4] U.S. General Accounting Office, Defense Acquisitions: Improvements
Needed in Space Systems Acquisition Management Policy, GAO-03-1073
(Washington, D.C.: Sept. 15, 2003).
[5] U.S. General Accounting Office, Best Practices: Better Management
of Technology Development Can Improve Weapon System Outcomes, GAO/
NSIAD-99-162 (Washington, D.C.: July 30, 1999); and U.S. General
Accounting Office, Best Practices: Better Matching of Needs and
Resources Will Lead to Better Weapon System Outcomes, GAO-01-288
(Washington, D.C.: Mar. 8, 2001).
[6] DOD Instruction 5000.2, Operation of the Defense Acquisition
System, May 12, 2003.
[7] DOD Directive 5000.1 and Instruction 5000.2.
[8] U.S. General Accounting Office, Military Space Operations: Common
Problems and Their Effects on Satellite and Related Acquisitions, GAO-
03-825R (Washington, D.C.: June 2, 2003).
