Defense Acquisitions
Changes in E-10A Acquisition Strategy Needed before Development Starts Gao ID: GAO-05-273 March 15, 2005The Air Force is on the verge of making a major commitment to the multi-billion dollar E-10A Multi-sensor Command and Control Aircraft program. Due to the substantial investment needed and technological challenges in developing the aircraft, the Subcommittee on Tactical Air and Land Forces asked GAO to examine the soundness of the E-10A business case as well as the risks associated with the current acquisition strategy.
As the E-10A Multi-sensor Command and Control Aircraft program nears its official starting point, questions remain regarding critical elements of its business case, including the need for the aircraft, the maturity level of its technology, and its funding. Plans call for the E-10A to couple a new radar system with a sophisticated and software intensive battle management command and control system aboard a Boeing 767. E-10A is planned to fill a current gap in U.S. capabilities and provide a defense against weapons such as cruise missiles. The Office of the Secretary of Defense is still working on a study to determine whether the E-10A program is the most cost-effective way to fill that gap. E-10A program funding plans changed dramatically in December 2004 when the DOD proposed reducing the total program budget by about 45 percent for the next 2 fiscal years. The business case for starting a development program requires demonstrated evidence that (1) the warfighter need exists and that it can best be met with the chosen concept and (2) the concept can be developed and produced within existing resources--including design knowledge, demonstrated technologies, adequate funding, and adequate time to deliver the product. E-10A requirements and resources are still in flux. GAO found risks associated with the current E-10A acquisition strategy that could lead to costly changes later in the program. The program is set to move into production before critical knowledge is acquired. For example, the first fully assembled E-10A, outfitted with its radar and battle management command and control systems, would not be delivered in time to complete testing before the decision is made to begin production. Testing and production are scheduled to start at the same time in 2010. Furthermore, four of six E-10As are scheduled to begin production before the results of testing are available. By not demonstrating that the system can perform as expected before entering production, the program increases the risk of changes and delays later in the program. This strategy requires significant concurrency among the technology development, product development, and production phases.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-05-273, Defense Acquisitions: Changes in E-10A Acquisition Strategy Needed before Development Starts
This is the accessible text file for GAO report number GAO-05-273
entitled 'Defense Acquisitions: Changes in E-10A Acquisition Strategy
Needed before Development Starts' which was released on March 15, 2005.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Chairman, Subcommittee on Tactical Air and Land Forces,
Committee on Armed Services, House of Representatives:
United States Government Accountability Office:
GAO:
March 2005:
Defense Acquisitions:
Changes in E-10A Acquisition Strategy Needed before Development Starts:
GAO-05-273:
GAO Highlights:
Highlights of GAO-05-273, a report to the Chairman, Subcommittee on
Tactical Air and Land Forces, Committee on Armed Services, House of
Representatives
Why GAO Did This Study:
The Air Force is on the verge of making a major commitment to the multi-
billion dollar E-10A Multi-sensor Command and Control Aircraft program.
Due to the substantial investment needed and technological challenges
in developing the aircraft, the Subcommittee on Tactical Air and Land
Forces asked GAO to examine the soundness of the
E-10A business case as well as the risks associated with the current
acquisition strategy.
What GAO Found:
As the E-10A Multi-sensor Command and Control Aircraft program nears
its official starting point, questions remain regarding critical
elements of its business case, including the need for the aircraft, the
maturity level of its technology, and its funding. Plans call for the E-
10A to couple a new radar system with a sophisticated and software
intensive battle management command and control system aboard a Boeing
767. E-10A is planned to fill a current gap in U.S. capabilities and
provide a defense against weapons such as cruise missiles. The Office
of the Secretary of Defense is still working on a study to determine
whether the E-10A program is the most cost-effective way to fill that
gap. E-10A program funding plans changed dramatically in December 2004
when the DOD proposed reducing the total program budget by about 45
percent for the next 2 fiscal years. The business case for starting a
development program requires demonstrated evidence that (1) the
warfighter need exists and that it can best be met with the chosen
concept and (2) the concept can be developed and produced within
existing resources--including design knowledge, demonstrated
technologies, adequate funding, and adequate time to deliver the
product. E-10A requirements and resources are still in flux.
GAO found risks associated with the current E-10A acquisition strategy
that could lead to costly changes later in the program. The program is
set to move into production before critical knowledge is acquired. For
example, the first fully assembled E-10A, outfitted with its radar and
battle management command and control systems, would not be delivered
in time to complete testing before the decision is made to begin
production. Testing and production are scheduled to start at the same
time in 2010. Furthermore, four of six E-10As are scheduled to begin
production before the results of testing are available. By not
demonstrating that the system can perform as expected before entering
production, the program increases the risk of changes and delays later
in the program. This strategy requires significant concurrency among
the technology development, product development, and production phases.
E-10A Acquisition Strategy:
[See PDF for image]
[End of figure]
What GAO Recommends:
GAO recommends that the Secretary of Defense ensure that open questions
about the E-10A business case are answered before the program advances
into the development and demonstration phase and officially begins. GAO
also recommends that if the E-10A program goes forward adequate time be
allotted to test a prototype before moving into production. DOD
concurred with the first recommendation, but noted that the questions
may not be resolved until the Milestone B decision, which could now be
delayed until 2010. DOD partially concurred with the second
recommendation, stating that DOD policy did not require demonstration
of a design at that point in the acquisition process. GAO disagrees
with this interpretation.
www.gao.gov/cgi-bin/getrpt?GAO-05-273.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Michael Sullivan at (202)
512-4841 or sullivanm@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
Uncertainties Exist in E-10A Business Case as It Approaches Its
Decision Point:
E-10A Acquisition Strategy Does Not Capture Critical Knowledge before
Program Decision Points:
Conclusions:
Recommendations:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Comments from the Department of Defense:
Figures:
Figure 1: Comparison of E-10A Strategy and Best Practices Model:
Figure 2: E-10A Test and Production Schedule:
Abbreviations:
DOD: Department of Defense:
OSD: Office of the Secretary of Defense:
United States Government Accountability Office:
Washington, DC 20548:
March 15, 2005:
The Honorable Curt Weldon:
Chairman:
Subcommittee on Tactical Air and Land Forces:
Committee on Armed Services:
House of Representatives:
Dear Mr. Chairman:
In April 2005, the Air Force plans to make a major commitment to the E-
10A Multi-sensor Command and Control Aircraft (E-10A) program and
officially begin its development and demonstration phase. The program,
estimated to cost over $7.3 billion through fiscal year 2013, is being
designed to fill a gap in U. S. capabilities and provide a defense
against weapons such as cruise missiles. It is also being designed to
be an airborne battle management platform capable of directing forces
to respond to moving targets in the air and on the ground. Given the
technical challenges and significant investment associated with this
aircraft, you requested that we review the E-10A program to determine
whether its business case is clearly defined and its acquisition
process is well executed. This report examines (1) the soundness of the
Air Force's business case[Footnote 1] including E-10A requirements,
technologies, and cost; and (2) the risks associated with the planned
acquisition strategy.
Because the E-10A is not officially a weapons system acquisition
program until the Milestone B[Footnote 2] decision has been approved,
the Air Force considered key elements of the business case as pre-
decisional and subject to change. Therefore, we did not have the
opportunity to review all elements related to the critical E-10A
technology assessments, the cost estimate, or the funding profile. The
Air Force did provide, however, a technology readiness assessment of
the radar subsystem, the planned overall acquisition strategy, and the
analysis of alternatives supporting the selection of the specific E-10A
aircraft platform. Despite these limitations, we were able to assess
the relative state of the E-10A business case as compared to the
expectations of best practices. We did this by drawing on other
information available that indicated the current instability of
estimated cost and funding and that questioned the maturity of the
technology associated with a key subsystem. We conducted this review
from January 2004 to January 2005 in accordance with generally accepted
government auditing standards. More details about our methodology are
in appendix I.
Results in Brief:
Questions remain in the E-10A business case over it being the most cost-
effective solution as well as its technology maturity, cost, and
funding. Officials in the Office of the Secretary of Defense (OSD) are
still studying E-10A alternatives to determine if it is the most cost-
effective way to satisfy the planned mission to identify, track, and
target time-sensitive threats like cruise missiles.[Footnote 3] The
study is also assessing on board command and control needs. Current
plans are for 25 crew to carry out mission requirements and 2 crew to
fly the aircraft. The OSD expects to present the results of the study
by March 2005. We found that radar technologies are ready for system
development, but because the readiness assessment of the battle
management technologies was not finalized, there was no evidence that
these technologies were sufficiently mature. Review organizations
within the Air Force and OSD are examining Air Force assessments of
technology maturity and costs as well as the annual allocation of funds
needed to develop and produce the E-10A. We were not able to review
these assessments because they were not yet complete. Program officials
also stated that the December 2004 reduction of $600 million in E-10A
funding proposed by OSD will require significant changes to the
business case because it would reduce planned funding for the total
program by about 45 percent in fiscal years 2006 and 2007. The Air
Force is determining how this will impact the program schedule and
costs as it will likely cause them to truncate key activities in these
years. An OSD official said the reduction will likely slow development
of the E-10A airframe activities because the available funds will need
to be applied to the radar improvement program that is also supporting
the Global Hawk program.
The Air Force acquisition strategy for acquiring the E-10A calls for
moving through development and into production before critical
knowledge is captured about design, manufacturing, and reliability. For
example, the strategy does not allow for adequate product integration
and prototype demonstration to ensure the design is stable at the
critical design review stage. Additionally, a fully integrated
prototype--a working model of the E-10A with the radar system and
command and control computer system working together--will not be
delivered in time to allow testing prior to the production decision.
Both testing and production are scheduled to concurrently start in
2010. By not demonstrating the system can perform as expected before
entering production, the program risks costly design changes and delays
later in the program. Furthermore, four of six E-10As are scheduled to
begin production before testing is completed. We have found this to be
consistently a high-risk acquisition approach in our past reviews of
Department of Defense (DOD) acquisitions.
We are making recommendations to the Secretary of Defense to ensure
that the open business case questions are answered before starting the
E-10A program. We also recommend that the Secretary direct the Air
Force to revise the acquisition strategy to ensure sufficient time to
integrate and demonstrate the design in flight testing of an E-10A
prototype before moving the program into production. DOD concurred with
our first recommendation, but noted that the open questions will not be
fully resolved until the Milestone B decision, which could be delayed
until 2010 due to recent budget reductions. DOD partially concurred
with our second recommendation. It stated that DOD policy did not
require integration and demonstration of a design prior to critical
design review. We disagree with this interpretation because DOD policy
requires a demonstration of the integrated prototype prior to entering
demonstration. Nevertheless, DOD acknowledged that it is restructuring
the program to demonstrate the key technologies in a prototype prior to
starting system development and demonstration.
Background:
The E-10A program comprises three primary elements: the aircraft,
radar, and battle management command and control subsystem. The
aircraft is a Boeing 767-400ER, the largest 767 variant Boeing makes.
The Air Force has only contracted for one aircraft to date, because a
final decision on the operational platform has not been made. This
aircraft is a commercial product that will be modified for military use
and used as a testbed. At this time there is only 1 unfilled order for
the 400 model in the Boeing assembly line and 25 other unfilled orders
for other smaller 767 models.[Footnote 4] If the Boeing production line
were to close down before the Air Force is positioned to make a
production decision on the E-10A it would have to find an alternative.
Alternatives could include a different aircraft type or model or the
purchase of 767-400ER aircraft from commercial airline companies.
The radar planned for the E-10A began development in 1997 as a response
to the growing concern about cruise missile proliferation. Initially,
it was intended to upgrade the radar on the Joint Surveillance Target
Attack Radar System (Joint STARS). The upgraded radar was to have
advanced sensor technology, providing air-to-air capability for cruise
missile defense and significant increases in ground surveillance
capability. Shortly after the program began development, the Air Force
restructured the program to develop a modular, scalable radar suitable
for use on a variety of airborne platforms. OSD approved the
development of the multiple platform radar in 2003. It is being
designed for inclusion on the Global Hawk and E-10A programs.
The Air Force began evaluating the need to improve its airborne battle
management command and control capabilities in 2002. The planned E-10A
battle management command and control subsystem is software intensive
and intended to enable the E-10A to process and display sensor data
from the radar and eventually from off board sensors so that the
onboard crew can take actions against time sensitive targets. The Air
Force issued a contract in September 2004 to begin preliminary design
efforts for this subsystem.
We have a body of work focused on best practices in product development
and weapon systems acquisition.[Footnote 5] This work has found that
key to success is formulation of a business case that matches product
requirements to available resources--proven technologies, sufficient
engineering capabilities, time, and funding. Several basic factors are
critical to establishing a sound business case for undertaking a new
product development. First, the needs of the party seeking the new
product, the user, must be accurately defined, alternative approaches
to satisfying these needs properly analyzed, and quantities needed for
the chosen system must be well understood. The developed product must
be producible at a cost that matches the users' expectations and
budgetary resources. Finally, the developer must have the resources to
design and deliver the product with the features that the customer
wants when it is needed. If the financial, material and intellectual
resources to develop the product properly are not available,
development does not go forward. Additionally, an evolutionary and
knowledge-based acquisition strategy that captures critical knowledge
before key decision points in the program is needed to execute the
business plan. This calls for a realistic assessment of risks and
costs; doing otherwise undermines the intent of the business case and
invites failure. Ultimately, preserving the business case and attaining
critical knowledge in time for decisions strengthens the ability of
managers to say "no" to pressures to accept high risks or unknowns.
If best practices are not followed, we have found a cascade of negative
effects becomes magnified in the product development and production
phases of an acquisition program. These have led to acquisition
outcomes that included significant cost increases and schedule delays,
poor product quality and reliability, and delays in getting the new
capability to the warfigher. These outcomes have been demonstrated in
other programs such as the F/A-22 fighter, C-17 airlifter, V-22
tiltrotor aircraft, PAC-3 missile, and others.
Uncertainties Exist in E-10A Business Case as It Approaches Its
Decision Point:
Questions remain as the Air Force develops the E-10A program's business
case to support the decision to begin development in April 2005. The
DOD has identified a need for a cruise missile defense capability and
the Air Force has selected the E-10A to meet this need. There are,
however, unanswered questions in both the requirement and resource
elements of the E-10A business case. OSD is still studying whether the
E-10A is the most cost-effective alternative for the cruise missile
requirement and the extent of battle management command and control
needed on board to satisfy the intended need. Finally, assessments of
the technology maturity, estimated costs, and funding availability are
still in process.
OSD Is Still Studying the Need for the E-10A:
OSD officials from the Program Analysis and Evaluation Directorate are
not satisfied that the studies done by the Air Force to select the E-
10A sufficiently analyzed alternative systems.[Footnote 6] As a result,
they are reviewing alternative systems and attempting to determine the
most cost-effective solution to satisfy the warfighter's needs. OSD
officials agree that the E-10A could provide an increased capability in
identifying and tracking ground moving and time-sensitive targets.
However, they believe that if there are less costly systems that can
provide similar capabilities, it could be more cost-effective to buy
those systems. The Air Force began efforts in 1997 to develop a radar
sensor that would detect cruise missiles as part of the Joint STARS
program. The Air Force examined different size and power combinations
for the radar and which platforms had the capacity to carry the radar
and still perform multiple missions. These analyses assumed that only
manned airborne platforms could meet these requirements. The Air Force
completed a formal analysis of alternatives in February 2002 of
different possible host platforms for the radar. The study indicated
that other aircraft could meet many of the requirements but were based
on older commercial technology that was less efficient to operate. The
Air Force analysis concluded that the Boeing 767-400ER was the optimal
choice given the future multi-mission purpose of the system, and the
size, weight, and performance requirements of the radar.
OSD officials are also uncertain about the degree of battle management
command and control capability needed onboard the E-10A versus
transmitting the information gathered by the E-10A to other command and
control centers. According to the Air Force, the need for an onboard
capability is driven by the large amounts of data that would be
collected and analyzed, the limited bandwidth to transmit the data, and
the need to have line-of-sight communications for time-sensitive
targeting, particularly against cruise missiles. OSD officials said
they are looking at whether the battle management subsystem has to be
part of the E-10A platform to meet the timelines identified by the Air
Force. They expect to present their results by March 2005. Air Force
officials told us that some of the battle management functions are
currently performed by ground units, but these ground units cannot
adequately respond to real-time events involving moving targets like
cruise missiles. The E-10A's primary function will be battle management
command and control of cruise missile detection and time-sensitive
targeting activities. As a result, its battle management capabilities
will be tailored to support those functions. These capabilities were
validated in October 2004 by the Joint Requirements Oversight Council
in preparation for the program's upcoming Milestone B decision.
To provide these capabilities, an onboard crew will be required. The
current E-10A crew size is estimated at 27 staff--2 flight crew, 21
mission operators, and 4 technicians. According to the Air Force, the
crew size could change depending on the mission and the degree of
automation on the system. However, the Air Force has not performed any
incremental analysis to show crew size for individual specific
missions, such as doing cruise missile defense only.
Assessments of Needed Resources to Develop and Produce the E-10A Are
Incomplete:
To date, the Air Force has not identified sufficient or available
resources to meet the warfighter's requirements and to start the
development program. The Air Force program office has completed its
assessments of E-10A critical technologies, cost estimates, and funding
needs but these assessments are being reviewed by OSD. While some
resources will meet the requirements, others are either unproven or in
a state of flux. Radar development started under a separate program,
the Radar Technology Insertion Program, and most radar technologies
were reported as mature. Because the Air Force did not provide GAO its
technical assessment of the battle management command and control
system critical technologies, we consider the maturity levels unproven,
even though program officials told us these technologies meet minimum
maturity standards. In addition to technologies, the financial
resources for the program are in a state flux. The E-10A cost estimate
for development and production is still a work in process and funding
was recently reduced by $600 million for fiscal years 2006 and 2007,
which according to DOD officials will substantially impact the program.
Status of Technologies:
Most radar technologies are at a high level of maturity, but evidence
was not provided to support stated maturity levels of the battle
management command and control subsystem. The Air Force assessed radar
technologies prior to the October 2003 start of the Radar Technology
Insertion Program. The critical technologies identified in the radar
improvement program included the radar architecture, modes, receiver/
exciter, and signal processor among others. Of the nine technologies
identified, six were assessed as mature to our best practice standard;
the remaining three were one level below the best practice requirement
for mature technologies, a level DOD policy states is sufficient to
begin development. These three technologies are the pulse compression
unit, the structure, and the modes. Since the 2003 radar technology
assessment, the radar improvement program completed its final design
review in June 2004. Numerous tests have been conducted on small-scale
radar prototypes to mitigate program risks. These tests electronically
drove a signal through the radar, demonstrating the basic functionality
of the design. However, the radar subsystem being designed for the E-
10A has demonstrated neither form nor fit, nor has it been integrated
on the aircraft platform. Although the integration process is an
inherently high-risk endeavor, Air Force officials stated they have a
process in place to manage these risks. The actual size of the E-10A's
radar will be significantly larger than the tested prototype and will
require the E-10A testbed aircraft in order to complete the
demonstration currently scheduled to occur in 2010. The process of
scaling the radar to the appropriate size and ensuring that all the
individual modules work together has yet to be accomplished.
Recognizing this, program officials have identified the integration of
the radar as a critical technology for the E-10A weapon system. The
level of this technology's maturity has not yet been finalized. OSD
officials accepted the Air Force's assessment of the radar technologies
but expect more detailed information on the technologies when the E-10A
weapon system undergoes its Milestone B review in April 2005.
An assessment of the battle management command and control subsystem
technologies was not provided for our review. This subsystem is complex
and software intensive. E-10A program officials told us these
technologies would meet the minimum DOD standard for starting a
program. However, the Air Force only recently directed the contractor
to begin systems engineering efforts to determine a preliminary design
for this subsystem. Development of critical software needed to
demonstrate the technologies has not started. The first increment of
software is not scheduled to be delivered until January 2008. On other
major weapon system development programs, we have found software
development to be a substantial cause for delays in technology
development, system deliveries, and increased costs. Therefore, even
though program officials have stated technologies are sufficiently
mature, we think stronger evidence will be needed to demonstrate their
claim.
Status of Cost Estimate and Funding Needs:
The Air Force has completed its cost estimate for the total E-10A
program and released it to OSD for review. The cost estimates for each
of the three major program elements contain risk. The biggest area of
cost uncertainty is the battle management command and control
subsystem. It is a highly complex software-intensive system. A contract
was issued in September 2004 for about $71 million to begin early
design and engineering efforts to support a preliminary design review
in late 2005. Until this initial design and engineering effort is
completed, the program will not be able to establish high confidence in
its estimated costs. In addition, the aircraft contract only calls for
the delivery of one commercial 767-400ER for testing. To convert this
aircraft to military use, there will be additional costs for installing
communication antennas, a refueling receptacle, hull hardening, and FAA
airworthiness certification. According to the Air Force, these costs
have been factored into its latest program estimate. The initial cost
estimate for the radar program, managed separately from the E-10A
program, has grown. Prior to entering system development, OSD
determined that projected costs were understated and directed the Air
Force to increase its funding by $154 million.
The Air Force acknowledges that funding for the E-10A program is also a
major concern. Funding cuts have delayed its start. It has undergone
two congressional budget reductions; the first cut in fiscal year 2003
($343 million) required a significant program replanning effort. The
second cut in fiscal year 2005 ($115 million) resulted in schedule
delays for the planned test program, system integration lab, testbed
aircraft delivery, and the E-10A's first flight. The Air Force states
these cuts have caused the planned initial operating capability date to
slip 3 years to 2015. A third cut, recently proposed by OSD in December
2004, reduces the program's budget request by $300 million in both
fiscal year 2006 and 2007--a total reduction of $600 million. The
program office is in the process of evaluating the impact of these
reductions and officials indicated that because these represent a
reduction of about 45 percent in each year, they will have a
significant impact on the program if they are sustained. OSD officials
indicated that efforts related to aircraft development and the delivery
of the test aircraft will likely bear the bulk of the reductions. This
will have an impact on planned program milestones. They said it was
important to keep the radar program funded because it is developing the
radar planned for the new Global Hawk unmanned aerial vehicle in
addition to the E-10A.
E-10A Acquisition Strategy Does Not Capture Critical Knowledge before
Program Decision Points:
The E-10A acquisition strategy raises concern as key decisions are
planned before critical product knowledge is available. For example,
the strategy for developing the first E-10A increment does not allow
for adequate integration or prototype demonstration to ensure the
design is stable at the system critical design review. System
integration allows program officials to measure the stability of a
product's design and its ability to meet established requirements. Both
commercial companies and DOD recognize the attainment of this knowledge
as being demonstrated by the completion of most engineering drawings
and some demonstration of the system level capabilities in a prototype.
A stable design that meets requirements should be achieved by critical
design review, before system demonstration and initial manufacturing of
production representative products begins. However, the Air Force does
not expect to deliver the battle management command and control and
radar subsystems to the integration laboratory until 2008 and 2009,
after critical design review, scheduled in 2007. The transition of the
battle management command and control and radar subsystems from the
integration lab to the 767-400ER test airframe is not scheduled to
begin until late-2009, nearly 2 years after the critical design review
and only a few months prior to the program's production commitment
decision. As a result, critical knowledge about the basic performance
of key subsystems integrated into an actual E-10A prototype will not
occur until 2010 (see fig. 1).
Figure 1: Comparison of E-10A Strategy and Best Practices Model:
[See PDF for image]
Note: In the above figure, technology development is extended into
production because there are several technologies that cannot be
assessed as mature until they are actually integrated and flight tested.
[End of figure]
Additionally, the fully integrated E-10A prototype will not be
available for testing prior to the scheduled decision to begin
production. This strategy requires significant concurrency among the
technology development, product development, and production phases and
places decision makers at a disadvantage by not knowing if the E-10A
can demonstrate it meets system performance and reliability
requirements before transitioning into production. In fact, the results
of operational testing are not scheduled to be available until four of
the six planned E-10As are already in production in 2011, greatly
increasing the risks of costly design changes and schedule delays later
in the program (see fig. 2). Our past reviews have found this to be a
high-risk acquisition approach.[Footnote 7]
Figure 2: E-10A Test and Production Schedule:
[See PDF for image]
Note: The schedule depicted above is based on the program plan in
effect at the time of our review. Since that time, the program's budget
has been reduced, necessitating revisions to this schedule. We have not
had the opportunity to review the new program plan.
[End of figure]
The Air Force is planning to use an incremental approach to achieve the
E-10A's full capability with each subsequent increment adding
capability. Although an incremental approach can reduce risks, the
failure to capture critical knowledge while developing the first
increment will likely reduce the benefits of such an approach. As
currently planned there will be four distinct E-10A increments. Program
officials are planning to conduct major program decision reviews prior
to beginning development and demonstration of each increment. This
approach, if implemented as planned, will provide decision makers with
an opportunity to review the program's progress and risk before making
further investment decisions thus reducing risk in the program. The
first increment is expected to provide the users with many of the
system's basic required capabilities.[Footnote 8] Those capabilities
include cruise missile defense and on-board command and control
capability for processing, displaying, and communicating the data
needed to address time-sensitive targets. Subsequent increments will
enhance the system's capabilities, moving them closer to objective
levels by increasing the amount of data processing and analysis done by
computers and decreasing the amount done by human analysts with
computer assistance, thus shortening the time it takes to make
decisions. However, if the first increment falters, the Air Force will
likely spend increasing amounts of time and money to achieve this
initial capability, thereby delaying subsequent increments.
Conclusions:
The current conditions surrounding the development of the E-10A
business case portend the potential for poor outcomes if requirement,
resource, and acquisition strategy deficiencies are not resolved before
system development and demonstration begins. The decision to start a
major weapon systems acquisition program for the E-10A requires an
executable business case that demonstrates the E-10A is the best way to
satisfy the gap in warfighter's capability and that the concept can be
developed and produced within existing resources. An evolutionary and
knowledge-based acquisition strategy is needed to ensure this business
case can be executed within planned goals. The Air Force and OSD are
still determining if a sound business case exists. Questions still
surrounding the business case include:
* Is the E-10A the most cost-effective alternative?
* How extensive of a battle management command and control capability
is needed?
* Are technologies at a high level of maturity?
* Is there sufficient funding to develop and deliver the capability in
time?
The acquisition strategy also fails to capture critical design,
manufacturing, and reliability data in time to make investment
decisions for moving the program through the development program into
production. The gaps in knowledge increase the likelihood that the Air
Force will not be able to deliver on the cost, schedule, and
performance goals in its business case.
Recommendations for Executive Action:
Because gaps exist in the information needed to make a sound business
case to start a major acquisition program, we recommend that the
Secretary of Defense ensure that the open business case questions are
answered before a decision is made to start the E-10A program.
Additionally, to ensure a greater likelihood of success, if the E-10A
program is approved to begin, we recommend the Secretary direct the Air
Force to revise the acquisition strategy to ensure sufficient time is
included in the schedule to (1) integrate and demonstrate the design
before moving past the critical design review and (2) test a production
representative E-10A prototype before starting production.
Agency Comments and Our Evaluation:
DOD provided us with written comments on a draft of this report. The
comments appear in appendix II.
DOD concurred with our recommendation that the Secretary ensure that
the open business case questions are answered before a decision is made
to start the E-10A program. DOD provided some information on the
current status of these questions and implies that some of the business
case questions had been answered. We believe that until the OSD/Program
Analysis and Evaluation study is completed and final results are
provided to OSD acquisition decision makers, the business case
questions remain open.
DOD partially concurred with our recommendation that the Secretary
direct the Secretary of the Air Force to revise the E-10A acquisition
strategy to ensure sufficient time is available to (1) integrate and
demonstrate the design before moving past the critical design review
and (2) test a production representative E-10A before starting
production.
Regarding (1), DOD stated that OSD policy does not require the
integration and demonstration of a design before critical design
review. We disagree. Section E1.1.14 of Department of Defense Directive
5000.1, The Defense Acquisition System, states that "PMs—shall reduce
integration risk and demonstrate product design prior to the design
readiness review." DOD's design readiness review is required to end the
system integration phase of system development and demonstration.
Additionally, DOD's entrance criterion for the demonstration phase
requires a demonstration of the integrated product in a prototype.
Nonetheless, DOD stated that it is restructuring the program with the
goal of demonstrating the radar and battle management technologies in a
prototype before starting systems development and demonstration. This
approach incorporates the knowledge-based approach inherent in
commercial best practices and endorsed by DOD policy. In its comments,
DOD acknowledges that this approach will increase confidence in the
program's cost estimate and allow time to evaluate the aircraft
platform.
Regarding (2), DOD stated that the Milestone C production decision for
low rate initial production decision will be based on the initial test
results from a representation E-10A aircraft system. While the program
schedule in effect at the time of our review did not indicate this, we
believe this approach is more consistent with a knowledge based
acquisition strategy. By testing a production representative aircraft
prior to committing to production, DOD will be able to reduce program
risks and make informed decisions based on actual system capabilities
and performance information.
DOD also provided technical comments to our report. We made changes
where appropriate but many of these comments were based on a new
acquisition strategy that plans to delay the E-10A program Milestone B
decision until 2010. We did not make DOD's recommended changes to the
report that reflected this new schedule because it has not been
approved and we have not had the opportunity to review it.
We are sending copies of this report to the Secretary of Defense and
the Secretaries of the Air Force, the Army, and the Navy. We will also
provide copies to others on request. In addition, the report will be
available at no charge on the GAO Website at http://www.gao.gov.
Please contact me at (202) 512-4841 if you have any questions
concerning this report. Other key contributors to this report were
Martin Campbell, Michael Hazard, Travis Masters, Rae Ann Sapp, David
Schilling and John Krump.
Sincerely yours,
Signed by:
Michael J. Sullivan:
Director:
Acquisition and Sourcing Management:
[End of section]
Appendix I: Scope and Methodology:
During our review we discussed the E-10A program with officials from
the following organizations in the Office of the Secretary of Defense,
Undersecretary of Defense for Acquisition Technology and Logistics; the
Director, Defense Systems/Developmental Test and Evaluation; the
Director, Operational Test and Evaluation; the Director, Defense
Research and Engineering; and the Director, Program Analysis and
Evaluation. We also discussed the E-10A with the technical director of
the Joint Theater Air Missile Defense Organization. In addition, we
discussed the program with officials from several organizations in the
Air Force. These officials included representatives from the
Information Dominance Directorate with the Office of the Assistant
Secretary for Acquisition; the Directorate of Operational Requirements;
the Command Control Communications Intelligence and Reconnaissance
Center at Langley Air Force Base; the Electronic Systems Center at
Hanscom Air Force Base; and the Aeronautical Systems Center at Wright
Patterson Air Force Base.
To determine the progress the Air Force had made in developing the
business case for the E-10A, we obtained available information on the
system's requirements and resources. However, the information we
received on resources such as technology maturity, cost, funding,
quantities, and schedule was limited. We discussed this information
with knowledgeable program office and oversight officials. We also
contacted officials studying force structure issues that could impact
the requirements for the E-10A program. To assess the validity of the
proposed business case, we compared the E-10A information with best
commercial practices and DOD policy guidance for new development
programs. Because the E-10A program has not yet been approved to enter
system development and demonstration, specific information on the
system's technology readiness assessment and total program cost and
funding were not available. As a result, we could not conduct a
detailed assessment of these elements of the business case. However,
because of other related information, such as the status of the
software intensive battle management command and control subsystem, the
significant reduction in funding for fiscal years 2006 and 2007, and
the ongoing studies to answer OSD concerns, we were able to conclude
that at the time of our review key business case elements were still
not mature enough to begin product development. For example, complex
and software intensive subsystems in other programs have caused major
problems that have delayed achieving technology maturity and the Air
Force has only recently directed the contractor to begin early systems
engineering effort to determine a preliminary design for the E-10A
battle management subsystem. Additionally, the $600 million reduction
in funding planned for the first 2 years will almost certainly require
the program to extend its planned schedule resulting in additional
costs and funding requirements not yet estimated. These are business
case elements that need to be firmly established before entering the
upcoming Milestone B decision point.
To determine the soundness of the E-10A's acquisition strategy, we
obtained available information on the program's original and revised
acquisition plans from the program office and discussed it with
functional oversight and program officials. In addition, we compared
the E-10A's planned strategy to best commercial practices and DOD's
knowledge-based acquisition policy. However, since our analysis, the
program's budget request was reduced by a total of $600 million in
fiscal years 2006 and 2007.
We conducted our review from January 2004 to January 2005 in accordance
with generally accepted government auditing standards.
[End of section]
Appendix II: Comments from the Department of Defense:
OFFICE OF THE UNDER SECRETARY OF DEFENSE:
ACQUISITION, TECHNOLOGY AND LOGISTICS:
3000 DEFENSE PENTAGON:
WASHINGTON, DC 20301-3000:
MAR 09 2005:
Mr. Michael J. Sullivan:
Director, Acquisition and Sourcing Management:
Government Accountability Office:
Washington, D.C. 20548:
Dear Mr. Sullivan:
This is the Department of Defense (DoD) response to the Government
Accountability Office (GAO) Draft Report, dated February 4, 2005,
"Defense Acquisitions: Changes in E-10A Acquisition Strategy Needed
Before Development Starts" (GAO code 120314/GAO-05-273).
The DoD concurs with the draft report's first recommendation and
partially concurs with the second. The rationale for the DoD's position
is provided at enclosure 1. Recommended changes to enhance clarity and
correctness are at enclosure 2.
Sincerely,
Signed by:
Glenn F. Lamartin:
Director:
Defense Systems:
Enclosures:
1. DoD Comments to the GAO Recommendations.
2. Recommended Changes:
Enclosure 1:
GAO DRAFT REPORT - DATED FEBRUARY 4, 2005:
GAO CODE 120314/GAO-05-273:
"DEFENSE ACQUISITIONS: CHANGES IN E-10A ACQUISITION STRATEGY NEEDED
BEFORE DEVELOPMENT STARTS"
DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATIONS:
RECOMMENDATION 1: The GAO recommended that the Secretary of Defense
ensure that the open business case questions are answered before a
decision is made to start the E-10A program. (p. 13 & 14/GAO Draft
Report):
DOD RESPONSE: Concur. The GAO conducted the Business Case Analysis from
January 2004 to January 2005. During this period, the E-10A Systems
Group was drafting the required statutory and regulatory documents to
support a Milestone B decision, which is considered the initiation of a
major defense systems acquisition program. The draft documents were at
the level of maturity expected leading up to a milestone, and the
program was on schedule to meet the Milestone B Defense Acquisition
Board in March 2005. During the Business Case Analysis, the E-10A
Systems Group discussed the content of the draft documents with the GAO
and agreed to provide the final documents once they were approved.
However, since the draft documents were not fully coordinated through
Air Force and OSD channels when the GAO study concluded in January
2005, final documents were never released to the GAO.
Current status of answers to the open questions is as follows:
1) Is the E-10A the most cost effective alternative? Yes; based on an
Analysis of Alternatives (AoA) completed in March 2002. The conclusions
were accepted by the OSD Overarching Integrated Product Team. That
said, additional information is expected from the results of an OSD/
Program Analysis and Evaluation study that was directed in the Multi-
Platform Radar Technology Insertion Program (MP-RTIP) Acquisition
Decision Memorandum (ADM).
2) How extensive of a battle management command and control capability
(BMC2) is needed? The BMC2 requirement is defined in the E-10A
Capabilities Development Document (CDD) approved by the Joint
Requirements Oversight Council (JROC) in November 2004. Supporting
analysis was presented at the BMC2 Analysis Meeting in September 2004.
This justification is being considered along with on-going Office of
the Secretary of Defense (OSD) analysis to determine the amount of
required E-10A onboard BMC2 capability.
3) Are technologies at a high level of maturity? Technology maturity is
addressed in the completed E-10A Technology Readiness Assessment (TRA).
Technology maturity varies with the subsystem. Of 25 critical
technologies, 9 are at technology readiness level (TRL) 6 and 16 are at
TRL level 7 or higher. This meets requirements, or exceeds preferred
levels, for entering the System Design and Development phase. A recent
Department systems engineering evaluation of the combined E-10A and
radar program revealed sound system-level planning and effective
processes. The Department agrees, however, that system integration of
the radar subsystem is inherently high risk and is working to mitigate
that risk.
4) Is there sufficient funding to develop and deliver the capability in
time? The current OSD policy is for programs to be fully funded at
Milestone B. The Air Force has consistently budgeted the E-10A program
to meet its agreed initial operational capability (IOC) date; however,
the Department accepts that the $600 million proposed in PB06 for
reduction in FY06-07 will slip the IOC from the original schedule.
The program is being restructured in response to the PB06 funding
reductions. Milestone B might not occur until 2010. Since the answers
to the GAO's open business case questions can be found in Milestone B
documents that are required by statute and regulation, that information
will be available at the restructured program's Milestone B. The Air
Force will seek formal approval of its restructure plan in June 2005.
RECOMMENDATION 2: The GAO recommended that the Secretary of Defense
direct the Secretary of the Air Force to revise the acquisition
strategy to ensure sufficient time is included in the schedule to: (1)
integrate and demonstrate the design before moving past the critical
design review, and (2) test a production representative E-10A prototype
before starting production. (p. 14/GAO Draft Report):
DOD RESPONSE: Partially Concur. A restructured plan is under review.
For (1), while OSD policy does not require the integration and
demonstration of a design before moving past the critical design review
as recommended by the GAO, the Air Force is restructuring the program
with the goal of demonstrating the MP-RTIP radar and key BMC2
technologies in a prototype prior to starting System Development and
Demonstration. This approach will also increase confidence in program
cost estimates and allow time to evaluate the aircraft platform
aspects. With respect to (2), the Milestone C production decision for
low rate initial production will be based on the initial test results
of a representation aircraft E-10A system.
[End of section]
FOOTNOTES
[1] The business case is defined as demonstrated evidence that (1) the
warfighter need exists and that it can best be met with the chosen
concept, and (2) the concept can be developed and produced within
existing resources--including design knowledge, demonstrated
technologies, adequate funding, and adequate time to deliver the
product.
[2] Milestone B is the entrance point for the system development and
demonstration phase, which is considered the initiation of a major
defense systems acquisition program. It requires authorization by the
Milestone Decision Authority based on advice from senior-level advisors
on the Defense Acquisition Board. The Defense Acquisition Board meeting
to discuss the E-10A Milestone B is scheduled for April 12, 2005.
Department of Defense Instruction 5000.2, Operation of the Defense
Acquisition System, Paragraph 3.7, System Development and
Demonstration, Subparagraphs 3.7.1.2 and 3.7.2.3.
[3] A time-sensitive target is a target of such high priority that it
requires an immediate response, either because it poses a danger to
friendly forces or it is a highly lucrative, fleeting target of
opportunity.
[4] Source: Boeing Website, http://www.boeing.com.
[5] GAO, Defense Acquisitions: DOD's Assessments of Major Weapon
Programs, GAO-04-248 (Washington, D.C.: Mar. 31, 2004). Other recent
reports discussing best practices include GAO, Best Practices:
Capturing Design and Manufacturing Knowledge Early Improves Acquisition
Outcomes, GAO-02-701 (Washington, D.C.: July 15, 2002) and Defense
Acquisitions: DOD Faces Challenges in Implementing Best Practices, GAO-
02-469T (Washington, D.C.: Feb. 27, 2002).
[6] Both the Air Force and the Joint Theater Air and Missile Defense
Organization conducted analyses to assess future mixtures of systems,
including the E-10A. Both studies determined that if the E-10A performs
as expected it could significantly increase the capability of the
warfighters. However, the Air Force study focused only on ground-moving
target indication and the Joint Theater Air and Missile Defense
Organization focused primarily on cruise missile defense.
[7] GAO, Best Practices: Capturing Design and Manufacturing Knowledge
Early Improves Acquisition Outcomes, GAO-02-701 (Washington, D.C; July
15, 2002).
[8] The E-10A Capabilities Development Document lists 157 separate
capabilities/ requirements to be achieved by increment 1.3. Of those
requirements, 50 will be upgraded or changed as subsequent increments
are developed. Thus the remaining 107 (68 percent) will be met by
increment 1.0. The individual capabilities/requirements were not
weighted in this analysis.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
