Missile Defense
Knowledge-Based Process Would Benefit Airborne Laser Decision-Making Gao ID: GAO-02-949T July 16, 2002The Air Force launched an acquisition program to develop and produce a revolutionary laser weapon system, known as the Airborne Laser, in 1996. Being developed for installation in a modified Boeing 747 aircraft, it is intended to destroy enemy ballistic missiles almost immediately after their launch. The Air Force originally estimated development costs at $2.5 billion and projected fielding of the system in 2006. However, by August 2001, the Air Force determined that the development cost estimate rose 50 percent to $3.7 billion, and the fielding date slipped to 2010. The Department of Defense transferred responsibility for the Airborne Laser in October 2001 to the Ballistic Missile Defense Organization. Subsequently, the Defense Secretary designated the Ballistic Missile Defense Organization as the Missile Defense Agency and granted the agency expanded responsibility and authority. The Air Force was unable to meet the Airborne Laser's original cost and schedule goals because it did not fully understand the level of effort that would be required to develop the critical system technology needed to meet the user's requirements. The Missile Defense Agency's new strategy for developing the Airborne Laser incorporates some knowledge-based practices that characterize successful programs. However, the agency has not established knowledge-based decision points and associated criteria for moving forward from technology development to product development and on to production. Without decision points and criteria, the agency risks beginning new and more costly activities before it has the knowledge to determine the money and time required to complete them and whether additional investment in those activities is warranted. This testimony summarizes a July report (see GAO-02-631).
GAO-02-949T, Missile Defense: Knowledge-Based Process Would Benefit Airborne Laser Decision-Making
This is the accessible text file for GAO report number GAO-02-949T
entitled 'Missile Defense: Knowledge-Based Process Would Benefit
Airborne Laser Decision-Making' which was released on July 16, 2002.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products‘ accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
Testimony:
Before the Subcommittee on National Security, Veterans‘ Affairs, and
International Relations, Committee on Government Reform, House of
Representatives:
United States General Accounting Office:
GAO:
For Release on Delivery Expected at 10:00 a.m., EDT, Tuesday, July 16,
2002:
MISSILE DEFENSE:
Knowledge-Based Process Would Benefit Airborne Laser Decision-Making:
Statement of Robert E. Levin, Director, Acquisition and Sourcing
Management:
Technology Development:
GAO-02-949T:
Mr. Chairman and Members of the Subcommittee:
Thank you for inviting me. I am pleased to be here today to discuss the
Missile Defense Agency‘s strategy for acquiring ballistic missile
defense systems. Pursuant to your request, Mr. Chairman, we issued a
report on July 12, 2002, that examined whether the Missile Defense
Agency‘s new strategy for developing the Airborne Laser includes
practices that are characteristic of successful developmental
programs.[Footnote 1] Our observations today will reflect the knowledge
that we gained from that work.
If the Department of Defense is successful in its efforts to develop
and field the Airborne Laser, it could be an important system because
it would give the United States some capability to destroy enemy
missiles over the enemy‘s own territory. Enemy states that could launch
ballistic missiles with chemical or biological warheads toward the
United States, its deployed troops, or allies could be deterred from
doing so if that enemy knew that the chemical and biological agents
might fall back to earth over its own territory.
When the Air Force launched the Airborne Laser program in 1996, it
estimated that developing the system would cost $2.5 billion and that
it would be fielded by 2006. However, by August 2001, the Air Force
determined that maturing the technologies and developing the system
would cost $3.7 billion, or about 50 percent more, and the system would
not be fielded until 2010, 4 years later than originally planned.
Against this backdrop of cost increases and schedule delays, the
Department of Defense, in October 2001, transferred responsibility for
the Airborne Laser from the Air Force to the Ballistic Missile Defense
Organization, now known as the Missile Defense Agency.
Because of your interest in the agencyís recently expanded
responsibility and authority for acquiring a capability to protect
against enemy ballistic missiles, you requested that today we:
* describe the Secretary of Defenseís specific direction to the agency;
* explain the agencyís requirements-setting process and plans for
testing;
* provide details on the agencyís investment decision-making process;
* explain how the maturity of technology critical to the systemís
design will be measured;
* comment on the role of the Director, Operational Test and Evaluation;
and:
* remark on changes in the agencyís test plans since Antiballistic
Missile Treaty restrictions are no longer in place.
To address our objectives, we reviewed documents and held discussions
with officials at the Boeing Company, Seattle, Washington; Lockheed
Martin, Sunnyvale, California; TRW, Los Angeles, California; Airborne
Laser Program Office, Kirtland Air Force Base, New Mexico; Missile
Defense Agency, Arlington, Virginia; and the Office of the Director,
Operational Test and Evaluation, Arlington, Virginia. We conducted our
review from August 2001 through July 2002 in accordance with generally
accepted government auditing standards.
In summary Mr. Chairman, we found the following:
* In January 2002, the Secretary of Defense directed the Missile
Defense Agency to quickly develop a ballistic missile defense system
that included various elements, such as the Airborne Laser. These
elements are to work together to defend the United States, its deployed
troops, allies, and friends by engaging enemy ballistic missiles at
various points during their flight. The Secretary also directed the
agency to cancel existing requirements documents and, instead, develop
elements with currently available technology that would be capable of
defeating some, if not all, of the threat. This capability could then
be improved over time.
* In response to this direction, the agency adopted changes that are
characteristic of successful development programs. First, the agency
adopted a flexible requirements-setting process that allows it to
refine system requirements based on the results of system engineering.
This process can result in less risk of cost and schedule growth
because it does not establish requirements until systems engineering
shows what is achievable. However, the agency must set the Airborne
Laser‘s requirements when it determines that it has a match between the
technology, money, and time needed to design and demonstrate an
operational system so that the agency can use those requirements to
measure the progress being made during product development. The agency
is also implementing other changes to improve Airborne Laser‘s
development. For example, the agency is allowing more time to mature
and test technologies critical to the system‘s design, and it is
improving ground test facilities so that in the future improved
components can be tested on the ground before being installed in an
aircraft.
* The agency has not yet implemented another practice that has proven
beneficial in successful development programs and that we recommended
to the agency in our July 12 report. Successful developers make
decisions on whether to proceed with a program when the focus and cost
of their activities are about to change. The criteria for beginning new
activities is a determination by the developer that the program has the
knowledge to begin the activities and that investing in them is the
best use of the developer‘s resources. Our work over the years has
found that the focus and cost of activities will change when the
program is ready to move from technology development to system
integration, then again into system demonstration, and finally on into
production. Although, the Missile Defense Agency does not currently
make decisions at each of these points or use the same criteria proven
by successful developers, the agency is considering how to implement
such a process.
* The Airborne Laser Program Office used technology readiness levels to
assess the maturity of the technology critical to the design of the
Airborne Laser. Our work shows that developers greatly enhanced their
likelihood of success by beginning development only when they had
demonstrated a system prototype in an operational environment. The
program office‘s assessment showed that some Airborne Laser technology
is almost to this point, but technology such as the mirrors and windows
that focus and control the laser beam and allow it to pass safely
through the aircraft (collectively referred to as optics) require
additional engineering work before reaching this stage of maturity.
Except for its evaluation of the laser, we agreed with the program
office‘s technology assessment. We believe further testing is needed to
demonstrate the maturity of the laser technology because the program
has only tested a one-module laser (rather than the six-module laser
planned for the first Airborne Laser configuration) in a controlled
laboratory environment using surrogate components.
* By law, the Director of Operational Test and Evaluation (OT&E) is
responsible for operational rather than developmental testing. However,
OT&E officials said that being involved in developmental tests provides
them insight and understanding to prepare for live-fire testing and
later operational testing and evaluation. The officials told us that
they have primarily been working on issues related to future live-fire
lethality tests of the Airborne Laser, but they expect their
involvement in Airborne Laser‘s developmental tests will increase when
system-level flight tests begin. The OT&E Director has not been given
authority to approve Airborne Laser‘s developmental test and evaluation
master plan, but OT&E officials have been asked to assist agency
officials in preparing the plan and will provide comments before the
plan is finalized.
* The Airborne Laser Program Office said that the United States‘
withdrawal from the Antiballistic Missile Treaty has not changed its
plan to initially test the first Airborne Laser configuration against a
short-range ballistic missile. However, the agency is considering
future tests with longer-range missile targets. Such tests, which would
have been restricted by the treaty, could occur in fiscal year 2005 or
2006.
Defense Secretary Directs New Approach for Acquiring and Deploying
Missile Defenses:
In 2001, the Department of Defense conducted missile defense reviews to
determine how to best fulfill the nation‘s need to defend against enemy
ballistic missile attacks. As a result of these studies, the Secretary
of Defense in January 2002 directed that the Ballistic Missile Defense
Organization be elevated to the status of an agency and renamed as the
Missile Defense Agency. The Secretary‘s key priorities were to field
quickly an integrated missile defense system that could defend the
United States, deployed forces, allies, and friends by engaging enemy
ballistic missiles at various points during their flight. Another of
the Secretary‘s priorities was to provide an early capability by using
test assets or prototypes in the event of an emergency. To enable the
agency to achieve his priorities, the Secretary directed the agency to
abandon its traditional requirements-setting process that required a
military service to establish technical requirements when a weapon
system acquisition program was launched and adopt a more flexible,
capability-based process that would allow the agency to use available
technology to develop a weapon system that could engage some, if not
all, of the current threat. This ’base-line“ capability would then be
improved over time. The Airborne Laser system was one of many systems
affected by these changes.
Agency Adopts New Requirements-Setting Process and Other Practices:
In response to the Secretary‘s direction, the Missile Defense Agency
adopted a new Airborne Laser development strategy that incorporates
some of the practices characteristic of successful programs. These
practices include a more flexible requirements-setting process,
allowing more time to mature and test the Airborne Laser‘s critical
technologies, and improving test facilities.
Our work shows that the flexible requirements-setting process can
result in less risk of cost and schedule growth because requirements
are not set until systems engineering shows what is achievable.
However, the agency must set the Airborne Laser‘s requirements once it
determines that it has a match between the technology, money, and time
needed to design and demonstrate an operational system so that the
agency can use those requirements to measure the progress being made
during product development.
The Department of Defense ordinarily faces significant hurdles in
matching requirements to available resources (time, technology, and
money). The fundamental problem is two-fold. First, under the
Department‘s traditional process, requirements must be set before a
program can be approved and a program must be approved before the
product developer conducts systems engineering. Second, the competition
for funding encourages requirements that will make the desired weapon
system stand out from others. Consequently, many of the Department‘s
product development programs include unrealistic requirements set by
the user before the product developer has conducted the system
engineering necessary to identify the time, technology, and money
necessary to develop a product capable of meeting requirements.
The agency is also adopting other practices that are likely to improve
Airborne Laser‘s development. These practices include allowing more
time for testing and developing facilities to mature and test critical
technologies. The agency is initially developing and testing a six-
module laser system to demonstrate technologies critical to the
Airborne Laser‘s design. When the Air Force was responsible for the
Airborne Laser program, it planned to complete system-level flight
tests of the six-module Airborne Laser system in the last quarter of
fiscal year 2003, but the agency has delayed completion of the test to
the first quarter of fiscal year 2005. This delay allows additional
time to learn from and correct problems discovered during the tests. In
addition, the agency plans to increase the Airborne Laser‘s ground-
testing capability by awarding a contract in 2003 for what they are
calling an ’iron bird,“ which is essentially an aircraft hull with
laser equipment installed. The ’iron bird“ is expected to allow testing
of a fully integrated Airborne Laser system on the ground so that
technologies for future configurations can be evaluated before being
installed in an aircraft.
Allowing more time for testing is important because testing informs the
requirements process. Because testing allows developers to gauge the
progress being made in translating an idea into a weapon system, it
enables the developer to make a more informed decision as to whether a
technology is ready to be incorporated into a system‘s design. With
this knowledge, the developer can determine whether the technology is
so important to the system‘s design that additional time and money
should be spent to mature the technology or whether the system‘s
initial performance requirements should be reduced.
The ’iron bird“ is expected to reduce the cost of testing technologies
planned for future Airborne Laser configurations. With it, the agency
can mature new component-level technologies to higher levels in the
less expensive ground-testing environment before installing them on an
aircraft.
Changes in Agency‘s Decision-Making Process Are Needed:
We reported in July 2002, that the Missile Defense Agency‘s new
Airborne Laser acquisition strategy does not include decision points
with appropriate knowledge-based criteria for moving the Airborne Laser
program forward. However, the agency is now considering how it can
implement such a process.
In successful developments, developers make decisions when the focus
and cost of program activities is about to change. At these points, the
developers decide whether they have the knowledge to begin new
activities and whether investing further time and money in their
product is the best use of their resources. The first decision point
occurs when the focus of the developer‘s activities change from
technology development to system integration. This point is reached
when the developer has incorporated technology into subcomponents with
the form, fit, and function needed in an operational system and the
developer is ready to design a system that integrates those
subcomponents. The criterion for deciding to move the program forward
is having the knowledge to match requirements and available resources
(time, technology, and funds). The second decision point occurs between
system integration and system demonstration when the developer has
successfully integrated subsystems and components into a design that
not only meets the customer‘s performance requirements but also has
optimized the design for reproducibility, maintainability, and
reliability. A developer moves the program forward at this point only
if the design is stable, which is generally considered to be the point
at which about 90 percent of the design‘s engineering drawings have
been released. The third decision point separates system demonstration
from production. The decision to invest in production is based on
having the knowledge to determine that the product performs as required
during testing and that the manufacturing processes will produce a
product within cost, schedule, and quality targets.
Product Development:
Production:
Knowledge Point 1: Needs match resources:
Knowledge Point 2: Design is stable:
Knowledge Point 3: Production can meet cost, schedule, and quality
targets:
System Integration:
System Demonstration n:
Program Launch:
Figure 1: The Knowledge-Based Process:
Decisions are made at these points not only because the focus of
activities is changing, but also because the cost of the activities are
increasing. Our work shows that product development is typically much
more costly than technology development. This is because during
technology development, small teams of technologists work to perfect
the application of scientific knowledge to a practical problem. As
product development begins, developers begin to make larger investments
in human capital, bringing on a large engineering force to design and
manufacture the product. In addition, product development requires
significant investments in facilities and materials. These investments
increase continuously as the product approaches the point of
manufacture.
The Airborne Laser acquisition process has three phases separated by
two decision points. The phases are development, transition, and
procurement. The development phase includes all developmental
activities and system level demonstrations of military utility.
Transition includes preparation of the operational requirements
document by the appropriate armed service and operational testing; and
production includes producing and fielding the weapon system. The first
decision point occurs between development and transition. A decision
will be made to begin the transition phase if the agency determines
that it has the technology in-hand to produce a system that merits
fielding. At the end of the transition phase, the Airborne Laser would
enter the formal Department of Defense acquisition process at Milestone
C--the point at which the Department decides whether a system should
begin low-rate initial production.
The Missile Defense Agency‘s current decision-making process puts at
risk the agency‘s ability to develop a useful military capability on
time and within budget. This is because the agency‘s process does not
include an established set of decision points with appropriate
knowledge-based criteria for deciding whether to invest in system
integration and, subsequently, system demonstration and production. For
example, the agency does not separate technology development from
system integration with a decision point or use the knowledge the
program has attained to determine if the technology can be incorporated
into a mature system design within available time and funding
constraints. Agency officials are considering how to fit such decision
points into Airborne Laser‘s acquisition process, as well as the
acquisition process of other elements.
Technology Readiness Levels Are Used to Measure Technology Maturity:
In 2002, the Airborne Laser Program Office assessed the maturity of
technologies critical to the development of the Airborne Laser system.
To make this assessment, the program office used a tool known as
technology readiness levels that was developed by the National
Aeronautical and Space Administration. A comparison of a 1999
assessment and the 2002 assessment shows that the Airborne Laser
program has made progress in maturing critical technologies, but much
remains to be done.
In 1996, the Air Force launched the Airborne Laser program to develop a
defensive system that could destroy enemy missiles from a distance of
several hundred kilometers. Engineers concluded that if they were to
meet this requirement, the system would need a fourteen-module oxygen
iodine laser. They also determined that the system would need a beam
control/fire control assembly that could (1) safely move the laser beam
through the aircraft, (2) shape the beam so that it would not be
scattered or weakened by the atmosphere, and (3) hold the beam on
target despite the movement of the aircraft. In addition, engineers
determined that the system would need a battle management and control
system capable of planning and executing an engagement. To determine if
the technology was ’in hand“ to meet this requirement, the Air Force
planned to build a six-module Airborne Laser configuration and test it
against a short-range ballistic missile. Under the Missile Defense
Agency‘s new development strategy, this six-module configuration is now
known as Block 2004.[Footnote 2]
Officials began their 2002 assessment by determining the technologies
critical to designing Block 2004 and future configurations. These
technologies are: (1) devices that stabilize the laser system aboard
the aircraft so that the beam can be maintained firmly on the target;
(2) optics--mirrors and windows--that focus and control the laser beam
and allow it to pass safely through the aircraft; (3) optical coatings
that enhance the optics‘ ability to pass laser energy through the
system and to reflect the laser energy; (4) hardware that works in
tandem with computer software to actively track the target missile; (5)
devices that measure atmospheric turbulence and compensate for it so
that it does not scatter or weaken the laser beam; and (6) safety
systems that automatically shut down the high energy laser in the event
of an emergency. At our request, the Airborne Laser Program Office also
assessed the maturity of the oxygen iodine laser.
The program office assessed the optics and stabilizing devices at
technology readiness level four, the optical coatings at level five,
and the safety systems, atmospheric compensation, and target tracking
devices at level six. At level four, scientists have shown that a
technology is technically feasible, but have not shown whether the
technology will have the form, fit, or function required in the
operational system. When a technology progresses to level five, the
technology being tested is incorporated into hardware whose form and
fit are coming closer to that needed for an operational component and
that hardware is integrated with reasonably realistic supporting
elements so that the technology can be tested in a simulated
environment. Finally, at technology readiness level six, integrated
testing of a prototype system has occurred in a relevant environment.
Our prior work has shown that demonstrating a technology readiness
level of seven, that is, demonstrating that components can work
together as a system in a realistic setting, prior to establishing cost
and schedule estimates and beginning system integration, is an
important determinant of program success.
We disagreed with the agency‘s assessment in only one instance--the
assessment of the maturity of the system‘s laser component. The agency
assessed the maturity of the oxygen iodine laser at a readiness level
of six while we consider it to be at a level four. The tests of a one-
module laser that the program office relied upon to prove the maturity
of the laser were conducted in a controlled laboratory environment with
surrogate components. In our opinion, the program office will
demonstrate the laser technology in a relative environment (technology
readiness level six) when the six-module system is integrated and
successfully tested at full power within the high fidelity laboratory
environment of the Airborne Laser Systems Integration Laboratory,
currently under construction at Edwards Air Force Base, California.
Operational Test Community Expects to be Involved in Developmental and
Operational Testing:
By law, the Director of the Office of Operational Test and Evaluation
(OT&E) is the principal advisor to the Secretary of Defense and the
Under Secretary of Defense for Acquisition, Technology, and Logistics
on operational test and evaluation of Department of Defense weapon
systems and is also responsible for monitoring and reviewing live fire
testing activities.[Footnote 3] The Director may not be assigned any
responsibility for developmental test and evaluation, other than to
provide advice to officials responsible for such testing. However, OT&E
officials said that being involved in developmental tests aids them in
preparing for live fire testing and later operational testing and
evaluation, for which they are responsible.
The officials told us that they have primarily been working on issues
related to future live-fire lethality tests of the Airborne Laser, but
they expect their involvement in Airborne Laser‘s developmental tests
will increase when system-level flight tests begin. This expectation is
based on the agency‘s request that the OT&E officials work with Missile
Defense Agency officials to construct developmental test and evaluation
master plans for the integrated missile defense system and each
element. Although OT&E has not been given authority to approve the
Airborne Laser‘s developmental test and evaluation master plan, OT&E
officials said that they will assist in preparing the plan and will
review and offer comments before it is finalized.
In addition to participating in the creation of the developmental test
and evaluation master plans, the Director, OT&E, will participate in
approving each element‘s operational test plan and will evaluate their
operational tests. Further, under the agency‘s new strategy, OT&E
officials are members of the Missile Defense Support Group and its
working group. These groups are expected to provide insight and advice
to the Missile Defense Agency Director and to the Senior Executive
Council that is responsible for all major missile defense decisions.
The agency has also assured the operational test community that it will
be included in all meetings and reviews regarding testing so that test
officials can annually review and report on the adequacy and
sufficiency of the Missile Defense Agency‘s testing program.
Targets in Future Tests Could Change:
Airborne Laser program officials told us that the Airborne Laser was
originally planned as defense against short-range ballistic missiles.
However, with the United States withdrawal from the Antiballistic
Missile Treaty, the Missile Defense Agency is considering tests against
longer-range missiles. While such a change would not affect the program
office‘s plan to test the Airborne Laser against a short-range missile
in December 2004, tests conducted in fiscal year 2005 or 2006 could
include targets representative of longer-range missiles. The
Antiballistic Missile Treaty would have restricted tests that include
longer-range missiles as Airborne Laser targets.
Conclusion:
In an effort to field a missile defense capability quickly, the
Department of Defense has directed the Missile Defense Agency to adopt
a new acquisition strategy. Some of the practices that we observed are
being implemented in the Airborne Laser program are practices that have
been proven in successful development programs. For example, similar to
successful developers, the agency has deferred the establishment of the
Airborne Laser‘s requirements until the knowledge is available to set
realistic requirements. Successful developers also improve their
product incrementally, just as the agency plans to improve the Airborne
Laser through a series of upgrades. However, the agency has the
opportunity to make its acquisition process more knowledge-based. By
establishing knowledge-based decision points at key junctures, the
agency would be in a better position to decide whether to invest in the
next phase of the Airborne Laser‘s development. Also, the agency would
be better able to hold the Airborne Laser Program Office accountable
for planning all of the activities required to develop a quality
product, approaching those activities in a systematic manner so that no
important steps are skipped and problems are resolved sooner rather
than later, and making cost and schedule projections when they have the
knowledge to make realistic estimates. With this disciplined process in
place, the agency will be much better positioned to decide whether to
invest further in the Airborne Laser or use available time and funds
for some other element of the missile defense system.
Our July 12 report recommended that the Director of the Missile Defense
Agency not only establish decision points to separate technology
development from system integration; system integration from system
demonstration; and system demonstration from production, but also
establish knowledge-based criteria that would be used to determine
whether additional investments should be made in the Airborne Laser
program. We are encouraged that the agency is now considering actions
that could prove to be significant steps toward implementing these
recommendations.
Mr. Chairman, this concludes my statement. I would be pleased to
respond to any questions you or members of the committee may have.
FOOTNOTES
[1] U.S. General Accounting Office, Missile Defense: Knowledge-Based
Decision Making Needed To Reduce Risks in Developing Airborne Laser,
GAO-02-631 (Washington, D.C.: July 12, 2002)..
[2] This configuration is known as Block 2004 because it is to be
completed in December 2004.
[3] 10 USC 139
