Missile Defense
Knowledge-Based Decision Making Needed to Reduce Risks in Developing Airborne Laser Gao ID: GAO-02-631 July 12, 2002The Air Force launched an acquisition program to develop and produce a revolutionary laser weapon system, known as the Airborne Laser, in 1996. Being developed for installation in a modified Boeing 747 aircraft, it is intended to destroy enemy ballistic missiles almost immediately after their launch. The Air Force originally estimated development costs at $2.5 billion and projected fielding of the system in 2006. However, by August 2001, the Air Force determined that the development cost estimate rose 50 percent to $3.7 billion, and the fielding date slipped to 2010. The Department of Defense transferred responsibility for the Airborne Laser in October 2001 to the Ballistic Missile Defense Organization. Subsequently, the Defense Secretary designated the Ballistic Missile Defense Organization as the Missile Defense Agency and granted the agency expanded responsibility and authority. The Air Force was unable to meet the Airborne Laser's original cost and schedule goals because it did not fully understand the level of effort that would be required to develop the critical system technology needed to meet the user's requirements. The Missile Defense Agency's new strategy for developing the Airborne Laser incorporates some knowledge-based practices that characterize successful programs. However, the agency has not established knowledge-based decision points and associated criteria for moving forward from technology development to product development and on to production. Without decision points and criteria, the agency risks beginning new and more costly activities before it has the knowledge to determine the money and time required to complete them and whether additional investment in those activities is warranted.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-02-631, Missile Defense: Knowledge-Based Decision Making Needed to Reduce Risks in Developing Airborne Laser
This is the accessible text file for GAO report number GAO-02-631
entitled 'Missile Defense: Knowledge-Based Decision Making Needed to
Reduce Risks in Developing Airborne Laser' which was released on July
16, 2002.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products‘ accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
Report to the Chairman, Subcommittee on National Security, Veterans‘
Affairs, and International Relations, Committee on Government Reform,
House of Representatives:
United States General Accounting Office:
GAO:
July 2002:
Missile Defense:
Knowledge-Based Decision Making Needed to Reduce Risks in Developing
Airborne Laser:
Missile Defense:
GAO-02-631:
Contents:
Letter:
Results in Brief:
Background:
Original Cost and Schedule Goals Are Based on Inadequate Knowledge:
New Strategy Incorporates Some Knowledge-Based Practices, but
Additional Practice Would Reduce Program Risk:
Conclusion:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Scope and Methodology:
Appendix I: Technology Readiness Level Assessment Matrix:
Appendix II: Comments from the Department of Defense:
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Acknowledgments:
Figures:
Figure 1: Airborne Laser Aboard Boeing 747 Aircraft:
Figure 2: Current Airborne Laser Technology Readiness Levels:
Figure 3: The Knowledge-Based Process:
Letter:
July 12, 2002:
The Honorable Christopher Shays
Chairman, Subcommittee on National Security,
Veterans‘ Affairs, and International Relations
Committee on Government Reform
House of Representatives:
Dear Mr. Chairman:
In 1996, the Department of Defense, through the Air Force, launched an
acquisition program to develop and produce a revolutionary laser weapon
system. The system, known as the Airborne Laser because it is being
developed for installation in a modified Boeing 747 aircraft, is
intended to destroy enemy ballistic missiles almost immediately after
their launch (in the so-called ’boost phase“) before they pose a threat
to civilian populations and military assets. The Air Force originally
estimated development costs at $2.5 billion and projected fielding of
the system to begin in 2006. However, by August 2001, the Air Force
determined that maturing the technologies and developing the system
would cost about
50 percent more and take another 4 years. The development cost estimate
increased to $3.7 billion, and the fielding date slipped to 2010.
Against this backdrop of cost increases and schedule delays, the
Department of Defense transferred responsibility for the Airborne Laser
in October 2001 from the Air Force to the Ballistic Missile Defense
Organization, which shared responsibility with the armed services for
developing ballistic missile defense systems. Subsequently, in January
2002, the Secretary of Defense announced broad changes to the
department‘s strategy for developing and acquiring missile defense
systems. Specifically, the Defense Secretary designated the Ballistic
Missile Defense Organization as the Missile Defense Agency and granted
the agency expanded responsibility and authority. The Secretary
directed the agency to develop an integrated system with various
elements that have the capability to attack enemy missiles in all
phases of their flight, transition responsibility for the production
and fielding of systems to the individual services, and encourage
incremental improvements by inserting new technologies through a series
of block upgrades.[Footnote 1] The Airborne Laser is one of many
systems affected by the new strategy.
Concerned about significant cost and schedule problems associated with
developing the Airborne Laser, you asked us to determine (1) why the
systemís development is costing more and taking longer than the Air
Force originally estimated and (2) whether the Missile Defense Agencyís
new strategy for developing the Airborne Laser incorporates the
practices that characterize successful programs.
Results in Brief:
The Air Force was unable to meet the Airborne Laser‘s original cost and
schedule goals because it established those goals before it fully
understood the level of effort that would be required to develop the
critical system technology needed to meet the user‘s requirements. When
the Air Force launched the Airborne Laser acquisition program,
Department of Defense policy required that program cost and schedule
goals be established[Footnote 2]. In 1996, at program launch, the Air
Force did not have enough knowledge about the technology challenges
facing the program. As a result, the Air Force underestimated the
complexity of the engineering task at hand and misjudged the amount of
time and money that the program would need. Some critical technologies
that the system‘s design depends upon remain immature, making it very
difficult, even today, for analysts to establish realistic cost and
schedule goals.
The Missile Defense Agency‘s new strategy for developing the Airborne
Laser incorporates some knowledge-based practices that characterize
successful programs. For example, one practice that the agency
implemented is a requirements process that gives the agency the
flexibility to develop a system that has some capability without being
held to requirements that cannot be met with currently available
technology. A second knowledge-based practice is the provision of
additional time and facilities for testing. Increased testing allows
agency officials to reach a better understanding of the capabilities of
the technology so that they can establish more realistic requirements
and ultimately more accurate estimates of the time and money needed to
meet those requirements. A third practice is the collection of the
types of information needed to determine whether the technology is ’in-
hand“ to give war fighters an Airborne Laser with some, if not all,
desired capabilities. For example, the agency intends to compare
developed capabilities with data derived from intelligence sources on
the likely launch points and types of missiles that the system could
encounter.
However, the agency has not established knowledge-based decision points
and associated criteria for moving forward from technology development
to product development and on to production. Separating technology
development from product development has been a critical determinant
for successful program outcomes. Without decision points and criteria,
the agency risks beginning new and more costly activities before it has
the knowledge to determine the money and time required to complete the
activities and whether additional investment in those activities is
warranted. Also, the agency risks beginning the activities before it
has the knowledge to complete them without the need for expensive
rework.
We are making recommendations that are intended to make the Missile
Defense Agency‘s acquisition process more disciplined and provide
better information for decision makers as additional investments in the
Airborne Laser are considered.
In commenting on a draft of this report, the Department of Defense
partially concurred with our recommendations. The department stated
that Secretary of Defense direction is not needed to implement our
recommendations, the Missile Defense Agency‘s acquisition process for
ballistic missile defense already uses tailored versions of the
knowledge-based practices recommended by us, and the agency intends to
expand the use of knowledge-based criteria in the future. The Missile
Defense Agency‘s acquisition process separates acquisition into three
phases--development, transition, and production. While the process
definitely separates product development from production, it clearly
does not separate technology development from product development.
Also, it does not establish the knowledge-based criteria characteristic
of successful programs at any of these decision points. Because we have
not seen evidence in the Airborne Laser‘s strategy that such decision
points and criteria are in place, we have retained our recommendations.
Background:
The effort to develop the Airborne Laser is based on over 25 years of
scientific development in the Departments of Defense and Energy. It
evolved primarily from Airborne Laser laboratory research to develop
applications for high-energy lasers. This research culminated in a
demonstration that showed that a low-power, short-range laser was
capable of destroying a short-range air-to-air missile. Although this
demonstration was considered militarily insignificant because of the
laser‘s low power and short range, it did succeed in identifying
technologies that were necessary for the development of an operational
Airborne Laser system. The research showed that an operational system
would need optics that could compensate for the atmospheric turbulence
that weakens and scatters a laser beam, optical devices that could
withstand the heat produced by a high-energy laser without the added
weight of water-cooling devices, and a new chemical laser with higher
energy levels that would produce a stronger laser beam.
In 1996, the Air Force launched the Airborne Laser program to develop a
defensive system that could destroy enemy missiles from a distance of
several hundred kilometers. Engineers determined that if they were to
meet this requirement, the system would need a 14-module[Footnote 3]
oxygen iodine laser. They also determined that the system would need a
beam control/fire control assembly that could (1) safely move the laser
beam through the aircraft, (2) shape the beam so that it was not
scattered or weakened by the atmosphere, and (3) hold the beam on
target, despite the movement of the aircraft. In addition, engineers
determined that the system would need a battle management and control
system capable of planning and executing an engagement.
The Air Force planned to have the science and technology community
develop extensive knowledge about the laser and beam control/fire
control technologies before it launched an Airborne Laser acquisition
program. However, according to the retired manager of the science and
technology project, the budgets for technology efforts were limited,
and the science and technology community could not fund the technology
maturation effort. The Air Force knew that a program office was more
likely to command the large budget needed to fully mature technologies,
so it launched an acquisition program and assigned the program manager
responsibility for both technology and product development. The program
manager planned to demonstrate critical Airborne Laser technologies by
first building a six-module version of the oxygen iodine laser,
installing it along with other system components aboard a Boeing 747
aircraft (see
fig. 1), and testing the capability of this scaled system in system-
level flight tests. The tests would conclude in 2003 with an attempt to
shoot down a short-range ballistic missile target at a distance of 100
kilometers. If this final test were successful, the Airborne Laser
would have moved into product development.
Figure 1: Airborne Laser Aboard Boeing 747 Aircraft:
[See PDF for image]
Source: Airborne Laser Program Office.
[End of figure]
Original Cost and Schedule Goals Are Based on Inadequate Knowledge:
The Air Force launched the Airborne Laser acquisition program and
identified cost and schedule goals before officials had the knowledge
to make realistic projections. In 1996, when the program was launched,
Department of Defense regulation 5000.2 required, and still requires
today, that when a military service initiates a major acquisition
program, it must establish cost and schedule goals. However, the Air
Force could not make realistic estimates when it began the program
because it had no way of knowing how much engineering effort would be
needed to complete the development of technology critical to the
system. Even today, some critical technologies that the system‘s design
depends upon remain immature, making it very difficult for analysts to
determine how long it will take and how much it will cost to develop
and produce the system.
Technologies Were Immature at Program Launch:
At the time the Airborne Laser program was launched, the laser and beam
control/fire control technologies needed to develop the Airborne Laser
system was immature. The Department of Defense‘s science and technology
community was actively researching and developing the laser and had
produced a weak beam in a laboratory setting--but this major component
had not reached the level of maturity needed to proceed into product
development. The technology necessary to develop the beam control/fire
control was even less advanced. Most of the scientists‘ work was
limited to analytical studies, wherein a few tests of laboratory
hardware were linked together to work somewhat like the intended
component.
Because technology development is a process of discovery, the Air Force
soon learned that there were too many unknowns regarding the
development of Airborne Laser technology to make good cost and schedule
estimates. As the technology development progressed, unanticipated
technical challenges affected the program‘s cost and schedule.
Department of Defense analysts reported that the Airborne Laser program
experienced cost and schedule growth because the program and its
contractors underestimated the complexity of
(1) designing laser components, (2) the system‘s engineering analysis
and design effort, and (3) engineering the system to fit on board the
aircraft. As system development progressed and the Air Force gained a
better understanding of the technical complexity of the system, the Air
Force increased its cost and schedule estimates.
Some Critical Technologies Remain Immature:
The Air Force has made some progress in developing the Airborne Laser‘s
critical technologies, but many remain immature. We asked the Airborne
Laser program office to determine the technologies most critical to the
Airborne Laser system and to use technology readiness levels[Footnote
4] to assess the maturity of each. The officials determined that if the
Airborne Laser is to meet the requirements established by the war
fighters, then engineers must mature technologies in six areas, all of
which are needed to successfully design the system. These technologies
are:
* devices that stabilize the laser system aboard the aircraft so that
the beam can be maintained firmly on the target,
* optics--mirrors and windows--that focus and control the laser beam
and allow it to pass safely through the aircraft,
* optical coatings that enhance the optics‘ ability to pass laser
energy through the system and to reflect the laser energy,
* hardware that works in tandem with computer software to actively
track the target missile,
* devices that measure atmospheric turbulence and compensate for it so
that it does not scatter or weaken the laser beam, and:
* safety systems that automatically shut down the high energy laser in
the event of an emergency.
At our request, the program office also assessed the maturity of the
oxygen iodine laser.
As figure 2 shows, program officials assessed the optical coatings at
level five and the safety systems, atmospheric compensation, and
target-tracking components at level six. At technology readiness level
five, the technology being tested is incorporated into hardware whose
form and fit are coming closer to that needed for an operational
component and integrated with reasonable realistic supporting elements
so that the technology can be tested in a simulated environment. At
level six, the technology is incorporated into a prototype and tested
in a high-fidelity laboratory environment or in a simulated operational
environment. The program officials identified the optics and
stabilizing devices as the least mature--at level four. At this level,
engineers have shown that a technology is technically feasible but have
not shown whether the technology will have the form, fit, or function
required in the operational system. We agreed with all but one of the
program officials‘ assessments for these technologies.
Figure 2: Current Airborne Laser Technology Readiness Levels:
Note: Appendix I contains information describing technology readiness
levels.
[See PDF for image]
Source: GAO‘s analysis.
[End of figure]
Our one disagreement centered on the maturity of the laser component of
the system. While the program office assessed it at a technology
readiness level of six, we consider the laser technology to be at level
four because tests have been conducted only for a one-module laser in a
controlled laboratory environment using surrogate components. For
example, the tests used a stable laser resonator, rather than the
unstable resonator that will be used in system-level flight
tests.[Footnote 5] We also found that during tests of the one-module
laser, the resonator was operating in multimode rather than single-
mode.[Footnote 6] The resonator in the operational system will operate
in single mode. Furthermore, the chemical storage and delivery
subcomponents used in these tests were not representative of those that
will be incorporated into the system‘s design. According to program
office officials, conducting a more realistic test would have cost time
and money that were not available.
Documents summarizing the tests of the one-module laser stated that the
tests were successful in reducing the technical risks associated with
the one-module system but that a new set of technical risks linked with
developing a multimodule system must still be addressed during testing
of the six-module system. In our opinion, the program office will
demonstrate the laser technology in a relative environment (technology
readiness level six) when the six-module system is integrated and
successfully tested at full power within the high-fidelity laboratory
environment of the Airborne Laser Systems Integration Laboratory,
currently under construction at Edwards Air Force Base, California.
According to the program office, this type of demonstration will not
occur until February 2003.
New Strategy Incorporates Some Knowledge-Based Practices, but
Additional Practice Would Reduce Program Risk:
The Missile Defense Agency‘s new strategy for developing the Airborne
Laser incorporates some of the knowledge-based practices that
characterize successful programs, but the agency would benefit from
adopting another that would add greater discipline to its acquisition
process. The new strategy allows more flexibility in setting
requirements, makes time and facilities available to mature and test
the critical technologies, and collects information needed to match the
war fighters‘ requirements to demonstrated technology. However, the
agency has not established decision points with associated knowledge-
based criteria for moving forward from (1) technology development to
system integration, (2) system integration to system demonstration, and
(3) system demonstration to production. At each of these points, the
agency would stop to assess its knowledge and decide whether investment
in the program‘s next phase is warranted.
New Strategy Introduces Knowledge-Based Practices:
The first new practice allows the Missile Defense Agency to refine
requirements on the basis of the results of system engineering. The
Department of Defense ordinarily faces significant hurdles in matching
requirements to resources. The fundamental problem is twofold. First,
under the department‘s traditional process, requirements must be set
before a program can be approved and a program must be approved before
the product developer conducts systems engineering. Second, the
competition for funding encourages requirements that will make the
desired weapon system stand out from others. Consequently, many of the
department‘s product development programs include unrealistic
requirements set by the user before the product developer has conducted
the system engineering necessary to identify the time, technology, and
money necessary to develop a product capable of meeting requirements.
A second practice that is likely to improve the Airborne Laser‘s
development is making the time and facilities available to mature and
test critical technologies. To implement this practice, the agency
increased the time available to test the six-module laser system and is
building a new test facility. Instead of following the Air Force‘s plan
to complete system-level flight tests of the six-module system in the
last quarter of fiscal year 2003, the agency has delayed the
demonstration to the first quarter of fiscal year 2005. This delay will
allow additional time to learn from and correct problems discovered
during system-level tests that are scheduled to begin in the last
quarter of fiscal year 2003 and end with the fiscal year 2005
demonstration. In addition, the agency plans to increase the Airborne
Laser‘s ground-testing capability by awarding a contract in 2003 for
what the agency is calling an ’iron bird,“ which is essentially an
aircraft hull with installed laser equipment. The ’iron bird“ is
expected to allow testing of a fully integrated Airborne Laser system
on the ground so that technologies for future blocks can be evaluated
before being installed in an aircraft.
The information gained from testing informs the requirements process.
Because testing allows developers to gauge the progress being made in
translating an idea into a weapon system, it enables the developer to
make a more informed decision as to whether a technology is ready to be
incorporated into a system‘s design. With this knowledge, the developer
can determine whether the technology is so important to the system‘s
design that additional time and money should be spent to mature the
technology or whether the system‘s initial performance requirements
should be reduced.
A third practice that the agency plans to adopt is matching
requirements to available technology. According to the Missile Defense
Agency‘s Technical Director, the agency defines the war fighters‘
requirement as a system that has the capability to destroy some threat
ballistic missiles during their boost phase at a range representative
of an operational scenario. The Technical Director told us that the
agency will attain the knowledge to determine if it has the technology
in-hand to meet this requirement by examining each block‘s capabilities
during simulated and system-level flight test and comparing those
capabilities with data derived from intelligence sources on the likely
launch points and types of missiles that the system could encounter.
Our previous work with successful development programs shows that once
the technology is in-hand to meet the customer‘s requirements, the
developer can make more accurate initial estimates of the cost and time
needed to develop and produce an operational system.
Successful Developers Recognize Need for Knowledge-Based Decision
Points:
Successful developers have instilled discipline in their acquisition
processes by requiring that certain criteria for attaining knowledge
are met as an acquisition program moves forward. (See fig. 3.) They
recognize that the focus and cost of activities change over time and
that less rework is required if all activities with the same focus are
completed before beginning other activities.
Figure 3: The Knowledge-Based Process:
[See PDF for image]
Source: GAO‘s analysis.
[End of figure]
In successful development programs, decisions are made when the
knowledge is available to support those decisions. The first decision
point, or knowledge point, occurs when the focus of a developer‘s
activities changes from technology development to system integration--
the first phase of product development. The criterion for deciding to
move forward is having the knowledge to match requirements and
available resources (time, technology, and funds). The second knowledge
point occurs between system integration and system demonstration when
the developer has successfully integrated subsystems and components
into a stable design that not only meets the customer‘s performance
requirements but also is optimized for reproducibility,
maintainability, and reliability. The decision criterion used here is
usually having completed about 90 percent of the engineering drawings.
The third knowledge point separates system demonstration from
production. The decision to invest in production is generally based on
a determination that the product performs as required during testing
and that the manufacturing processes will produce a product within
cost, schedule, and quality targets.
The cost of a program‘s activities increases as it moves closer to
production. In commercial acquisitions, product development is
typically much more costly than technology development. During
technology development, small teams of technologists work to perfect
the application of scientific knowledge to a practical problem. As
product development begins, developers begin to make larger investments
in human capital, bringing on a large engineering force to design and
manufacture the product. In addition, product development requires
significant investments in facilities and materials. These investments
increase continuously as the product approaches the point of
manufacture. In fact, industry experts estimate that identifying and
resolving a problem during product development can cost 10 times more
than correcting that problem during technology development and that
correcting the problem during manufacturing is even more costly.
Knowledge-Based Decisions Missing from Airborne Laser Strategy:
We examined the Airborne Laser‘s acquisition strategy and determined
that it does not include decision points at which officials would use
knowledge-based criteria to determine if the program is ready to move
from technology development to system integration, system integration
to system demonstration, and system demonstration to production. We
found that the agency‘s process has three phases: development,
transition, and production.
* Development includes all developmental activities and system-level
demonstrations of military utility.
* Transition will involve preparation of the operational requirements
document by the appropriate armed service and conducting operational
testing.
* Production will involve producing and fielding the final weapon
system.
The agency‘s strategy also calls for developing the Airborne Laser
incrementally, rather than trying to initially develop a system with
all desired capabilities. In the near term, the agency plans to
complete the six-module laser system aircraft, now known as block
2004,[Footnote 7] and use it to demonstrate critical Airborne Laser
technologies. Beginning in March 2003, the agency intends to begin
developing another demonstration aircraft, known as block 2008, which
will incorporate new capabilities and technologies. The Airborne Laser
program manager told us that blocks 2004 and 2008 are primarily test
assets for the purpose of technology demonstration. While some of the
block 2008 activities are focused on improving subsystems and
components, such as reducing the weight of laser components and
improving optics, other activities are focused on the integration of
these pieces into a block 2008 design.[Footnote 8]
The agency expects to develop subsequent blocks, or system
configurations to introduce additional capabilities. If system-level
tests show that any one of these configurations performs at a level
that merits fielding, the Air Force will prepare a requirements
document based on the configuration‘s demonstrated capabilities and
make plans for operational testing and production. This ’baseline“
capability would be improved in subsequent blocks as more advanced
technology becomes available and as the threat warrants.
We did not find that the agency‘s strategy includes a disciplined
process that separates technology development, system integration,
system demonstration, and production with decision points supported by
knowledge-based criteria. Instead, the agency has put in place a
decision point for moving from the development to the transition phase.
According to the agency‘s strategy, when the agency determines that it
has the technology in-hand to produce a system that merits fielding, it
will begin to transition the system over to the appropriate military
service. Also, at the end of the transition phase, a system would enter
the formal Department of Defense acquisition process at Milestone C--
the point at which the decision is made to enter low rate initial
production. We did not find, however, an established set of decision
points with associated criteria that would enable the agency to make a
knowledge-based decision on whether to invest in system integration
and, subsequently, system demonstration and production. That is, even
though the agency might know that it has the technology in-hand to
develop a useful military capability, it has not established a first
decision point where it would determine the cost and time needed to
move the program forward and whether the program should proceed into a
system integration phase during which the design would be matured and
optimized for reproducibility, maintainability, and reliability.
Neither does the agency‘s strategy include a second decision point that
would allow agency officials to use the knowledge they have attained
regarding the design‘s maturity to determine whether to invest further
to demonstrate that the system meets requirements and that
manufacturing processes are in place to repeatedly produce a quality
product. Only after the agency successfully moves the program through
all of these decision points and successfully demonstrates the system‘s
capabilities and manufacturing processes would the agency‘s production
decision be fully knowledge based. Without this disciplined process,
the agency would be accepting greater cost and schedule risks and is
much less likely to realize the full potential benefits of its new
approach to developing missile defense systems.
Conclusion:
The revolutionary nature of missile defense weapon systems demands
cutting-edge technology. Although there is no one approach that ensures
that a developer can deal successfully with the unknowns inherent in
developing a product from such technology, the knowledge-based process
has proven to yield good results within cost and schedule estimates.
The Missile Defense Agency has implemented practices that are part of
the knowledge-based approach, and these practices are likely to improve
the agency‘s ability to gather the knowledge it needs to develop an
Airborne Laser capability acceptable to the war fighter. However, the
agency has the opportunity to make its acquisition process more
disciplined. By establishing knowledge-based decision points at key
junctures, the agency would be in a better position to decide whether
to move from one development phase to the next. Also, the agency would
be better able to hold system developers accountable for planning all
of the activities required to develop a quality product, approaching
those activities in a systematic manner so that no important steps are
skipped and problems are resolved sooner rather than later, and making
cost and schedule projections when they have the knowledge to make
realistic estimates. With this disciplined process in place, the agency
is much more likely to achieve a needed capability for the war fighter
within established cost and schedule goals.
Recommendations for Executive Action:
To make its acquisition process more disciplined and provide better
information for decision makers as additional investments in the
Airborne Laser are considered, we recommend that the Secretary of
Defense direct the Director of the Missile Defense Agency to establish
decision points separating technology development from system
integration, system integration from system demonstration, and system
demonstration from production. For each decision point, we recommend
that the Secretary instruct the Director to establish knowledge-based
criteria and use those criteria to determine where additional
investments should be made in the program.
Agency Comments and Our Evaluation:
In commenting on a draft of this report, the Department of Defense
partially concurred with our recommendations (see appendix II). The
department stated that Secretary of Defense direction is not needed to
implement our recommendations, the Missile Defense Agency‘s acquisition
process for ballistic missile defense already uses tailored versions of
the knowledge-based practices recommended by us, and the agency intends
to expand the use of knowledge-based criteria in the future.
The Department of Defense has not fully implemented the knowledge-based
process recommended in our reports. Effective product development
depends on gaining sufficient knowledge about technology, design, and
manufacturing processes at key points in a system‘s development. At
those points, using metrics--such as technology readiness levels to
measure the maturity of technology--that are commonly understood allow
informed trade-offs to be made between resources, including cost and
time, and performance. We have found that product development
activities, such as building engineering prototypes of an integrated
system and then demonstrating that the system can be manufactured to
acceptable cost and quality standards, are ineffective unless the
technologies needed to meet the product‘s intended capabilities are
fully matured and ready for system integration. Virtually every world-
class product developer we have spoken with agrees with this.
The Airborne Laser program does not appear to have established this
type of decision-making process. The Missile Defense Agency appears to
have set up a development phase that combines maturing technologies
with establishing a stable design. It does not include any visible
decision points or standards to clearly indicate when technology
development is concluded and system integration work to establish a
design begins. Thus, it appears to us that this acquisition process
forces the agency to manage significant risk from immature technologies
simultaneously with trying to build a stable product design during this
phase. Further, separating system integration from system demonstration
and system demonstration from production and using common metrics in
deciding to move forward will enhance the future likelihood that
decisions on the Airborne Laser will be cost-effective. Such a process
will also enhance decision-makers‘ ability across the range of missile
defense elements by facilitating comparisons across elements.
Therefore, we have retained our recommendations.
Scope and Methodology:
To address our objectives, we reviewed the contractor‘s monthly cost
performance reports, Defense Contract Management Agency analyses of
those reports, and Defense Acquisition Executive Summaries and Selected
Acquisition Reports prepared by the Airborne Laser program office. We
also discussed cost and schedule problems with Airborne Laser program
officials, Kirtland Air Force Base, New Mexico; and contractor
officials at the Boeing Company, Seattle, Washington; Lockheed Martin,
Sunnyvale, California; and TRW, Los Angeles, California. In addition,
we obtained a technology readiness level analysis of the system‘s
critical technologies from the Airborne Laser program office. We
compared this analysis with information obtained during our prior
review to determine if progress had been made in maturing the critical
technologies to higher technology readiness levels. We obtained
detailed briefings from program office personnel and Missile Defense
Agency officials, Arlington, Virginia; and from the contractors about
the status of critical technologies and the problems associated with
maturing the technologies required for the laser, the beam control/fire
control system, and the required aircraft modifications. We also
obtained detailed briefings from program office and Missile Defense
Agency officials regarding the new Missile Defense Agency acquisition
process and the implementation of this process within the Airborne
Laser program. We conducted our review from July 2001 through May 2002
in accordance with generally accepted government auditing standards.
As agreed with your office, unless you publicly announce the contents
of this report earlier, we plan no further distribution until 30 days
from the report date. At that time, we will send copies of this report
to the congressional defense committees; the Secretary of Defense; the
Director, Missile Defense Agency, the Secretary of the Air Force; and
the Director, Office of Management and Budget. We will also make copies
available to other interested parties upon request. In addition, the
report will be available at no charge on the GAO Web site at http://
www.gao.gov.
Please contact me at (202) 512-4841 if you or your staff have any
questions concerning this report. Key contributors to this report are
identified in appendix III.
Sincerely yours,
R. E. Levin
Director, Acquisition and
Sourcing Management:
Signed by R. E. Levin:
[End of section]
Appendix I: Technology Readiness Level Assessment Matrix:
Technology readiness level: 1. Basic principles observed and reported.;
Description: Lowest level of technology readiness. Scientific research
begins to be translated into applied research and development. Examples
might include paper studies of a technology‘s basic properties.
Technology readiness level: 2. Technology concept and/or application
formulated.; Description: Invention begins. Once basic principles are
observed, practical applications can be invented. The application is
speculative, and there is no proof or detailed analysis to support the
assumption. Examples are still limited to paper studies.
Technology readiness level: 3. Analytical and experimental critical
function and/or characteristic proof of concept.; Description: Active
research and development is initiated. This includes analytical studies
and laboratory studies to physically validate analytical predictions of
separate elements of the technology. Examples include components that
are not yet integrated or representative.
Technology readiness level: 4. Component and/or breadboard. Validation
in laboratory environment.; Description: Basic technological
components are integrated to establish that the pieces will work
together. This is relatively ’low fidelity“ compared with the eventual
system. Examples include integration of ’ad hoc“ hardware in a
laboratory.
Technology readiness level: 5. Component and/or breadboard validation
in relevant environment.; Description: Fidelity of breadboard
technology increases significantly. The basic technological components
are integrated with reasonably realistic supporting elements so that
the technology can be tested in a simulated environment. Examples
include ’high fidelity“ laboratory integration of components..
Technology readiness level: 6. System/subsystem model or prototype
demonstration in a relevant environment.; Description: Representative
model or prototype system, which is well beyond the breadboard tested
for technology readiness level five, is tested in a relevant
environment. Represents a major step up in a technology‘s demonstrated
readiness. Examples include testing a prototype in a high-fidelity
laboratory environment or in simulated operational environment.
Technology readiness level: 7. System prototype demonstration in an
operational environment.; Description: Prototype near or at planned
operational system. Represents a major step up from technology
readiness level six, requiring the demonstration of an actual system
prototype in an operational environment, such as in an aircraft,
vehicle, or space. Examples include testing the prototype in a test bed
aircraft.
Technology readiness level: 8. Actual system completed and ’flight
qualified“ through test and demonstration.; Description: Technology has
been proven to work in its final form and under expected conditions. In
almost all cases, this technology readiness level represents the end of
true system development. Examples include developmental test and
evaluation of the system in its intended weapon system to determine if
it meets design specifications..
Technology readiness level: 9. Actual system ’flight proven“ through
successful mission operations.; Description: Actual application of the
technology in its final form and under mission conditions, such as
those encountered in operational test and evaluation. In almost all
cases, this is the end of the last ’bug fixing“ aspects of true system
development. Examples include using the system under operational
mission conditions.
[End of table]
[End of section]
Appendix II: Comments from the Department of Defense:
OFFICE OF THE UNDER SECRETARY OF DEFENSE:
3000 DEFENSE PENTAGON WASHINGTON, DC 20301-3000:
ACQUISITION, TECHNOLOGY AND LOGISTICS:
0 8 JUL2002:
Mr. Robert E. Levin:
Director, Acquisition and Sourcing Management U. S. General Accounting
Office:
441 G. Street, N.W. Washington, DC 20548:
Dear Mr Levin:
This is the Department of Defense (DoD) response to the GAO Draft
Report ’MISSILE DEFENSE: Knowledge-based Decision Making Needed to
Reduce Risks in Developing Airborne Laser,“ dated June 25, 2002 (GAO
Code 120079/GAO-02-631).
The DoD has reviewed the draft report and partially concurs with the
recommendations. Specific comments for each recommendation are
enclosed. We have recommended some factual corrections. We also
provided some administrative comments under separate cover. My action
officer for this effort is Major Mark Arbogast, (703) 695-7328, mark.
arbogast@osd.mil.
We appreciate the opportunity to comment on the draft report.
Glenn F. Lamartin Director:
Strategic and Tactical Systems:
Signed by an official for Glenn F. Lamartin:
Attachment:
GAO DRAFT REPORT - DATED JUNE 25, 2002 GAO CODE 120079/GAO-02-631:
’MISSILE DEFENSE: KNOWLEDGE-BASED DECISION MAKING NEEDED TO REDUCE
RISKS IN DEVELOPING AIRBORNE LASER“:
DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATIONS:
RECOMMENDATION 1: The GAO recommended that the Secretary of Defense
direct the Director of the Missile Defense Agency to:
Establish decision points separating technology development from system
integration; system integration from system demonstration; and system
demonstration from production. (p. 16/GAO Draft Report):
DOD RESPONSE: Partially concur. Secretary of Defense direction is not
needed to implement the recommendation. The Department of Defense‘s
acquisition process for ballistic missile defense consists of three
phases: development, transition, and procurement and operations. That
process already uses the knowledge-based practices recommended by the
GAO, tailoring them as appropriate to meet the objectives of the
Ballistic Missile Defense System (BMDS) program. The Department plans
to demonstrate system performance in a realistic test environment
before the transition phase and the subsequent procurement and
operations phase.
RECOMMENDATION 2: The GAO recommended that the Secretary of Defense
direct the Director of the Missile Defense Agency (MDA) to:
Establish knowledge-based criteria, for each of the decision points
mentioned in the initial recommendation, and use that criteria to
determine where additional investments should be made in the program.
(p. 16/GAO Draft Report):
DOD RESPONSE: Partially concur. Secretary of Defense direction is not
needed to implement the recommendation. MDA uses an event-based
approach to development, where a program moves forward based on its
progress as demonstrated by events, not by calendar date. MDA intends
to expand the use of knowledge-based criteria as the basis
for these key events, including technology development, system
integration, system demonstration, and production; tailoring the
criteria, consistent with a capability-based spiral development
approach directed by the Secretary of Defense for the BMDS.
[End of section]
Appendix III GAO Contact and Staff Acknowledgments:
GAO Contact:
R. E. Levin (202) 512-4841:
Acknowledgments:
In addition to the contact named above, Christina Chaplain, Marcus
Ferguson, Tom Gordon, Subrata Ghoshroy, Barbara Haynes, Matt Lea, Hai
Tran, Adam Vodraska, and John Warren made key contributions to this
report.
[End of section]
FOOTNOTES
[1] The Missile Defense Agency plans to develop a series of Airborne
Laser configurations, which are referred to as ’blocks.“ It is expected
that each block will include improved technology that was not available
in the prior block.
[2] This policy implements statutory planning and reporting
requirements for major defense acquisition programs.
[3] The chemical reaction that generates the laser energy occurs in the
laser modules. The amount of laser energy produced increases as the
number of laser modules increases.
[4] Technology readiness levels were developed by the National
Aeronautics and Space Agency and are recommended for use by the
Department of Defense and the military services. (See appendix I for
their definition.)
[5] A resonator consists of two mirrors placed at opposite ends of a
laser cavity. As the reaction of chemicals within the laser cavity
produces photons of light, the photons are reflected back and forth
between the two mirrors, which generates additional photons and creates
a state of high energy within the cavity. In a stable resonator, one
mirror is fully reflective while the other is partially reflective and
partially transmissive. Energy that escapes from the laser cavity
through the transmissive portion of the mirror in a stable resonator
forms a high-energy beam. In an unstable resonator, both mirrors are
fully reflective, and one is much smaller in diameter. As the photons
are reflected from the larger mirror in the direction of the smaller
mirror, energy escapes from the laser cavity around the edges of the
smaller mirror and forms a doughnut-shaped beam.
[6] As photons are generated in a laser resonator, the photons
oscillate or move in different ways. A resonator operating in single-
mode suppresses all photons except those oscillating at a certain
frequency so that the beam produced can be directed at one spot on the
target. A resonator operating in multimode does not suppress any
photons, regardless of their frequency. While a multimode resonator
directs more energy toward the target, all of that energy will not be
focused on one area of the target.
[7] The six-module system is referred to as block 2004 because testing
will conclude in December 2004. Testing of a second configuration,
known as block 2008, will be completed in December 2008.
[8] One of the major technical challenges is accommodating the laser‘s
weight. Engineers determined that the six-module system would weigh
180,000 pounds, but the original system requirement was that the system
must weigh no more than 175,000 pounds with
14 laser modules. Because each additional module weighs about 6,000
pounds, the agency intends to redesign some components to reduce their
mass and redistribute the weight using a passenger version of the
Boeing 747 as the block 2008 aircraft. The passenger version of the 747
can accommodate the crew on an upper deck, thereby allowing the laser‘s
weight to be moved forward where it places less stress on the aircraft
frame.
GAO‘s Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO‘s commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO‘s Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ’Today‘s Reports,“ on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select ’Subscribe to daily E-mail alert for newly
released products“ under the GAO Reports heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW,
Room LM Washington,
D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.
General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.
20548:
