The Department of Homeland Security Needs to Fully Adopt a Knowledge-based Approach to Its Counter-MANPADS Development Program
Gao ID: GAO-04-341R January 30, 2004In late 2002, terrorists fired surface-to-air missiles at an Israeli airliner departing from Mombasa, Kenya--the first time man-portable air defense systems (MANPADS) had been used to attack commercial aircraft in a non-combat zone. Given concerns about the vulnerability of the commercial airline industry and the potential impact of an attack in the United States, the House Committee on Armed Services and the House Committee on Transportation and Infrastructure, Aviation Subcommittee, requested that we conduct an assessment of the federal government's efforts to address the MANPADS threat against commercial aircraft, including its nature and extent; the Department of Defense's monitoring of Stinger missiles exported to other countries; and U.S. bilateral and multilateral efforts to address international MANPADS proliferation. After we began work on this assessment, the Department of Homeland Security (DHS) took steps to initiate a 2-year system development and demonstration program for a counter-MANPADS system and awarded the initial contracts in January 2004. On December 4, 2003, we briefed Congress on our views about DHS's approach to developing the system. This report summarizes that information and transmits the portion of the briefing related to DHS's counter-MANPADS development effort. Our assessment of the other federal efforts to address the threat is ongoing, and we expect to complete our report in the spring of 2004.
DHS faces significant challenges in adapting a military counter-MANPADS system to commercial aircraft. These challenges include establishing system requirements, maturing technology and design, and setting reliable cost estimates. For instance, DHS has to account for a wide variety of aircraft types in designing and integrating the system. Our past work on the best practices of product developers in government and industry has found that the use of a knowledge-based approach is a key factor in successfully addressing such challenges. This approach includes the use of exit criteria or controls to ensure that sufficient knowledge has been attained at critical phases of the product development process. Based on input we provided during the course of our review, DHS updated its initial solicitation to incorporate these knowledge-based exit criteria. We think this a positive first step, and we are recommending that the Secretary of Homeland Security ensure that the knowledgebased approach is fully implemented throughout the course of its counter-MANPADS development program. DHS fully concurred.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Michael J. Sullivan Team: Government Accountability Office: Acquisition and Sourcing Management Phone: (202) 512-3519GAO-04-341R, The Department of Homeland Security Needs to Fully Adopt a Knowledge-based Approach to Its Counter-MANPADS Development Program
This is the accessible text file for GAO report number GAO-04-341R
entitled 'The Department of Homeland Security Needs to Fully Adopt a
Knowledge-based Approach to Its Counter-MANPADS Development Program'
which was released on January 30, 2004.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
January 30, 2004:
The Honorable Duncan Hunter:
Chairman:
The Honorable Ike Skelton:
Ranking Minority Member:
Committee on Armed Services:
House of Representatives:
The Honorable John L. Mica:
Chairman:
The Honorable Peter A. DeFazio:
Ranking Minority Member:
Subcommittee on Aviation:
Committee on Transportation and Infrastructure:
House of Representatives:
Subject: The Department of Homeland Security Needs to Fully Adopt a
Knowledge-based Approach to Its Counter-MANPADS Development Program:
In late 2002, terrorists fired surface-to-air missiles at an Israeli
airliner departing from Mombasa, Kenya--the first time man-portable air
defense systems (MANPADS) had been used to attack commercial aircraft
in a non-combat zone. Given concerns about the vulnerability of the
commercial airline industry and the potential impact of an attack in
the United States, you requested that we conduct an assessment of the
federal government's efforts to address the MANPADS threat against
commercial aircraft, including its nature and extent; the Department of
Defense's monitoring of Stinger missiles exported to other countries;
and U.S. bilateral and multilateral efforts to address international
MANPADS proliferation. After we began work on this assessment, the
Department of Homeland Security (DHS) took steps to initiate a 2-year
system development and demonstration program for a counter-MANPADS
system and awarded the initial contracts in January 2004. On December
4, 2003, we briefed your staff on our views about DHS's approach to
developing the system. This report summarizes that information and
transmits the portion of the briefing related to DHS's counter-MANPADS
development effort (see encl. I). Our assessment of the other federal
efforts to address the threat is ongoing, and we expect to complete our
report in the spring of 2004.
Background:
Due in part to the Mombasa attack, the White House convened a task
force to develop a strategy to reduce the MANPADS threat against
commercial aircraft. In reviewing available technical countermeasures,
the task force identified an on-board jammer (directed infrared
countermeasure, or DIRCM) as the most promising technology to meet
current threats while potentially satisfying operational constraints
imposed by the commercial aircraft industry such as minimizing the cost
to operate and maintain these systems.
The Congress directed DHS to submit a plan to develop and demonstrate a
counter-MANPADS device for commercial aircraft.[Footnote 1] On October
3, 2003, DHS released a solicitation that outlines a 2-year, two-phased
system development and demonstration program to produce prototype
systems that would satisfy performance, operational, and cost
constraints. In Phase I, which begins in January 2004, DHS intends to
conduct preliminary design and analysis activities. In Phase II, which
begins about 6 months later, they plan to develop and test the
prototypes.
The objective of the DHS program is to (1) migrate existing military
countermeasure technologies to the civil aviation environment and (2)
minimize the total life-cycle cost of the system, which includes
development, procurement, installation, operation and support costs.
The solicitation focuses primarily on the DIRCM concept, which combines
a missile warning system (MWS) to detect a missile launch and a laser
to jam the guidance system of the missile. DOD currently uses DIRCM
technology on some of its large transport aircraft, such as the C-17.
Results in Brief:
DHS faces significant challenges in adapting a military counter-MANPADS
system to commercial aircraft. These challenges include establishing
system requirements, maturing technology and design, and setting
reliable cost estimates. For instance, DHS has to account for a wide
variety of aircraft types in designing and integrating the system. Our
past work on the best practices of product developers in government and
industry has found that the use of a knowledge-based approach is a key
factor in successfully addressing such challenges. This approach
includes the use of exit criteria or controls to ensure that sufficient
knowledge has been attained at critical phases of the product
development process. Based on input we provided during the course of
our review, DHS updated its initial solicitation to incorporate these
knowledge-based exit criteria. We think this a positive first step, and
we are recommending that the Secretary of Homeland Security ensure that
the knowledge-based approach is fully implemented throughout the course
of its counter-MANPADS development program. DHS fully concurred.
DHS Faces Challenges in Adapting Military Counter-MANPADS System to
Commercial Aircraft:
In proceeding through Phases I and II of the counter-MANPADS
development program, DHS intends to establish system requirements,
mature technology and design, and set reliable cost estimates. Such
issues are interrelated and their resolution will have a direct impact
on DHS's ability to effectively implement its program. A brief
discussion of these program issues follows:
Requirements involving new technologies, system maintenance, system
integration, and system security for the counter-MANPADS system are to
be developed and this may involve trade-offs between competing
objectives. For example, DHS intends to study the trade-off of system
performance objectives with total cost to derive the most effective
solution at a realistic life-cycle cost.
Technology and design problems include a high false alarm rate
affecting the current generation of Missile Warning Systems (MWS) used
by the military; changes needed to adapt military countermeasures to
commercial aircraft; and the use of classified jam codes by civilian
aircrews. MWS's current high false alarm rate may increase system
failures. Whenever the MWS detects a missile launch, the system cycles.
If a false alarm has been received and the system is cycling
unnecessarily, it will reach its mean-time-between-failure (MTBF) rate
threshold much faster. A new MWS that would resolve this issue is being
developed but is not yet mature. The solicitation also requires a "call
back notification system" that would alert air traffic controllers once
it detects a missile launch, but it has not been developed. A MWS with
a high false alarm rate connected to a call back system could cause
unnecessary airport closures.
The DHS solicitation also requires a system reliability MTBF rate of
greater than 3,000 hours--10 times the current rate for DIRCM systems
on military aircraft of 300 hours. A low reliability rate drives the
operations and support costs and the ability of the airlines to
maintain the system. DHS also intends to require a common system
attached to the wide variety of commercial aircraft, even though the
design, placement, and integration of this system are all unknowns that
will affect each aircraft type differently. For example, a system
container that causes a 1-percent excess drag on a Boeing 747 aircraft
will cause greater drag on a smaller 737-model aircraft, which will
affect the airlines' fuel consumption and increase costs.
Design issues surround the classified jam codes used in DIRCM. Military
pilots are cleared to handle the classified material and military
aircraft are safeguarded, but DHS has not yet developed a plan to
handle the classified material and safeguard the hardware on a
commercial aircraft. According to DHS officials, any system developed
will need to have a tamper-proof design.
Operational test requirements are not clearly specified by the
solicitation. DHS compressed its development cycle to comply with
congressional direction. According to program officials, DHS has acted
to reduce the time and cost required to conduct testing by having the
contractor conduct required testing. However, DHS intends to approve
both test concepts and the test plans. They stated that DHS will
closely monitor all contractor conducted testing to ensure tests are
correctly executed and accurately reported. Finally, operational
testing will be conducted on DOD ranges using its test facilities. DOD
uses independent live fire and operational test and evaluation that are
not under the control of the developers to demonstrate that the
developed system is suitable and effective before authorizing full-
scale production.
Reliable cost estimates regarding the procurement, integration,
operation, and support of DIRCM system on commercial aircraft do not
currently exist. DHS intends to conduct its own cost estimates and also
require independent analysis, which will be difficult because (1) DHS
does not know how many units will be required and, therefore, cannot
determine a price based on economies of scale; (2) costs for
integrating the system on different types of aircraft are not yet
determined; and (3) the reliability rate is unknown and therefore
operations and support costs are difficult to estimate.
In addition, the production of a large number of countermeasure systems
quickly would probably require a significant capital investment to
increase production capacity. An industry official placed their current
production capability at roughly four DIRCM systems per month.
According to a program official, DHS does not currently know how many
or what combination of the estimated 6,800 aircraft in the U.S. fleet
might be equipped. One option would be to initially equip the planes of
the Civil Reserve Air Fleet (CRAF),[Footnote 2] but doing even that
could take years.
Knowledge-based Approach Has Been Adopted by Successful Product
Developers:
In the last several years, we have undertaken a body of work on how
leading developers in industry and government use a knowledge-based
approach to develop products that reduces risks and increases the
likelihood of successful outcomes. This best practices model enables
decision makers to be reasonably certain about their products at three
critical junctures or knowledge points during development and helps
them make informed investment decisions.
Knowledge Point 1: Before product development is started, a match must
be made between the customer's needs and the available resources--which
include technical and engineering knowledge, time, and funding.
Knowledge Point 2: A product's design must be able to meet performance
requirements and become stable about midway through development.
Knowledge Point 3: The developer must show that the product can be
manufactured within cost, schedule, and quality targets and is
demonstrated to be reliable before production begins.
A knowledge-based approach also involves the use of controls or exit
criteria to ensure that the required knowledge has been attained at
each critical juncture. It ensures that managers will (1) conduct
activities to capture relevant product development knowledge, (2)
provide evidence that knowledge was captured, and (3) hold decision
reviews to determine that appropriate knowledge was captured to allow a
move to the next phase. If the knowledge attained at each juncture does
not confirm the business case on which the effort was originally
justified, the program does not go forward.
Use of a knowledge-based approach has enabled leading organizations to
deliver high quality products on time and within budget. Product
development efforts that have not followed a knowledge-based approach
can be frequently characterized by poor cost, schedule, and performance
outcomes.
DHS Has Begun to Use Knowledge-based Approach:
DHS included a number of knowledge-based elements in its original
solicitation released on October 3, 2003. Among other things, it plans
to conduct design reviews, require periodic performance assessments
from the contractor, utilize an integrated product team to identify and
resolve issues, conduct systems engineering work in both phases, and
require the development of a prototype to help identify and resolve
specific design and manufacturing risks. In terms of systems
engineering, for example, DHS intends to use a comprehensive, iterative
technical management process that includes translating operational
requirements into configured systems, integrating technical inputs,
managing interfaces, and characterizing and managing risks.
During our review, we asked DHS to identify its controls or exit
criteria for use in determining whether needed knowledge had been
attained by the end of Phases I and II of its program. In its
solicitation of October 3, 2003, DHS had required the contractor to
satisfy certain criteria in order to receive payment for each
milestone. However, the Phase I exit criteria were not knowledge-based.
Rather, they were based on the contractor providing information, such
as the Long Lead Items List, at key payment milestones. They did not
require the contractor to demonstrate that key product knowledge has
been obtained. Also, the Phase II exit criteria were not identified and
were to be proposed by the contractor and subject to negotiation.
We presented DHS officials with recommended exit criteria from our past
reports (see examples in encl. I, p. 20), and they agreed to integrate
them into an updated solicitation and use them in monitoring the
contractors' progress. For example, at Knowledge Point 1, exit criteria
include the demonstration that critical technologies are mature and
system requirements are finalized. At Knowledge Point 2, criteria
include the completion of 90 percent of engineering drawings at design
review and the demonstration that a prototype's design meets
requirements. At Knowledge Point 3, criteria include the demonstration
that manufacturing processes are under statistical control.
Conclusion:
To address the significant challenges involved in adapting a military
counter-MANPADS system to commercial aircraft, DHS would benefit from
fully adopting the knowledge-based approach used by leading developers
in government and industry. This approach is predicated on the use of
exit criteria at each phase of the development process to ensure that
needed knowledge is attained before proceeding to the next phase. To
their credit, DHS officials responsible for this effort have agreed to
this approach, successfully incorporated exit criteria into their
updated solicitation, and agreed to use them to monitor
progress.[Footnote 3] This is a positive first step, but DHS needs to
ensure that the knowledge-based approach is fully utilized throughout
this development effort.
Recommendation for Executive Action:
We recommend that the Secretary of Homeland Security fully adopt the
knowledge-based approach, including the use of exit criteria, to help
ensure that key decisions in DHS's effort to develop and demonstrate a
counter-MANPADS system are based on sufficient information.
Agency Comments and Our Evaluation:
In written comments to a draft of this report (see encl. II), DHS fully
concurred with our findings and recommendation. DHS also provided
separate technical comments, which we have incorporated as appropriate.
Scope and Methodology:
In conducting our review, we compared DHS's plan for its counter-
MANPADS system development and demonstration program plan against the
best practices of commercial and military acquisitions identified in
our past reports and focused whether DHS will have sufficient
information to make knowledge-based decisions at each milestone. To
determine what military countermeasures are available for adaptation to
commercial aircraft and what their performance capabilities, cost, and
schedule ramifications might be, we met with DOD, Air Force, Army, and
Navy officials and analyzed relevant documentation, including studies
and test reports. We interviewed representatives from Northrop Grumman,
Boeing, BAE Systems, Raytheon Corporation, and Sanders Design
International regarding countermeasure systems currently in production
or development. We also met with representatives from the airline
industry, air transportation association, and RAND. We conducted our
work from April 2003 through January 2004 in accordance with generally
accepted government auditing standards.
We are sending copies of this report to the Secretary, Department of
Homeland Security; the Director, Office of Management and Budget; and
other interested congressional committees. Copies are available to
others upon request. In addition, the report will be available at no
charge on the GAO Web site at http://www.gao.gov.
Should you or your staff have any questions on matters discussed in
this report, please contact me on (202) 512-4841 or Jim Morrison at
(202) 512-7078. Principal contributors to this report were Mike Aiken,
Natalie Britton, Terry Parker, and Richard Strittmatter.
Robert E. Levin:
Director, Acquisition and Sourcing Management:
Signed by Robert E. Levin:
Enclosures:
[See PDF for image]
[End of figure]
Comments from the Department of Homeland Security:
U.S. Department of Homeland Security
Washington, DC 20528:
Homeland Security:
January 22, 2004:
R.E. Levin:
Director, Acquisition and Sourcing Management
United States General Accounting Office:
Dear Mr. Levin:
Thank you for the opportunity to comment on your GAO Draft Report,
dated January 12, 2004, titled: The Department of Homeland Security
Needs to Fully Adopt a Knowledge-based Approach to Its Counter-MANPADS
Development Program. As you know, the Department of Homeland Security
considers the Counter-MANPADS program one of its vital initiatives in
air transportation security for the American people. We appreciate the
GAO's participation effort, and your insights and suggestions in the
execution of the program. As we launch into the validation phase of
transitioning existing military technologies to civilian aircraft,
there are many technical, schedule, and cost unknowns.
We fully concur with your assessment that knowledge-based evaluations
of technical requirements, schedules, total system cost estimates, etc.
are essential to the success of this program. In fact, each of the
contractors selected to perform phase one of this effort have been told
the Government evaluators are employing knowledge-based evaluations
throughout the program. Based on the GAO's preliminary recommendations,
we had developed preliminary milestone exit criteria for the
solicitation.
For your consideration we have included two enclosures: the first is a
formal response to your recommendation that C-MANPADS Program Office
use a knowledge-based management approach. The second attachment
contains comments or suggests word or phrase changes that we hope would
prove useful in clarifying the C-MANPADS SPO's direction.
Again, thank you for the opportunity to comment.
Respectfully,
Dr. Charles E. McQueary, Under Secretary for Science and Technology:
Signed by Dr. Charles E. McQueary:
Enclosures: As stated:
GAO DRAFT REPORT - DATED JANUARY 12, 2004 (GAO-04-341R):
"DHS NEEDS TO FULLY ADOPT A KNOWLEDGE-BASED APPROACH TO ITS COUNTER-
MANPADS DEVELOPMENT PROGRAM":
DHS COMMMENTS ON THE GAO RECOMMENATION:
RECOMMENDATION: GAO recommends that the Secretary of Homeland Security
fully adopt the knowledge-based approach, including the use of exit
criteria, to help ensure that key decisions in DHS' effort to develop
and demonstrate a counter-MANPADS system are based on sufficient
information. (p. 8/GAO draft report):
DHS RESPONSE: Concur. As indicated in subject report, the DHS Counter-
MANPADS Special Program Office (SPO) began to incorporate knowledge-
based exit criteria into Counter-MANPADS solicitation based on input
received from GAO during the course of their review. DHS remains
committed to the adoption of a knowledge-based approach throughout the
program and has established knowledge-based entrance and exit criteria
as an integral part of each milestone.
[End of section]
FOOTNOTES
[1] House Report 108-76, p. 84.
[2] The CRAF Program enlists the U.S. airline industry to help create
an increased airlift capability for contingencies. Airlines sign
contracts with the government entitling Air Mobility Command (AMC) to
mobilize the aircraft and their aircrews when airlift needs exceed the
capability of military aircraft.
[3] Examples are included on page 20 of the enclosed briefing.
