Aviation Security
DHS and TSA Have Researched, Developed, and Begun Deploying Passenger Checkpoint Screening Technologies, but Continue to Face Challenges Gao ID: GAO-10-128 October 7, 2009Since fiscal year 2002, the Transportation Security Administration (TSA) and the Department of Homeland Security (DHS) have invested over $795 million in technologies to screen passengers at airport checkpoints. The DHS Science and Technology Directorate (S&T) is responsible, with TSA, for researching and developing technologies, and TSA deploys them. GAO was asked to evaluate the extent to which (1) TSA used a risk-based strategy to prioritize technology investments; (2) DHS researched, developed, and deployed new technologies, and why deployment of the explosives trace portal (ETP) was halted; and (3) DHS coordinated research and development efforts with key stakeholders. To address these objectives, GAO analyzed DHS and TSA plans and documents, conducted site visits to research laboratories and nine airports, and interviewed agency officials, airport operators, and technology vendors.
TSA completed a strategic plan to guide research, development, and deployment of passenger checkpoint screening technologies; however, the plan is not risk-based. According to TSA officials, the strategic plan and its underlying strategy for the Passenger Screening Program were developed using risk information, such as threat information. However, the strategic plan and its underlying strategy do not reflect some of the key risk management principles set forth in DHS's National Infrastructure Protection Plan (NIPP), such as conducting a risk assessment based on the three elements of risk--threat, vulnerability, and consequence--and developing a cost-benefit analysis and performance measures. TSA officials stated that, as of September 2009, a draft risk assessment for all of commercial aviation, the Aviation Domain Risk Assessment, was being reviewed internally. However, completion of this risk assessment has been repeatedly delayed, and TSA could not identify the extent to which it will address all three elements of risk. TSA officials also stated that they expect to develop a cost-benefit analysis and establish performance measures, but officials could not provide timeframes for their completion. Without adhering to all key risk management principles as required in the NIPP, TSA lacks assurance that its investments in screening technologies address the highest priority security needs at airport passenger checkpoints. Since TSA's creation, 10 passenger screening technologies have been in various phases of research, development, test and evaluation, procurement, and deployment, but TSA has not deployed any of these technologies to airports nationwide. The ETP, the first new technology deployment initiated by TSA, was halted in June 2006 because of performance problems and high installation costs. Deployment has been initiated for four technologies--the ETP in January 2006, and the advanced technology systems, a cast and prosthesis scanner, and a bottled liquids scanner in 2008. TSA's acquisition guidance and leading commercial firms recommend testing the operational effectiveness and suitability of technologies or products prior to deploying them. However, in the case of the ETP, although TSA tested earlier models, the models ultimately chosen were not operationally tested before they were deployed to ensure they demonstrated effective performance in an operational environment. Without operationally testing technologies prior to deployment, TSA does not have reasonable assurance that technologies will perform as intended. DHS coordinated with stakeholders to research, develop, and deploy checkpoint screening technologies, but coordination challenges remain. Through several mechanisms, DHS is taking steps to strengthen coordination within the department and with airport operators and technology vendors.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-10-128, Aviation Security: DHS and TSA Have Researched, Developed, and Begun Deploying Passenger Checkpoint Screening Technologies, but Continue to Face Challenges
This is the accessible text file for GAO report number GAO-10-128
entitled 'Aviation Security: DHS and TSA Have Researched, Developed,
and Begun Deploying Passenger Checkpoint Screening Technologies, but
Continue to Face Challenges' which was released on October 29, 2009.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
October 2009:
Aviation Security:
DHS and TSA Have Researched, Developed, and Begun Deploying Passenger
Checkpoint Screening Technologies, but Continue to Face Challenges:
GAO-10-128:
GAO Highlights:
Highlights of GAO-10-128, a report to congressional requesters.
Why GAO Did This Study:
Since fiscal year 2002, the Transportation Security Administration
(TSA) and the Department of Homeland Security (DHS) have invested over
$795 million in technologies to screen passengers at airport
checkpoints. The DHS Science and Technology Directorate (S&T) is
responsible, with TSA, for researching and developing technologies, and
TSA deploys them. GAO was asked to evaluate the extent to which (1) TSA
used a risk-based strategy to prioritize technology investments; (2)
DHS researched, developed, and deployed new technologies, and why
deployment of the explosives trace portal (ETP) was halted; and (3) DHS
coordinated research and development efforts with key stakeholders. To
address these objectives, GAO analyzed DHS and TSA plans and documents,
conducted site visits to research laboratories and nine airports, and
interviewed agency officials, airport operators, and technology
vendors.
What GAO Found:
TSA completed a strategic plan to guide research, development, and
deployment of passenger checkpoint screening technologies; however, the
plan is not risk-based. According to TSA officials, the strategic plan
and its underlying strategy for the Passenger Screening Program were
developed using risk information, such as threat information. However,
the strategic plan and its underlying strategy do not reflect some of
the key risk management principles set forth in DHS‘s National
Infrastructure Protection Plan (NIPP), such as conducting a risk
assessment based on the three elements of risk”threat, vulnerability,
and consequence”and developing a cost-benefit analysis and performance
measures. TSA officials stated that, as of September 2009, a draft risk
assessment for all of commercial aviation, the Aviation Domain Risk
Assessment, was being reviewed internally. However, completion of this
risk assessment has been repeatedly delayed, and TSA could not identify
the extent to which it will address all three elements of risk. TSA
officials also stated that they expect to develop a cost-benefit
analysis and establish performance measures, but officials could not
provide timeframes for their completion. Without adhering to all key
risk management principles as required in the NIPP, TSA lacks assurance
that its investments in screening technologies address the highest
priority security needs at airport passenger checkpoints.
Since TSA‘s creation, 10 passenger screening technologies have been in
various phases of research, development, test and evaluation,
procurement, and deployment, but TSA has not deployed any of these
technologies to airports nationwide. The ETP, the first new technology
deployment initiated by TSA, was halted in June 2006 because of
performance problems and high installation costs. Deployment has been
initiated for four technologies”the ETP in January 2006, and the
advanced technology systems, a cast and prosthesis scanner, and a
bottled liquids scanner in 2008. TSA‘s acquisition guidance and leading
commercial firms recommend testing the operational effectiveness and
suitability of technologies or products prior to deploying them.
However, in the case of the ETP, although TSA tested earlier models,
the models ultimately chosen were not operationally tested before they
were deployed to ensure they demonstrated effective performance in an
operational environment. Without operationally testing technologies
prior to deployment, TSA does not have reasonable assurance that
technologies will perform as intended.
DHS coordinated with stakeholders to research, develop, and deploy
checkpoint screening technologies, but coordination challenges remain.
Through several mechanisms, DHS is taking steps to strengthen
coordination within the department and with airport operators and
technology vendors.
What GAO Recommends:
GAO recommends, among other things, that TSA (1) conduct a risk
assessment and develop a cost–benefit analysis and performance measures
for passenger screening technologies, and (2) to the extent feasible,
ensure that technologies have completed operational tests and
evaluations before they are deployed. DHS concurred with the
recommendations; however, its implementation plans do not fully address
six of the eight recommendations in the report.
View [hyperlink, http://www.gao.gov/products/GAO-10-128] or key
components. For more information, contact Steve Lord at (202) 512-8777
or lords@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
TSA Has Taken Actions to Prioritize Investments in Passenger Checkpoint
Screening Technologies, but Lacks a Risk-Based Strategy:
Ten New Checkpoint Screening Technologies Are in Various Phases of
RDT&E, Procurement, and Deployment, but ETP Deployment Has Been Halted:
DHS Is Addressing Coordination and Collaboration Challenges with
Stakeholders to Research, Develop, and Deploy Checkpoint Screening
Technologies:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Comments from the Department of Homeland Security:
Appendix III: GAO Contact and Staff Acknowledgments:
Figures:
Figure 1: TSA Passenger Checkpoint Screening Functions:
Figure 2: NIPP Risk Management Framework:
Figure 3: Status of Six Checkpoint Screening Technologies that Had
Initiated Procurement and/or Deployment as of September 2008:
Abbreviations:
ADRA: Aviation Domain Risk Assessment:
AMS: Acquisition Management System:
ATSA: Aviation and Transportation Security Act:
DHS: Department of Homeland Security:
ETP: explosives trace portal:
FAA: Federal Aviation Administration:
IED: improvised explosive device:
IPT: Integrated Product Team:
MOU: memorandum of understanding:
NIPP: National Infrastructure Protection Plan:
PSP: Passenger Screening Program:
RDT&E: research, development, test and evaluation:
RMAT: Risk Management and Analysis Toolset:
S&T: Science and Technology Directorate:
TSA: Transportation Security Administration:
TSL: Transportation Security Laboratory:
TSO: Transportation Security Officer:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
October 7, 2009:
The Honorable James L. Oberstar:
Chairman:
Committee on Transportation and Infrastructure:
House of Representatives:
The Honorable Bennie G. Thompson:
Chairman:
Committee on Homeland Security:
House of Representatives:
The Honorable Jerry F. Costello:
Chairman:
Subcommittee on Aviation:
Committee on Transportation and Infrastructure:
House of Representatives:
Commercial aircraft have long been a target of terrorism at the hands
of hijackers and suicide bombers. The Transportation Security
Administration (TSA), the agency with primary responsibility for
securing the nation's civil aviation system after the September 11,
2001, terrorist attacks, has identified the need for improved
technology to detect explosives and other threat items at airport
passenger screening checkpoints to strengthen the nation's defenses
against acts of terrorism. From fiscal years 2002 through 2008, over
$795 million has been invested by TSA and the Department of Homeland
Security (DHS) for the research, development, test and evaluation
(RDT&E), procurement, and deployment of checkpoint screening
technologies.
TSA has implemented a multilayered system of security to protect
commercial aviation--the most publicly visible layer being the physical
screening of passengers and their carry-on items at airport screening
checkpoints. TSA's passenger checkpoint screening system--located at
all airports regulated by TSA--is comprised of three elements: (1) the
personnel, or screeners, responsible for conducting the screening of
airline passengers and their carry-on items; (2) the procedures
screeners are to follow to conduct screening; and (3) the technology
used during the screening process. Collectively, these elements--the
people, process and technology--help to determine the effectiveness and
efficiency of passenger checkpoint screening.[Footnote 1] We previously
reported that TSA had made efforts to enhance its passenger checkpoint
screening system by strengthening screener training, measuring the
performance of screeners and the screening system, and modifying
screening procedures to address terrorist threats.[Footnote 2]
Within DHS, the Science and Technology Directorate (S&T) and TSA have
responsibilities for researching, developing, and testing and
evaluating new technologies, including airport checkpoint screening
technologies. Specifically, S&T is responsible for the basic and
applied research and advanced development of new technologies, while
TSA, through its Passenger Screening Program (PSP), identifies the need
for new checkpoint screening technologies, provides input to S&T during
the research and development of new technologies, which TSA then
procures and deploys.[Footnote 3]
In 2004, we reviewed DHS's investments in the research and development
of technologies to secure the transportation sector, including
aviation, and found that DHS needed to strengthen the management of its
research and development efforts.[Footnote 4] In October 2007, we
testified that a key challenge related to securing the homeland
involves allocating resources based on risk.[Footnote 5] DHS and TSA
leadership have identified that risk-informed considerations will be a
cornerstone of departmental and agency policy. In particular, DHS's
National Infrastructure Protection Plan (NIPP) stated that TSA should
be considering risk management principles when allocating funding for
the research and development of security technologies. According to the
NIPP, security strategies should be informed by, among other things, a
risk assessment that includes threat, vulnerability, and consequence
assessments, information such as cost-benefit analyses to prioritize
investments, and performance measures to assess the extent to which a
strategy reduces or mitigates the risk of terrorist attacks.
In response to your request, this report provides the results of our
review of DHS's efforts, through S&T and TSA, to research, develop, and
deploy emerging screening technologies for use at airport passenger
checkpoints by addressing the following questions: (1) To what extent
has TSA developed a risk-informed strategy to prioritize investments in
the research and development of passenger checkpoint screening
technologies; (2) What new passenger checkpoint screening technologies
has DHS researched, developed, tested and evaluated, procured, and
deployed since its creation, and why did TSA halt the first technology
deployment that it initiated--the explosives trace portal (ETP); and
(3) To what extent has DHS coordinated the RDT&E, procurement, and
deployment of passenger checkpoint screening technologies internally
and with key stakeholders, such as airport operators and technology
vendors?
This report is a public version of a restricted report (GAO-09-21SU)
that we provided to you earlier this year. In this report in three
cases we provide updates regarding the Aviation Domain Risk Assessment
(ADRA), the NIPP, and the number of ETPs in airports.[Footnote 6] DHS
and TSA deemed some of the information in the restricted report to be
sensitive security information, which must be protected from public
disclosure. Although this report omits that information, such as
specific details associated with the methods and results of testing
during the research and development of the ETPs, it addresses the same
questions as the restricted report. Also, the overall methodology used
for both reports is the same.
To determine the extent to which TSA developed a risk-informed strategy
to prioritize investments in the research and development of new
checkpoint technologies, we analyzed program documents, including TSA's
August 2008 strategic plan for checkpoint technologies, technology
project plans, and budget documents. We also compared TSA's strategic
plan and DHS's responses regarding their efforts to develop a risk-
informed strategy for their research and development investments with
DHS's guidance on using risk management principles to prioritize
investments and allocate resources.
To determine what new passenger checkpoint screening technologies DHS
has researched, developed, tested and evaluated, procured, and
deployed, and the reasons why TSA halted the first technology for which
it initiated deployment--the ETP, we analyzed TSA's strategic plan,
TSA's PSP documentation, technical and operational requirements for new
technologies, laboratory test reports, and testing data from
operational pilots. Additionally, we interviewed TSA and S&T officials
to obtain information on current technologies being researched,
developed, and deployed, and conducted site visits to the
Transportation Security Laboratory (TSL) and Tyndall Air Force Base to
observe testing of new checkpoint technologies. We visited the TSL
because that is where S&T tests and evaluates transportation
technologies, including checkpoint screening technologies. We visited
Tyndall Air Force Base because technologies to detect bottled liquids
explosives were being tested there. We also interviewed TSA
headquarters officials and senior TSA officials from the airports where
TSA had initially deployed or planned to deploy the ETPs, including 29
Federal Security Directors, 1 Deputy Federal Security Director, and 5
Assistant Federal Security Directors for Screening.[Footnote 7] We
chose these officials because they are the senior TSA officials in
charge of security and managing TSA's role in deploying new
technologies at the airport. We also visited nine airports and selected
these locations based on the technologies that had been deployed or
were being tested on site, their geography, size, and proximity to
research and development laboratories. Of the nine airports we visited,
the ETPs had been deployed or were to be deployed to all of them and
other new checkpoint screening technologies were undergoing pilot
demonstrations or testing at two of them. We visited four airports on
the east coast, three airports on the west coast, and two airports
located in the west and southwestern regions of the United States. We
selected these locations because they represented small-, medium-, and
large-sized airports and different regions in the United States.
To determine the extent to which TSA coordinated and collaborated
internally and with key external stakeholders--airport operators and
technology vendors--on the RDT&E, procurement, and deployment of
checkpoint technologies, we analyzed program documents, including a
memorandum of understanding (MOU) between S&T and TSA. Additionally, we
interviewed S&T and TSA officials, seven checkpoint technology vendors,
and airport operators[Footnote 8] and other officials at 40 airports
where ETPs had initially been or were to be deployed. Because we
selected a nonprobability sample of airports to visit and officials to
interview there, we cannot generalize the results of what we learned to
airports nationwide. However, the information we gathered from these
locations--insights based on observing airport operations and on
perspectives of officials who were involved with DHS's efforts to
operationally test, evaluate, and deploy checkpoint technologies--
could only be obtained through direct observation or from officials
stationed at these select sites where technologies were being deployed
and tested. We also selected a nonprobability sample of 8 out of the
157 total requirements for the ETP to determine whether some of its key
requirements had been tested prior to procuring and deploying the
machines.[Footnote 9] In addition, we reviewed S&T's and TSA's
coordination and collaboration activities and compared them to TSA
program guidance and leading practices for collaborating agencies
regarding communication, planning, and federal coordination internally
and with external stakeholders.[Footnote 10] Appendix I contains
additional information on the objectives, scope, and methodology of our
review.
We conducted this performance audit from June 2006 through April 2009,
with some updated information as of September 2009 as previously
disclosed, in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objectives.
We believe that the evidence obtained provides a reasonable basis for
our findings and conclusions based on our audit objectives.
Results in Brief:
TSA completed a strategic plan in August 2008, which identified a
strategy to invest in the RDT&E, procurement, and deployment of
passenger checkpoint screening technologies; however, the plan and its
underlying strategy are not risk informed. TSA's strategy does not
incorporate some key risk management principles--a risk assessment,
cost-benefit analysis, and performance measures--as required by the
NIPP. To guide investments in checkpoint screening technologies, TSA
officials stated that they consider risks to the checkpoint by
analyzing threat information and other factors. However, this approach
does not address all three risk elements required by the NIPP, which
specifies that risk assessments are to be based on threat,
vulnerability, and consequence assessments. Officials stated that they
have drafted the Aviation Domain Risk Assessment (ADRA), a risk
assessment of the entire aviation sector, including the passenger
checkpoint, which is to include an assessment of all three risk
elements. TSA officials anticipated finalizing the ADRA in February
2008, but have postponed its completion multiple times. As of September
2009, officials expected completion of the ADRA by the end of calendar
year 2009, but could not identify the extent to which the ADRA would
address risks to the checkpoint. Therefore, we could not determine when
the ADRA will be completed, to what extent it will incorporate all
three elements of a risk assessment, and whether it will identify and
assess risks to the checkpoint. In addition, TSA officials stated that
they have not yet conducted a cost-benefit analysis to set priorities
for the PSP, or established performance measures that assess how
deployed technologies have reduced or mitigated risk, as required by
the NIPP. Officials acknowledged that a cost-benefit analysis and
performance measures should be completed; however, they could not
provide timeframes for completing them. Without incorporating these DHS
risk management principles into the PSP strategy, TSA cannot ensure
that it is targeting the highest priority security needs at
checkpoints; measure the extent to which deployed technologies reduce
the risk of terrorist attacks; or make needed adjustments to its PSP
strategy.
S&T and TSA have placed 10 new checkpoint screening technologies in
various phases of RDT&E, procurement, and deployment, but halted the
deployment of the ETP due to performance problems and high installation
costs. TSA has initiated, but not yet completed, deployments of 4 of
the 10 technologies; initiated procurements, but not yet deployed, 2
more technologies, including the Whole Body Imager; and has 4
additional technologies, including a shoe scanner, in research and
development. In 2006, TSA deployed 101 ETPs to airports, the first
deployment of a checkpoint technology initiated by the agency.[Footnote
11] The ETP was deployed even though TSA officials were aware that
tests conducted during 2004 and 2005 on earlier ETP models suggested
they did not demonstrate reliable performance in an airport
environment. Furthermore, the ETP models that were subsequently
deployed were not first tested to prove their effective performance in
an operational environment, contrary to TSA's acquisition guidance,
which recommends such testing. As a result, TSA lacked assurance that
the ETP would meet its functional requirements in airports. TSA
officials stated that they deployed the machines without resolving
these issues to respond quickly to the threat of suicide bombers. After
being deployed, the ETPs broke down frequently and were more expensive
to maintain than expected, according to TSA officials. TSA continued to
use them at checkpoint lanes even though TSA could not identify whether
ETPs were more effective than existing screening procedures. In the
future, using validated technologies would enhance TSA's efforts to
improve checkpoint security.
DHS S&T and TSA share responsibilities related to the RDT&E,
procurement, and deployment of checkpoint screening technologies, and
have coordinated and collaborated with each other and key external
stakeholders; however, coordination and collaboration challenges remain
that DHS is addressing. The Homeland Security Act of 2002 and the
Aviation and Transportation Security Act, which established DHS and
TSA, respectively, each address the need for coordination and
collaboration with stakeholders. S&T and TSA coordination efforts
include a 2006 memorandum of understanding for using the TSL, and the
establishment of the Capstone Integrated Product Team for Explosives
Prevention in 2006 to help DHS, TSA, and the U.S. Secret Service to,
among other things, identify priorities for explosives prevention.
However, S&T and TSA officials stated that some technology projects
were delayed because TSA had not consistently communicated clear
requirements to S&T to test technologies, and S&T had not consistently
communicated to TSA about projects at the TSL or the time frames to
complete them. According to S&T and TSA officials, coordination and
collaboration between them has improved since the summer of 2007. TSA
has also taken steps to build partnerships with airport operators and
technology vendors, such as by hosting conferences with them; however,
the agency has not established a systematic process for coordinating
with these stakeholders related to passenger checkpoint technologies.
For example, 11 of 33 airport operators[Footnote 12] and 4 of 7 vendors
we interviewed told us that TSA had not solicited or shared information
with them regarding checkpoint technology needs and priorities. TSA
officials acknowledged the need to improve relationships with external
stakeholders. According to TSA officials, an Industry Outreach Manager
was hired in 2007 and a draft communications plan to provide guidance
and a more systematic process to coordinate with these stakeholders is
being reviewed, but no completion date could be provided.
To help ensure that DHS's S&T and TSA take a comprehensive, risk-
informed approach to the RDT&E, procurement, and deployment of
passenger checkpoint screening technologies, and to increase the
likelihood of successful procurements and deployments of such
technologies, in the restricted version of this report, we recommended
that TSA conduct a complete risk assessment, including threat,
vulnerability and consequence assessments, that would apply to the PSP;
develop cost-benefit analyses to assist in prioritizing investments in
new checkpoint technologies; develop quantifiable performance measures
to assess the extent to which investments in research and development
have mitigated the risks of a terrorist attack; determine if changes
need to be made to the PSP strategy as a result of the risk assessment,
cost-benefit analyses, and performance measures; to the extent
feasible, ensure that operational testing has been successfully
completed before deploying checkpoint technologies to airports; and
evaluate the benefits and costs of the ETPs currently being used in
airports in order to determine whether it is cost effective to continue
to use the machines. In written comments on our report, DHS stated that
it agreed with our recommendations and identified actions planned or
underway to implement them. While DHS is taking steps to address our
first and second recommendations related to conducting a risk
assessment and cost-benefit analysis, the actions DHS reported TSA had
taken or plans to take do not fully address the intent of the remaining
recommendations. DHS also provided us with technical comments, which we
considered and incorporated in the report where appropriate. In
particular, we clarified the wording of a recommendation which
originally stated that TSA should develop quantifiable performance
measures to assess the extent to which investments in checkpoint
screening technologies mitigated the risks of a terrorist attack. We
altered the wording to state that performance measures should be
developed to assess progress towards security goals.
Background:
TSA's airport passenger checkpoint screening system is comprised of
three elements: the (1) personnel, or screeners, responsible for
operating the checkpoint, including the screening of airline passengers
and their carry-on items; (2) standard operating procedures that
screeners are to follow to conduct screening; and (3) technology used
during the screening process. Collectively, these elements determine
the effectiveness and efficiency of passenger checkpoint screening. In
strengthening one or more elements of its checkpoint screening system,
TSA aims to balance its security goals with the need to efficiently
process passengers. We previously reported that TSA had made progress
in enhancing its passenger checkpoint screening system by strengthening
screener training, measuring the performance of screeners and the
screening system, and modifying screening procedures to address
terrorist threats and efficiency concerns.[Footnote 13] We made
recommendations to DHS designed to strengthen TSA's efforts to train
screeners, modify screening standard operating procedures, and measure
the performance of the checkpoint screening system. DHS generally
agreed with our recommendations and TSA has taken steps to implement
them.
Passenger Checkpoint Screening Process:
Passenger screening is a process by which screeners inspect individuals
and their property to deter and prevent an act of violence or air
piracy, such as the carriage of any unauthorized explosive, incendiary,
weapon, or other prohibited item onboard an aircraft or into a sterile
area.[Footnote 14] Screeners inspect individuals for prohibited items
at designated screening locations. TSA developed standard operating
procedures and the process for screening passengers at airport
checkpoints. Figure 1 illustrates the screening functions at a typical
passenger checkpoint.
Figure 1: TSA Passenger Checkpoint Screening Functions:
[Refer to PDF for image: illustration]
The following are indicated on the illustration:
Video surveillance:
Physical barriers (walls/partitions):
Walk-though metal detector (passenger screening function):
X-ray screening (passenger screening function):
Manual or ETD searches[A] (passenger screening function); Only if
passenger is identified or randomly selected for additional screening
or if screener identifies a potential prohibited item on X-ray:
Hand-wand or pat-down (passenger screening function); Only if passenger
is identified or randomly selected for additional screening because he
or she met certain criteria or alarmed the walk-through metal detector.
Source: GAO and Nova Development Corporation.
[A] Explosives trace detection (ETD) machines detect small amounts of
explosives on or in passenger's carry-on items. ETDs work by detecting
vapors and residues of explosives. Human operators collect samples by
rubbing bags with swabs, which are chemically analyzed to identify any
traces of explosives material.
[End of figure]
Primary screening is conducted on all airline passengers prior to
entering the sterile area of an airport and involves passengers walking
through a metal detector and carry-on items being subjected to X-ray
screening. Passengers who alarm the walk-through metal detector or are
designated as selectees--that is, passengers selected for additional
screening--must then undergo secondary screening,[Footnote 15] as well
as passengers whose carry-on items have been identified by the X-ray
machine as potentially containing a prohibited item. Secondary
screening involves additional means for screening passengers, such as
by hand-wand, physical pat-down or, at certain airport locations, an
ETP, which is used to detect traces of explosives on passengers by
using puffs of air to dislodge particles from their body and clothing
into an analyzer. Selectees' carry-on items are also physically
searched or screened for explosives traces by Explosives Trace
Detection (ETD) machines.[Footnote 16] In addition, DHS S&T and TSA
have deployed and are pursuing additional technologies to provide
improved imaging or anomaly detection capacities to better identify
explosives and other threat objects.
Roles and Responsibilities for the RDT&E, Procurement, and Deployment
of Checkpoint Screening Technologies:
DHS and TSA share responsibility for the screening of passengers and
the research, development, and deployment of passenger checkpoint
screening technologies. Enacted in November 2001, the Aviation and
Transportation Security Act (ATSA) created TSA and charged it with the
responsibility of securing civil aviation, which includes the screening
of all passengers and their baggage.[Footnote 17] ATSA also authorized
funding to accelerate the RDT&E of new checkpoint screening
technologies. The Homeland Security Act of 2002, enacted in November
2002, established DHS, transferred TSA from the Department of
Transportation to DHS and, within DHS, established S&T to have primary
responsibility for DHS's RDT&E activities, and for coordinating and
integrating all these activities.[Footnote 18] The Intelligence Reform
and Terrorism Prevention Act of 2004 (Intelligence Reform Act), enacted
in December 2004, directed the Secretary of Homeland Security to give
high priority to developing, testing, improving, and deploying
checkpoint screening equipment that detects nonmetallic, chemical,
biological, and radiological weapons and explosives, in all forms, on
individuals and in their personal property.[Footnote 19]
Until fiscal year 2006, TSA had primary responsibility for investing in
the research and development of new checkpoint screening technologies,
and was responsible for developmental and operational test and
evaluation of new technologies.[Footnote 20] However, during fiscal
year 2006, research and development functions within DHS were
consolidated, for the most part, within S&T.[Footnote 21] After this
consolidation, S&T assumed primary responsibility for funding the
research, development, and developmental test and evaluation of airport
checkpoint screening technologies. S&T also assumed responsibility from
TSA for the Transportation Security Laboratory (TSL) which, among other
things, tests and evaluates technologies under development. TSA,
through the PSP that was transferred from the Federal Aviation
Administration (FAA) to TSA, continues to be responsible for
identifying the requirements for new checkpoint technologies;
operationally testing and evaluating technologies in airports; and
procuring, deploying, and maintaining technologies. This transfer of
responsibility from TSA to S&T did not limit TSA's authority to acquire
commercially available technologies for use at the checkpoint.
DHS and TSA's Processes for the RDT&E, Procurement, and Deployment of
Checkpoint Screening Technologies:
S&T and TSA's RDT&E, procurement, and deployment efforts are made up of
seven components: basic research, applied research, advanced
development, operational testing, procurement, operational integration,
and deployment. S&T is responsible for conducting basic and applied
research, and advanced development, including developmental test and
evaluation. TSA is responsible for conducting operational test and
evaluation, operational integration, procurement and deployment of new
technologies, including checkpoint screening technologies. These seven
components are described below.
* Basic research includes scientific efforts and experimentation
directed toward increasing knowledge and understanding in the fields of
physical, engineering, environmental, social, and life sciences related
to long-term national needs.
* Applied research includes efforts directed toward solving specific
problems with a view toward developing and evaluating the feasibility
of proposed solutions.
* Advanced development includes efforts directed toward projects that
have moved into the development of hardware and software for field
experiments and tests, such as acceptance testing.[Footnote 22]
* Operational test and evaluation verifies that new systems are
operationally effective, supportable, and suitable before deployment.
* Operational integration is the process employed to enable successful
transition of viable technologies and systems to the field environment.
* Procurement includes the efforts to obtain a product or service.
[Footnote 23]
* Deployment is a series of actions following the determination that
the product meets its requirements and is accepted by the program
manager and integrated product team; designated locations are
configured for product integration into the screening operating system
and the installed product passes site acceptance tests; and logistics
support is in place and all users are trained to use the product.
RDT&E, Procurement, and Deployment Funding for Checkpoint Screening
Technologies:
Over $795 million has been invested by DHS and TSA during fiscal years
2002 through 2008 for the RDT&E, procurement, and deployment of
checkpoint screening technologies. During this time, over $91 million
was invested in the RDT&E of checkpoint technologies and about $704
million was invested in the procurement and deployment of these
technologies. From fiscal years 2002 through 2005, TSA was responsible
for the RDT&E of checkpoint technologies; however, TSA officials could
not identify the amount of funding the agency invested for these
purposes during those years. After fiscal year 2005, TSA invested $14.5
million for test and evaluation of checkpoint technologies, but did not
fund the research and development of these technologies because
responsibility in general for research and development funding was
transferred from TSA to S&T beginning in fiscal year 2006. Therefore,
during fiscal years 2006 through 2008, S&T invested $77.0 million in
the RDT&E of checkpoint screening technologies. All of the
approximately $704 million for the procurement and deployment of
checkpoint screening technologies from fiscal years 2002 through 2008
was invested by TSA because the agency has been responsible for
procurement and deployment of these technologies since it was created.
Applying a Risk Management Approach to Checkpoint Technology
Investments:
Risk management is a tool that policy makers can use to help ensure
that strategies to develop protective programs and allocate resources
target the highest priority security needs. This information helps
officials determine which security programs are most important to
develop and fund, given that it is not possible to protect the country
against all threats because of limited resources. Law and related
policy, including the Intelligence Reform Act, the Implementing
Recommendations of the 9/11 Commission Act of 2007 (9/11 Commission
Act), and Homeland Security Presidential Directive 7, provide that
federal agencies with homeland security responsibilities are to apply
risk-informed principles to prioritize security needs and allocate
resources. Consistent with these provisions, DHS issued the National
Strategy for Transportation Security in 2005 that, among other things,
describes the policies that DHS is to apply when managing risks to the
security of the U.S. transportation system. Further, in June 2006, DHS
issued the NIPP, which provides a risk management framework to guide
strategies to develop homeland security programs and allocate resources
to them.[Footnote 24] According to the NIPP, its risk management
framework consists of six phases that help to identify and assess risks
and prioritize investments in programs, as illustrated in figure 2. The
NIPP designated TSA as the primary federal agency responsible for
coordinating critical infrastructure protection efforts within the
transportation sector.
Figure 2: NIPP Risk Management Framework:
[Refer to PDF for image: illustration]
Set Security Goals:
Identify Assets, Systems, Networks, and Functions:
Assess Risks:
Prioritize:
Implement Protective Programs:
Measure Effectiveness.
Source: DHS.
[End of figure]
A risk-informed strategy to develop and invest in critical
infrastructure protection, according to the NIPP, begins with setting
security goals. Setting security goals involves defining specific
outcomes, conditions, end points, or performance targets that
collectively constitute an effective protective posture. Once security
goals are established, decisionmakers are to identify what assets or
systems to protect and identify and assess the greatest risks to them,
that is, the type of terrorist attack that is most likely to occur and
that would result in the most severe consequences. Risk of a terrorist
attack, according to the NIPP, is to be assessed by analyzing
consequences of an attack; the threat--that is, the likelihood of an
attack; and the extent to which an asset or a system, in this case the
transportation system, is vulnerable to this type of attack.[Footnote
25] The potential consequences of any incident, including terrorist
attacks and natural or manmade disasters, is the first factor to be
considered in a risk assessment. In the context of the NIPP,
consequence is measured as the range of loss or damage that can be
expected in the event a terrorist attack succeeds. A consequence
assessment looks at the expected worst case or reasonable worst case
impact of a successful attack. A threat assessment is the
identification and evaluation of adverse events that can harm or damage
an asset and takes into account certain factors, such as whether the
intent and capability to carry out the attack exist. A vulnerability
assessment identifies weaknesses or characteristics of an asset or
system, such as its design and location, which make it susceptible to a
terrorist attack and that may be exploited. This analysis should also
take into consideration factors such as protective measures that are in
place which may reduce the risk of an attack and the system's
resiliency, that is, ability to recover from an attack.
Once the three components of risk--threat, vulnerability, and
consequence--have been assessed for a given asset or system, they are
used to provide an estimate of the expected loss considering the
likelihood of an attack or other incident. According to the NIPP,
calculating a numerical risk score using comparable, credible
methodologies provides a systematic and comparable estimate of risk
that can help inform national and sector-level risk management
decisions. To be considered credible, the NIPP states that a
methodology must have a sound basis; be complete; be based on
assumptions and produce results that are defensible; and specifically
address the three variables of the risk calculus: threat,
vulnerability, and consequence. The methodology should also be
comparable with other methodologies to support a comparative sector or
national risk assessment. To be comparable, a methodology must be
documented, transparent, reproducible, accurate, and provide clear and
sufficient documentation of the analysis process and the products that
result from its use.
The next steps in the DHS risk management framework involve
establishing priorities for program development based on risk
assessments; implementing these protective programs; and measuring
their effectiveness by developing and using performance measures.
Identifying and assessing risks helps decisionmakers to identify those
assets or systems that are exposed to the greatest risk of attack and,
based on this information, prioritize the development and funding of
protective programs that provide the greatest mitigation of risk given
available resources. The NIPP notes that because resources are limited,
risk analysis must be completed before sound priorities can be
established. To determine which protective measures provide the
greatest mitigation of risk for the resources that are available, the
NIPP directs policy makers to evaluate how different options reduce or
mitigate threat, vulnerability, or consequence of a terrorist attack.
To do so, the NIPP states that cost estimates are combined with risk-
mitigation estimates in a cost-benefit analysis to choose between the
different options. The last step in the NIPP, measuring the
effectiveness of security programs by developing and using performance
measures, provides feedback to DHS on its efforts to attain its
security goals. Performance metrics are to be developed and used to
affirm that specific goals and objectives are being met or to
articulate gaps in the national effort or supporting sector efforts.
Performance measures enable the identification of corrective actions
and provide decisionmakers with a feedback mechanism to help them make
appropriate adjustments in their strategies for protecting critical
infrastructure.
TSA Has Taken Actions to Prioritize Investments in Passenger Checkpoint
Screening Technologies, but Lacks a Risk-Based Strategy:
While TSA completed a strategic plan for the PSP in August 2008 that
identifies a strategy for researching, developing, and deploying
checkpoint screening technologies, the plan and the strategy were not
developed based upon all of the key risk management principles outlined
in DHS's NIPP. For instance, TSA has not conducted a complete risk
assessment for the PSP, conducted a cost-benefit analysis to prioritize
investments, or developed performance measures to assess the extent to
which the risk of attack has been reduced or mitigated by investments
in technologies. While the agency is currently reviewing a draft of the
Aviation Domain Risk Assessment (ADRA), as of September 2009, the ADRA
had not been finalized. Officials expect it to be finalized by the end
of calendar year 2009. TSA officials could not provide an expected
completion date. Therefore, we could not determine when TSA will
complete it or to what extent it will be consistent with DHS's risk
management framework. TSA officials acknowledged the importance of a
cost-benefit analysis and performance measures to guide technology
investments, and stated that they intend to develop them, but could not
identify when they would be completed. Until TSA completes these
activities, the agency lacks assurances that the PSP strategy addresses
the highest priority needs and mitigates the risk of an attack.
Further, TSA lacks information to adjust its strategy, if needed.
TSA Completed a Strategic Plan for the PSP that Identifies Goals and
Objectives:
TSA completed a strategic plan in August 2008 that identifies a
strategy and establishes goals and objectives for the PSP, and
submitted the plan to congressional committees in September 2008.
[Footnote 26] However, TSA officials stated that the NIPP was not used
as guidance in developing the plan. Instead, the officials stated that
the specific requirements for a strategic plan, as outlined in the
Intelligence Reform Act and 9/11 Commission Act, were used as guidance
to construct the plan. The strategic plan identifies three broad trends
that have been observed in the types of threats that TSA faces. First,
interest in catastrophic destruction of aircraft and facilities has
increased, in contrast to hijacking and hostage-taking that
characterized the majority of earlier attacks. Second, the range of
encountered weapons has expanded, many not previously recognized as
threats, nor detected by the technologies that were deployed. Third,
terrorists have attacked "soft" airport targets, including airport
lobbies, in other countries. To address these challenges, TSA's
strategic plan identifies that the agency's strategy is to utilize
intelligence; partner with law enforcement, industry partners, and the
public; and implement security measures that are flexible, widely
deployable, mobile, and layered to address the nation's complex open
transportation network. According to the plan, TSA is in the process of
implementing and evaluating a fundamental shift in strategy for the
security checkpoint that encompasses the critical elements of people,
process, and technology. In addition, the plan states that implementing
a new security approach called Checkpoint Evolution,[Footnote 27] which
started in the spring 2008, will bring the most significant changes
that have occurred in passenger screening since the airport security
checkpoint was first established in the 1970s.
TSA's strategic plan identifies that the key component of TSA's
strategy related to security checkpoints is to improve security
effectiveness and resource utilization at the checkpoints. Also, the
PSP manager stated that a goal of the PSP strategy is to achieve full
operating capability by the dates discussed for each checkpoint
screening technology listed in the strategic plan. To meet these goals,
the PSP strategic plan identifies three strategic objectives: (1)
improve explosive detection capability, (2) improve the behavior
detection capability of Transportation Security Officers (TSO), and (3)
extend the layers of security throughout the passenger journey. The
first objective, improving explosive detection capability, involves
combining new technology with procedures that emphasize an element of
unpredictability to improve explosive detection capability and prevent
would-be attackers from knowing the TSA security process. The second
objective, improving the behavior detection capability of TSOs,
involves shaping the checkpoint environment to better support and
enhance behavior detection capabilities by enabling TSOs to engage a
larger number of passengers more frequently throughout the checkpoint
queue using informal interviews and SPOT; improving the observation
conditions for TSOs trained in SPOT by enhancing the contrast between
passengers exhibiting signs of travel stress and those intending to do
harm to other passengers, aircraft, or the airport; and providing
communications tools for enhanced coordination between TSOs trained in
SPOT. The third objective, extending the layers of security throughout
the passenger journey, involves enabling additional layers of non-
intrusive security beyond the checkpoint and into public spaces;
increasing the interaction between TSOs and passengers to provide more
opportunities to identify irregular behaviors far ahead of the
potential threat reaching the checkpoint; and partnering with airlines,
airports, and the private sector to reduce vulnerabilities in soft
target areas.
TSA had been directed on multiple occasions to provide strategic plans
for explosives detection checkpoint technologies to congressional
committees. The Intelligence Reform Act mandated that TSA provide a
strategic plan that included, at a minimum, a description of the
current efforts to detect explosives on individuals and in their
personal property; operational applications of explosive detection
equipment at airport checkpoints; quantities of equipment needed to
implement the plan and a deployment schedule; funding needed to
implement the plan; measures taken and anticipated to be taken to
provide explosives detection screening for all passengers identified
for additional screening; and recommended legislative actions, if any.
[Footnote 28] The Intelligence Reform Act mandated that such a
strategic plan be submitted to congressional committees during the
second quarter of fiscal year 2005. According to TSA officials, a
strategic plan was developed and delivered to congressional committees
on August 9, 2005, in satisfaction of the statutory mandate. However,
the 9/11 Commission Act, enacted August 3, 2007, reiterated a
requirement for a strategic plan that TSA was mandated to submit in
accordance with the Intelligence Reform Act. Specifically, the 9/11
Commission Act required that the Secretary of Homeland Security issue a
strategic plan addressing its checkpoint technology program not later
than 30 days after enactment of the 9/11 Commission Act (that is, by
September 3, 2007) and required implementation of the plan to begin
within 1 year of the act's enactment.[Footnote 29] In response to the
9/11 Commission Act, TSA provided to Congress the Aviation Security
Report: Development of a Passenger Checkpoint Strategic Plan, September
2007.[Footnote 30] Finally, Division E of the Consolidated
Appropriations Act, 2008, enacted on December 26, 2007, required that
the Secretary of Homeland Security submit a strategic plan for
checkpoint technologies no later than 60 days after enactment of the
Act (that is, by February 25, 2008), and further restricted the use of
$10,000,000 appropriated to TSA for Transportation Security Support
until the Secretary submitted the plan to the Committees on
Appropriations of the Senate and House of Representatives.[Footnote 31]
As a result of the mandate for a strategic plan and the funding
restriction in the 2008 Consolidated Appropriations Act, TSA officials
told us that they interpreted this legislative language to mean that
congressional committees considered TSA's aviation security report in
September 2007 to be incomplete and insufficient. After approximately
12 months had elapsed since a strategic plan had been mandated in the
9/11 Commission Act, in August 2008 TSA completed its revised strategic
plan and delivered it to the committees in September 2008, which TSA
officials stated meets the mandate for a strategic plan in the 9/11
Commission Act, as well as the mandate for a strategic plan in the
appropriations act.
As previously discussed, the Intelligence Reform Act included
requirements for a deployment schedule, and descriptions of the
quantities of equipment and funding needed to implement the plan.
[Footnote 32] However, our analysis of TSA's August 2008 strategic plan
indicates that the strategic plan could include more complete
information about these requirements. For example, although TSA
provided some deployment information for each emerging checkpoint
technology listed in the strategic plan--such as the total quantity to
be deployed, expected full operating capability date, and types or
categories of airports where the equipment is to be deployed--it does
not include a year-by-year schedule showing the number of units for
each emerging technology that is expected to be deployed to each
specific airport. Regarding information on the funding needed to
implement the strategic plan, it includes a funding profile for each
fiscal year from 2007 through 2009. However, a number of the emerging
technologies are not expected to reach full operating capability until
fiscal year 2014. TSA officials stated that they have derived notional
(that is, unofficial) quantities to be deployed on an annual basis for
each technology through its respective full operating capability date,
but the officials stated that the funding profile in the strategic plan
does not reflect the funding needed for these future quantities because
the funding that will be appropriated for them after fiscal year 2009
is unknown. According to the officials, to implement the strategic plan
in the years beyond fiscal year 2009, the agency intends to use a year-
by-year approach whereby the quantities to be deployed in a particular
year, and the funding needed for that year, would not be officially
identified prior to the budget request for that year.
TSA officials stated that they used risk to inform the August 2008
strategic plan and the PSP strategy identified in it. Although TSA may
have considered that risk to some degree, our analysis does not confirm
that these efforts meet the risk-based framework outlined in the NIPP.
Specifically, TSA has not conducted a risk assessment or cost-benefit
analyses, or established quantifiable performance measures. As a
result, TSA does not have assurance that its efforts are focused on the
highest priority security needs, as discussed below.
TSA Has Not Conducted a Risk Assessment to Inform Its PSP Strategy, but
Is Finalizing an Assessment and Developing Information that May Help
Guide PSP Efforts:
TSA has not conducted a risk assessment that includes an assessment of
threat, vulnerability, and consequence, which would address passenger
checkpoint screening; consequently, the PSP strategy has not been
informed by such a risk assessment as required by the NIPP. Agency
officials stated that they prepared and are currently reviewing a draft
of a risk assessment of the aviation domain, known as the ADRA, which
is expected to address checkpoint security and officials expect it to
be finalized by the end of calendar year 2009; however, its completion
has been delayed multiple times since February 2008. Therefore, it is
not clear when this assessment will be completed. The ADRA, when
completed, is to provide a scenario-based risk assessment for the
aviation system that may augment the information TSA uses to prioritize
investments in security measures, including the PSP. However, officials
could not provide details regarding the extent to which the ADRA would
assess threat, vulnerability, and consequence related to the passenger
checkpoint. In 2004, we recommended that the Secretary of Homeland
Security and the Assistant Secretary for TSA complete risk assessments--
including a consideration of threat, vulnerability, and consequence--
for all modes of transportation, and use the results of these
assessments to help select and prioritize research and development
projects.[Footnote 33] TSA and DHS concurred with the recommendation,
but have not completed these risk assessments. Because TSA has not
issued the ADRA or provided details regarding what it will entail, and
because it is uncertain when the ADRA will be completed, it is not
clear whether the ADRA will provide the risk information needed to
support the PSP and TSA's checkpoint technology strategy. In the
meantime, TSA has continued to invest in checkpoint technologies
without the benefit of the risk assessment information outlined in the
NIPP. Consequently, TSA increases the possibility that its investments
will not address the highest priority security needs.
Although TSA has not completed a risk assessment to guide its PSP,
officials stated that they identify and assess risks associated with
the passenger screening checkpoint by relying on threat information,
vulnerability information from Threat Image Projection (TIP) scores,
limitations of screening equipment identified during laboratory
testing, covert tests, and expert judgment to guide its investment
strategy in the PSP.[Footnote 34] Specifically, TSA's Office of
Intelligence produces civil aviation threat assessments on an annual
basis, among other intelligence products. These assessments provide
information on individuals who could carry out attacks, tactics they
might use, and potential targets. TSA's most recent aviation threat
assessment, dated December 2008, identifies that terrorists worldwide
continue to view civil aviation as a viable target for attack and as a
weapon that can be used to inflict mass casualties and economic damage.
It also concluded that improvised explosive devices (IED) and
hijackings pose the most dangerous terrorist threat to commercial
airliners in the United States. The assessment identifies that these
devices may be concealed on persons, disguised as liquids, or hidden
within everyday, familiar objects such as footwear, clothing, toys, and
electronics. The threat assessment further identifies that terrorists
have various techniques for concealing explosives on their persons. In
addition to the annual civil aviation threat assessment, the Office of
Intelligence prepares for TSA's senior leadership team and other
officials a (1) daily intelligence briefing, (2) tactical intelligence
report that is produced one to four times per week, (3) weekly field
intelligence summary, (4) weekly suspicious incident report, and, when
necessary, (5) special events update, for example, during major
political events. However, according to the NIPP, relying on threat
information is not sufficient to identify and assess risks. Rather,
threat information, which indicates whether a terrorist is capable of
carrying out a particular attack and intends to do so, is to be
analyzed along side information on vulnerabilities--weakness in a
system that would allow such an attack to occur--and on the
consequences of the attack, that is, the results of a specific type of
terrorist attack, according to the NIPP.
TSA officials stated that, to guide the PSP, they also rely on programs
in place that are designed to assess vulnerabilities at airport
checkpoints. To identify vulnerabilities at airport checkpoints, TSA
officials stated that TSA analyzes TIP scores, known limitations of
screening equipment based on laboratory testing, and information from
its covert testing program. TSA conducts national and local covert
tests, whereby individuals attempt to enter the secure area of an
airport through the passenger checkpoint with a prohibited item in
their carry-on bags or hidden on their person. Officials stated they
use these sources of information to identify needed changes to standard
screening procedures, new technology requirements, and deployment
strategies for the PSP. When a checkpoint vulnerability is identified,
officials stated that TSA's Office of Security Technology engages other
TSA stakeholders through the PSP's Integrated Project Team process
[Footnote 35] to identify and develop necessary technology requirements
which may lead to new technology initiatives. Officials credited this
process with helping TSA identify needed changes to standard screening
procedures and deployment strategies for new technologies. For example,
according to a TSA official, a technology was developed as a result of
tests conducted by GAO that found that prohibited items and components
of an IED might be more readily identified if TSA were to develop new
screening technologies to screen these items.[Footnote 36]
Although TSA has obtained information on vulnerabilities at the
screening checkpoint, the agency has not assessed vulnerabilities (that
is, weaknesses in the system that terrorists could exploit in order to
carry out an attack) related to passenger screening technologies that
are currently deployed. The NIPP requires a risk assessment to include
assessments of threat, vulnerability, and consequence. TSA has not
assessed whether there are tactics that terrorists could use, such as
the placement of explosives or weapons on specific places on their
bodies, to increase the likelihood that the screening equipment would
fail to detect the hidden weapons or explosives. Although TIP scores
measure how effectively screeners identify prohibited items, they do
not indicate whether screening technologies currently deployed may be
vulnerable to tactics used by terrorists to disguise prohibited items,
such as explosives or weapons, thereby defeating the screening
technologies and evading detection. Similarly, TSA's covert testing
programs do not systematically test passenger and baggage screening
technologies nationwide to ensure that they identify the threat objects
and materials the technologies are designed to detect, nor do the
covert testing programs identify vulnerabilities related to these
technologies. We reported in August 2008 that, while TSA's local covert
testing program attempts to identify test failures that may be caused
by screening equipment not working properly or caused by screeners and
the screening procedures they follow, the agency's national testing
program does not attribute a specific cause of the test failure.
[Footnote 37] We recommended, among other things, that TSA require the
documentation of specific causes of all national covert testing
failures, including documenting failures related to equipment, in the
covert testing database to help TSA better identify areas for
improvement. TSA concurred with this recommendation and stated that the
agency will expand the covert testing database to document test
failures related to screening equipment. Moreover, TSA officials stated
that it is difficult to attribute a test failure to equipment, because
there is a possibility that the threat item used for the test was not
designed properly and, therefore, should not have set off the
equipment's alarm. TSA officials also stated that it is difficult to
identify a single cause for a test failure because covert testing
failures can be caused by multiple factors. As a result, TSA lacks a
method to systematically test and identify vulnerabilities in its
passenger and baggage screening equipment in an operational airport
setting. Consequently, TSA officials do not have complete information
to identify the extent to which existing screening technologies
mitigate vulnerabilities at the passenger checkpoints, so that they can
incorporate this information into the agency's security strategy, as
required by DHS guidance.
TSA's ADRA, once completed, is to cover the entire aviation domain and
include three parts--assessments of over 130 terrorist attack scenarios
to determine whether they pose a threat to the aviation system; an
assessment of known vulnerabilities or pathways within the aviation
system through which these terrorist attacks could be carried out; and
an assessment of consequences of these various types of terrorist
attacks, such as death, injury, and property loss. TSA officials stated
that, through the use of expert panels, the ADRA will evaluate these
threat scenarios to assess the likelihood that terrorists might
successfully carry out each type of attack on the aviation system, and
the likelihood and consequences of these various scenarios will be
prioritized to identify the most pressing risks that need to be
addressed. In the case of the passenger screening checkpoint, according
to officials, TSA will be examining all security measures that a
terrorist must breach in order to carry out a specific type of an
attack, such as carrying an IED on board an aircraft and detonating it
midflight. However, officials could not explain or provide
documentation identifying the extent to which the ADRA will provide
threat, vulnerability, and consequence assessments in support of the
PSP. In addition, the completion date for the ADRA has been delayed
multiple times. Because the ADRA has not been finalized and TSA has not
described how the ADRA will address the passenger checkpoint, we could
not determine the extent to which it will incorporate information on
checkpoint vulnerabilities, including vulnerabilities associated with
screening technologies and standard operating procedures.[Footnote 38]
In addition to the ADRA, TSA and DHS S&T are developing other
information that could inform their identification and assessments of
risks to the aviation transportation system. Specifically, TSA and S&T
are reviewing the scientific basis of their current detection standards
for explosives detection technologies to screen passengers, carry-on
items and checked baggage. As part of this work, TSA and S&T are
conducting studies to update their understanding of the effects that
explosives may have on aircraft, such as the consequences of detonating
explosives on board an in-flight aircraft. Senior TSA and DHS S&T
officials stated that the two agencies decided to initiate this review
because they could not fully identify or validate the scientific
support requiring explosives detection technologies to identify
increasingly smaller amounts of some explosives over time as required
by TSA policy. Officials stated that they used the best available
information to originally develop detection standards for explosives
detection technologies. However, according to these officials, TSA's
understanding of how explosives affect aircraft has largely been based
on data obtained from live-fire explosive tests on aircraft hulls at
ground level. Officials further stated that due to the expense and
complexity of live-fire tests, FAA, TSA, and DHS collectively have
conducted only a limited number of tests on retired aircraft, which
limited the amount of data available for analysis. As part of this
ongoing review, TSA and S&T are simulating the complex dynamics of
explosive blast effects on an in-flight aircraft by using a computer
model based on advanced software developed by the national
laboratories. TSA believes that the computer model will be able to
accurately simulate hundreds of explosives tests by simulating the
effects that explosives will have when placed in different locations
within various aircraft models. Officials estimated this work will be
completed in 3-to 4-month increments through 2008 and 2009. Officials
further stated that the prototype version of the model was validated in
the late summer of 2008, and that the model is currently being used.
TSA and S&T officials stated that they expect the results of this work
will provide a much fuller understanding of the explosive detection
requirements and the threat posed by various amounts of different
explosives, and will use this information to determine whether any
modifications to existing detection standards should be made moving
forward.
TSA Has Not Completed a Cost-Benefit Analysis to Help Establish Risk-
Based Priorities and Guide Its Investment Strategy:
TSA has not completed a cost-benefit analysis to prioritize and fund
the PSP's priorities for investing in checkpoint technologies, as
required by the NIPP's risk management framework. According to the
NIPP, policy makers who are designing programs and formulating budgets
are to evaluate how different options reduce or mitigate threat,
vulnerability, or consequence of a terrorist attack through a cost-
benefit analysis that combines cost estimates with risk-mitigation
estimates.[Footnote 39] However, in addition to lacking information on
risks to the screening checkpoint, TSA has not conducted a cost-benefit
analysis of checkpoint technologies being researched and developed,
procured, and deployed. Such a cost-benefit analysis is important
because it would help decisionmakers determine which protective
measures, for instance, investments in technologies or in other
security programs, will provide the greatest mitigation of risk for the
resources that are available.
One reason that TSA may have difficulty developing a cost-benefit
analysis for the PSP is that it has not developed life-cycle cost
estimates of each screening technology the PSP is developing,
procuring, or deploying. This information is important because it helps
decisionmakers determine, given the cost of various technologies, which
technology provides the greatest mitigation of risk for the resources
that are available. TSA officials prepared a PSP lifecycle cost
estimate in September 2005, but this estimate does not include cost
estimates for all technologies currently being researched, developed,
tested and evaluated, procured and/or deployed, such as the Advanced
Technology Systems, a technology to screen carry-on items that TSA is
currently procuring. TSA was subsequently instructed by DHS Joint
Requirements Council[Footnote 40] to complete lifecycle cost estimates
for the PSP; in December 2005, the council reviewed the PSP and
approved it to proceed to the Investment Review Board for an annual
review and potential approval of the PSP's fiscal year 2006 procurement
strategy. However, the council expressed concern about several issues
that should be resolved prior to the Investment Review Board's review,
including the need for complete lifecycle cost estimates for the
checkpoint screening technologies that were to be developed and
procured. TSA officials acknowledged that completing lifecycle cost
estimates are important and stated that they have not prepared a
lifecycle cost estimate since the council recommended that such an
estimate be developed due to lack of staff. These officials further
stated that TSA hired four full-time equivalent staff in fiscal year
2008, and two additional full-time equivalent staff are expected to be
hired in the fall of 2008. The officials anticipate that these staff
will help prepare lifecycle cost estimates. However, the officials did
not provide a timeframe for the completion of the estimates.
Although TSA officials identified the technologies they are procuring
and deploying, TSA officials could not provide us with information on
their priorities for the research and development of checkpoint
screening technologies or the processes they followed to develop these
priorities. According to S&T officials, TSA provided priorities for
near-term applied research and development projects to the S&T Capstone
Integrated Product Team (IPT) for Explosives Prevention.[Footnote 41]
This IPT establishes priorities for research projects to be funded by
S&T during the fiscal year. S&T officials stated that they rely on TSA
and other members of the IPT to use a risk-based approach to identify
and prioritize their agencies' or offices' individual research and
development needs prior to submitting them for consideration to the
IPT. However, TSA officials stated they did not submit priorities for
research and development to S&T. Without cost-benefit or other analysis
to compare the cost and effectiveness of various solutions, the agency
cannot determine whether investments in the research and development of
new checkpoint technologies or procedures most appropriately mitigate
risks with the most cost-effective use of resources. In addition,
without knowing the full cost of the technologies that the PSP is
developing, procuring, or deploying, TSA could potentially invest in a
technology in which the cost outweighs expected benefits.
TSA Lacks Measures to Evaluate the Extent to Which the PSP Reduces the
Risk of Terrorist Attacks:
TSA's strategy for the PSP does not have a mechanism--such as
performance measures or other evaluation methods--to monitor, assess,
or test the extent to which investments in new checkpoint technologies
reduce or mitigate the risk of terrorist attacks. The NIPP requires
that protective programs be designed to allow measurement, evaluation,
and feedback based on risk mitigation so that agencies may re-evaluate
risk after programs have been implemented and take corrective action if
needed, such as modifying existing programs to counter new risks or
implementing alternative programs. The NIPP identifies three types of
performance measures--descriptive, process/output, and outcome
measures--that can help gauge the effectiveness of protective programs.
[Footnote 42] Although the NIPP requires that protective programs be
designed to allow measurement, evaluation, and feedback based on risk
mitigation, TSA has not identified quantifiable measures of progress
which would allow the agency to assess the PSP's overall effectiveness.
TSA officials stated that they do not have overall performance measures
but are currently developing performance goals and measures for the
overall program. However, the officials could not provide a time frame
for their completion. In September 2004, we recommended that TSA
complete strategic plans for its research and development programs
which contain measurable objectives.[Footnote 43] Without measures to
monitor the degree to which the TSA's investments in the research,
development, and deployment of new screening technologies reduce or
mitigate terrorist threats, the agency is limited in its ability to
assess the effectiveness of the PSP or the extent to which it
complements other layers of security at the checkpoint.
Ten New Checkpoint Screening Technologies Are in Various Phases of
RDT&E, Procurement, and Deployment, but ETP Deployment Has Been Halted:
Since TSA's creation in 2001, 10 new checkpoint screening technologies,
including the ETP, have been in various phases of RDT&E, procurement,
and deployment, but TSA halted deployment of the ETP due to performance
problems and high installation costs. Of the 10 technologies, TSA has
initiated deployments for 4 of them, including the ETP and a Bottled
Liquids Scanner, but TSA has not deployed any of the 4 technologies to
airports nationwide. TSA also initiated procurements of two
technologies, including the Whole Body Imager; however, deployment of
these two technologies has not begun yet. Four checkpoint technologies
are in research and development, such as a shoe scanning device. In
June 2006, 6 to 11 months after TSA began to deploy the ETPs to
airports, the agency halted their deployment due to performance
problems--the machines broke down more frequently than specified by the
functional requirements and the machines were more expensive to install
and maintain in airports than expected. Because TSA did not follow its
acquisition guidance that recommends technologies be tested and
evaluated in an operational setting prior to procurement and
deployment, the agency lacked assurance that the ETPs performed as
required by the system's requirements. Although TSA officials were
aware that tests conducted on earlier ETP models during 2004 and 2005
suggested that they did not operate reliably in an airport environment
and that the ETP models that were subsequently deployed to airports had
not been tested in an operational environment to prove their
effectiveness, TSA deployed the ETPs to airports beginning in July 2005
for the Smiths Detection ETP and beginning in January 2006 for the
General Electric ETP without resolving these issues. TSA officials
stated that they deployed the ETPs to respond quickly to the threat
posed by a potential suicide bomber after suicide bombings had been
carried out onboard Russian airliners in 2004. TSA officials stated
that they plan to continue to use the 90 ETPs currently deployed to
airports. Because the ETPs were deployed without resolving their
performance problems and validating all of the functional requirements,
the ETPs have not been demonstrated to increase security at the
checkpoint. In the future, using validated technologies would enhance
TSA's efforts to improve checkpoint security.
S&T and TSA Investments in RDT&E Resulted in the Procurement or
Deployment of Six New Checkpoint Technologies:
As a result of S&T and TSA investments in the RDT&E of checkpoint
screening technologies since TSA's creation in 2001, six new screening
technologies are being procured and/or deployed, while four checkpoint
screening technologies are currently in the research and development
phase.[Footnote 44] Based on S&T and TSA RDT&E efforts, the agency has
initiated deployments of four technologies--the ETP, Fido PaxPoint
Bottled Liquids Scanner, Advanced Technology Systems, and Cast and
Prosthesis Scanner--three of which originated as commercial-off-the-
shelf technologies or commercial-off-the-shelf technologies that TSA
modified for use as checkpoint screening devices.[Footnote 45] However,
TSA has not completed the deployment for all of these four technologies
to airports nationwide. TSA officials stated that they did not deploy
additional checkpoint screening technologies because they were
primarily focused on deploying explosives detection systems to screen
checked baggage, as mandated by ATSA. TSA has also initiated
procurements of two additional technologies--Automated Explosives
Detection System for Carry-on Baggage and Whole Body Imager--but has
not deployed either of them yet. Figure 3 describes the status of the
six checkpoint screening technologies for which TSA has initiated
procurement and/or deployment.
Figure 3: Status of Six Checkpoint Screening Technologies that Had
Initiated Procurement and/or Deployment as of September 2008:
[Refer to PDF for image: table with illustrations for each technology]
Technology: Explosives Trace Portal (ETP); Illustration source: GAO;
Description: Detects traces of explosives on a passenger by using puffs
of air to dislodge particles from the passenger's body and clothing
that the machine analyzes for traces of explosives. Used for secondary
screening;
Status of Operational Testing: Completed for earlier models, but not
for models ultimately deployed. We discuss this in more detail later in
the report;
Status of Procurement: TSA procured 207 ETPs. In June 2006, TSA halted
further procurement due to high installation and maintenance costs and
performance issues. One hundred and sixteen of the procured units
remain in storage;
Status of Deployment to Airports: TSA deployed 101 portals to 36
airports during fiscal years 2005 and 2006. In June 2006, TSA halted
further deployment due to performance, maintenance, and installation
issues. Since June 2006, TSA has removed 11 ETPs from airports due to
maintenance issues and placed them in a warehouse for storage.
Technology: Bottled Liquids Scanner; Illustration source: ICx
Technologies, Inc.;
Description: Hand-held or table-top units that screen for liquid
explosives by detecting vapors of certain chemicals. Used for secondary
screening;
Status of Operational Testing: Completed for ICx Nomadics Fido PaxPoint
model, which is a type of hand-held device. Laboratory and operational
tests are ongoing for hand-held and/or table-top Bottled Liquids
Scanner devices;
Status of Procurement: TSA procured 215 Fido PaxPoint units during
fiscal year 2007 and 79 Smiths Detection Sabre 4000 units during fiscal
years 2007 and 2008. TSA planned to procure up to 750 hand-held and/or
table-top units in late fiscal year 2008. TSA increased its planned
procurement for fiscal year 2008 as a result of supplemental
appropriations received in fiscal year 2007 and appropriations
available in fiscal year 2008. Forty-one Smiths Detection units are at
TSA headquarters or in a warehouse in case they are needed for rapid
deployment;
Status of Deployment to Airports: TSA deployed 200 Fido PaxPoint units
from July 2007 to January 2008. TSA deployed 38 Smiths Detection Sabre
4000 units from July 2007 through December 2007, and 30 units are
currently in the process of being deployed. TSA plans to deploy a total
of 1,300 units at all category X through category IV airports.[A] Full
operating capability is planned for fiscal year 2011.
Technology: Advanced Technology Systems; Illustration source: Rapiscan
Systems, Inc. © 2009.;
Description: Intended to improve capability to detect threat items,
such as explosives. The units will replace the Threat Image Projection
Ready X-ray machines used at airports for primary screening of carry-on
items;
Status of Operational Testing: Completed;
Status of Procurement: TSA procured 250 units during fiscal year 2007.
Due to the availability of supplemental funding, appropriations
available in fiscal year 2008, and the need to expedite procurement of
these systems, the fiscal year 2008 planned procurement was 582 units,
of which 250 units have been procured. In fiscal year 2009, TSA plans
to award a contract to enhance current units;
Status of Deployment to Airports: From April 2008 to June 2008, 204
units were deployed to 12 airports, and about 287 additional units were
planned to be deployed by the end of fiscal year 2008. For units
deployed in fiscal year 2008, TSA plans to upgrade them in the field to
incorporate the enhancements under the contract to be awarded in fiscal
year 2009. TSA plans to deploy up to a total of 2,325 units at every
checkpoint lane in all category X through category IV airports. Full
operating capability is planned for fiscal year 2014.
Technology: Cast and Prosthesis Scanner; Illustration source:
CastScopeTM.;
Description: Provides a two-dimensional image of the area beneath a
cast or inside a prosthetic device. The device operates similarly to
the whole body imager, but for localized regions of a passenger's body.
Intended for use as a secondary screening device;
Status of Operational Testing: Completed;
Status of Procurement: TSA procured 34 units during fiscal year 2007.
Planned procurement was reduced from 40 to 34 units due to a system
maintenance cost increase. Due to a change in priorities, planned
procurement of 75 units in fiscal year 2008 was canceled because funds
were redirected to procure additional units of Advanced Technology
Systems and Whole Body Imagers. TSA has no plans to procure additional
units in the future;
Status of Deployment to Airports: Deployment of 34 units to 10 airports
began in July 2008 with the deployment of 5 units; the remaining units
were expected to be deployed by the end of September 2008.
Technology: Automated Explosives Detection System for Carry-on
Baggage[B]; Illustration source: Analogic Corporation;
Description: Creates a three-dimensional image of carry-on items to
detect explosives and non-metallic weapons. Being considered as a
secondary screening device[C];
Status of Operational Testing: Expected to be completed in September
2009;
Status of Procurement: TSA procured 20 units during fiscal year 2007
for operational testing. TSA had no plans to procure any units in
fiscal year 2008;
Status of Deployment to Airports: Deployment to checkpoints at category
III and IV airports is expected to begin after operational testing has
been completed in September 2009.
Technology: Whole Body Imager; Illustration source: American Science &
Engineering, Inc. © 2006;
Description: Scans passengers by producing a two-dimensional, full-body
computer-generated image that reveals object anomalies underneath
clothing, including plastic explosives and concealed metallic, non-
metallic, ceramic and plastic objects. TSA is evaluating the
feasibility of using this system as a primary and secondary screening
device[D];
Status of Operational Testing: Expected to be completed in fiscal year
2009;
Status of Procurement: TSA leased 15 units in fiscal year 2007 for
operational testing. Due to the availability of fiscal year 2008
appropriations, 135 units were planned for procurement in fiscal year
2008, of which 47 have been procured. In fiscal year 2009, TSA plans to
award a contract for enhanced units;
Status of Deployment to Airports: Deployment of 150 units is expected
to begin in fiscal year 2010. For units deployed in fiscal year 2008
for testing, TSA plans to upgrade them in the field to incorporate the
enhancements under the contract to be awarded in fiscal year 2009. TSA
plans to deploy a total of 878 units at all category X through category
IV airports. Full operating capability is expected in fiscal year 2014.
Source: TSA and S&T.
[A] TSA classifies the commercial airports in the United States into
one of five security risk categories (X, I, II, III, and IV). In
general, category X airports have the largest number of passenger
boardings, and category IV airports have the smallest. Categories X, I,
II, and III airports account for more than 90 percent of the nation's
air traffic.
[B] Although this technology has an automated detection capability, TSA
is not testing the automated detection function in an operational
environment.
[C] Research and development of this technology is continuing,
specifically, to develop a computed tomography (CT) X-ray for carry-on
baggage. This technology will permit fully-automated inspection of
passenger baggage as opposed to the TSA screeners having to interpret
the results of the baggage screening process. Operational testing of
the CT X-ray technology is to be completed in fiscal year 2009.
[D] Research and development of this technology is continuing,
specifically, to develop passive terahertz (THz) and active gigahertz
(GHz) technologies to improve detection performance and reduce
operational costs of commercially available systems. Operational
testing of the THz and GHz technologies is to be completed in fiscal
years 2009 and 2010, respectively.
[End of figure]
According to TSA's August 2008 strategic plan for checkpoint
technologies, there are several other ongoing efforts in addition to
the technologies discussed in figure 3.[Footnote 46] S&T and TSA are
researching and developing a shoe scanning device that is to conduct
automated weapons and explosive detection without requiring passengers
to remove their footwear. TSA plans to award a contract in fiscal year
2010, with full operating capability in fiscal year 2015. TSA plans to
deploy 1,300 units at all category X through category IV airports. TSA
also has two ongoing efforts related to boarding pass and credential
authentication, according to the agency's strategic plan. Starting in
2007, TSA assumed responsibility from airline contractors for travel
document checking, which is currently conducted manually. TSA plans to
replace the manual system with an automated one. Specifically, the
Boarding Pass Scanning System is expected to verify the authenticity of
a boarding pass at the checkpoint and enable the use of paperless
boarding passes by the airlines. In addition, the Credential
Authentication Technology System is planned to be an automated system
that authenticates identification presented by passengers and airport
employees. According to TSA, the agency plans to eventually combine
both of these authentication systems in a single travel document
checking system. TSA plans to award a contract for these two systems in
fiscal year 2009, with full operating capability expected in fiscal
year 2014. TSA plans to deploy a total of 878 units to replace the
existing document verification tools at all category X through category
IV airports. Another ongoing effort identified in TSA's strategic plan
is the Next Generation ETD. This system is planned to replace legacy
ETD systems and to be able to identify a larger range of explosives.
Specifically, this system is expected to have enhanced explosive
detection capability in terms of sensitivity and the ability to detect
new threats, as well as other improvements over legacy systems, which
are expected to produce lower lifecycle costs. TSA plans to deploy
1,500 units at all category X through category IV airports.
TSA also has two additional efforts to assess possible technologies.
One effort is called Standoff Detection, which is intended to display
images to detect anomalies concealed under passengers' clothing. TSA
plans to conduct an operational utility evaluation of test article
units during fiscal year 2009 to evaluate the technology's feasibility
within checkpoint screening operations. According to TSA, this
technology would assist the agency in applying layered security prior
to the checkpoint in soft target areas, such as airport lobbies, to
improve early awareness of a potential explosive threat. If the
technology proves effective in the checkpoint operation, TSA plans to
award a contract in fiscal year 2010, with full operational capability
expected by fiscal year 2014, and to deploy 351 units to every
checkpoint at category X and category I airports. The other effort is
called Explosives Characterization for Trace (Chemical-based)
Detection. This effort includes the research and development of trace
signatures, detection, and physical properties of explosives to improve
the detection and performance of deployed explosives trace detection
technologies.
TSA Procured and Deployed ETPs without Assurance that They Would
Perform as Intended in an Operational Setting:
During 2004 and 2005, prior to deployment of the ETPs, TSA conducted a
series of acceptance tests (that is, laboratory tests) of the General
Electric and Smiths Detection ETPs that suggested they had not
demonstrated reliable performance. Specifically, in 2004, TSA conducted
acceptance tests on early models of the General Electric and Smiths
Detection ETPs to determine whether the ETPs met key functional
requirements. Subsequently, in 2004 a General Electric ETP model was
field tested at five airports to determine how well the ETP performed
in an operational environment. A Smiths Detection ETP model was also
field tested at an airport in 2004. Based on initial test results, both
vendors of the ETPs modified the machines, and TSA conducted further
laboratory testing. The modified General Electric ETP was tested from
December 2004 through February 2005. During the January 2005 to May
2005 time frame, both the General Electric and Smiths Detection ETP
models were tested. Even though tests conducted during 2004 and 2005 of
the General Electric and Smiths Detection ETPs suggested they had not
demonstrated reliable performance, TSA deployed the Smiths Detection
ETP and General Electric ETP to airports starting in July 2005 and
January 2006, respectively, without resolving identified performance
issues.[Footnote 47]
Further, TSA did not test all 157 of the ETP's functional requirements
prior to procuring and deploying the General Electric and Smiths
Detection ETP models. Instead, TSA tested the ETP models against a
subset of the functional requirements. According to TSA's System
Development Life Cycle Guidance, testing of a system is to be conducted
to prove that the developed system satisfies its requirements in the
functional requirements document. TSA officials could not identify the
specific requirements that were tested or the reason(s) that all of the
requirements were not tested.
A TSA official stated that TSA had intended to resolve problems
regarding the ETPs' performance after they had been deployed, but TSA
officials could not explain how these problems were to be resolved.
Officials further stated that they worked for over 1 year during 2006
and 2007 with the ETP vendors to correct reliability and maintenance
issues after the ETPs were initially deployed, but could not resolve
them. Furthermore, according to S&T officials, when TSA conducted
limited field tests, the ETP manufacturers provided different
configurations from those used during the laboratory tests. According
to officials, once this was discovered, it took more than 6 months for
the ETP manufacturers to recreate the configurations that had passed
the laboratory tests. TSA officials stated that, during this 6-month
period, the agency decided to award a sole source contract to General
Electric to procure its ETP.
Regarding the reliability of the ETPs, of the 101 ETPs (71 from General
Electric and 30 from Smiths Detection) that were originally deployed to
36 airports, the General Electric ETP did not meet the system
requirement for operational availability due to frequent breakdowns.
Both vendors' ETPs were also more expensive to maintain than expected,
according to the TSA Chief Technology Officer serving during this
period. The functional requirements document requires the ETP to be
operationally available 98.38 percent of the time. However, the General
Electric ETPs were not always able to meet this requirement. TSA
officials could not provide information on the operational availability
of the Smiths Detection ETPs. For the General Electric ETPs, from
January through May 2006, they were operationally available an average
of 98.05 percent of the time, although the ETPs met the operational
availability requirement for 2 months during that period. Furthermore,
TSA's operational requirements specify that the ETP should function for
a minimum of 1,460 hours between critical failures. A critical failure
means that an ETP fails to operate and must be repaired as soon as
possible. However, the TSA Chief Technology Officer at the time stated
that the ETPs operated at a much lower average number of hours before a
critical failure occurred because, for example, the dirt and humidity
of some airport environments adversely affected the equipment.
Specifically, from January 2006 through May 2006, the General Electric
ETPs operated for an average of 559 hours before a critical failure,
which means that these ETPs operated on average 38 percent of the time
that they were required to operate before a critical failure occurred.
TSA officials could not provide information on the mean time between
critical failures for the Smiths Detection ETPs. TSA officials stated
that they tested the ETPs in several airports for several months prior
to deployment, but data from these tests did not identify a problem
with mean time between critical failures. One reason for this, a TSA
official stated, was that not enough data were collected during the
field tests. As usage of the ETPs increased, officials stated that they
discovered the ETP was not meeting operational availability
requirements. The ETPs also required replacement filters and other
consumables more often than expected, according to officials, which
drove up maintenance costs.
According to TSA officials, because of a variance in operational
availability hours among the deployed ETPs, maintenance problems, and
the high cost of ETP installation at airports, in June 2006, the agency
halted the deployment of the ETP to additional airports and stopped the
planned purchase of additional ETPs. TSA officials plan to continue to
use the 90 ETPs currently deployed to airports. However, without
validating that the ETPs meet their functional requirements, TSA
officials do not have assurance that it is worthwhile to continue to
use the ETPs in light of the cost to maintain and operate them. In
addition, TSA officials are considering what to do with the ETPs that
were procured and are currently in storage. As of April 2009, 116 ETPs
were in storage.[Footnote 48]
TSA did not follow the Acquisition Management System (AMS) guidance or
a knowledge-based acquisition approach before procuring the ETPs, which
contributed to the ETPs not performing as required after they were
deployed to airports. Specifically, AMS guidance provides that testing
should be conducted in an operational environment to validate that the
system meets all functional requirements before deployment. In
addition, our reviews have shown that leading commercial firms follow a
knowledge-based approach to major acquisitions and do not proceed with
large investments unless the product's design demonstrates its ability
to meet functional requirements and be stable.[Footnote 49] The
developer must show that the product can be manufactured within cost,
schedule, and quality targets and is reliable before production begins
and the system is used in day-to-day operations. As discussed earlier
in this report, TSA officials told us that they deployed the ETP
despite performance problems because officials wanted to quickly
respond to emergent threats. However, TSA did not provide written
documentation to us that described the process used at the time to make
the decision to deploy the ETP or the process that is currently used to
make deployment decisions.
Using Validated Technologies Would Enhance TSA's Efforts to Improve
Checkpoint Security:
TSA has relied on technologies in day-to-day airport operations that
have not been demonstrated to meet their functional requirements in an
operational environment. For example, TSA has substituted existing
screening procedures with screening by the Whole Body Imager even
though its performance has not yet been validated by testing in an
operational environment. In the future, using validated technologies
would enhance TSA's efforts to improve checkpoint security.
Furthermore, without retaining existing screening procedures until the
effectiveness of future technologies has been validated, TSA officials
cannot be sure that checkpoint security will be improved.[Footnote 50]
DHS Is Addressing Coordination and Collaboration Challenges with
Stakeholders to Research, Develop, and Deploy Checkpoint Screening
Technologies:
DHS S&T and TSA coordinated and collaborated with each other and key
stakeholders on their research, development, and deployment activities
for airport checkpoint screening technologies, and DHS is taking
actions to address challenges and strengthen these efforts.[Footnote
51] Because S&T and TSA share responsibilities related to the RDT&E,
procurement, and deployment of checkpoint screening technologies, the
two organizations must coordinate with each other and external
stakeholders, such as airport operators and technology vendors. For
example, in accordance with provisions of the Homeland Security Act and
ATSA, S&T and TSA are to coordinate and collaborate with internal and
external stakeholders on matters related to technologies and
countermeasures for homeland security missions. S&T and TSA signed an
MOU in August 2006 that establishes a framework to coordinate their
work at the TSL, which tests and evaluates technologies under
development. S&T also established a Capstone IPT for Explosives
Prevention in 2006 to bring S&T, TSA, and U.S. Secret Service
leadership together to identify gaps in explosives detection
capability; prioritize identified gaps; review relevant, ongoing S&T
programs; and develop capabilities to meet identified needs. However,
inconsistent communication and the lack of an overarching test and
evaluation strategy have limited S&T's and TSA's ability to coordinate
effectively with one another. To coordinate with the aviation
community, S&T and TSA have hosted industry days and conference calls
to discuss new technologies with airport operators and technology
vendors. Although TSA has taken actions to build partnerships with
airport operators and vendors, it has not established a systematic
process to coordinate with them related to checkpoint screening
technologies. However, TSA officials stated that they are in the
beginning stages of establishing a systematic process.
S&T and TSA Are Addressing Coordination and Collaboration Challenges
with Each Other on New Checkpoint Screening Technologies, but
Challenges Remain:
S&T and TSA have taken actions to coordinate and collaborate with each
other related to the RDT&E of checkpoint screening technologies, such
as by communicating priorities and requirements for technologies and
working with each other on the Capstone IPT for Explosives Prevention.
However, S&T and TSA coordination and collaboration were not always
effective due to inconsistent communication and the lack of an
overarching test and evaluation strategy. The Homeland Security Act
assigned responsibilities within the department for coordinating and
integrating the research, development, demonstration, testing, and
evaluation activities of the department, as well as for working with
federal and private sector stakeholders to develop innovative
approaches to produce and deploy the best available technologies for
homeland security missions. The act further assigned S&T with
responsibility for coordinating with other appropriate executive
agencies in developing and carrying out the science and technology
agenda of the department to reduce duplication and identify unmet
needs. ATSA had also assigned TSA with coordination responsibilities,
including the coordination of countermeasures with appropriate
departments, agencies, and instrumentalities of the U.S. government.
[Footnote 52]
S&T and TSA have taken several actions to coordinate and collaborate on
their research and development activities related to checkpoint
screening technologies. First, to coordinate the transition of the TSL
from TSA to S&T, minimize disruption of work, and prevent duplication
of effort, S&T and TSA signed an MOU that defines the roles and
responsibilities for the research and development of homeland security
technologies, including checkpoint screening, and establishes a
framework for how to coordinate their work. Additionally, S&T created
the Capstone IPT for Explosives Prevention, which is co-chaired by the
Assistant Secretary for TSA and the Director of the U.S. Secret
Service, to identify and prioritize capabilities needed to detect
explosives; review relevant, ongoing S&T programs; and develop
capabilities to meet the identified needs. The IPT was first convened
in December 2006 to identify research and development priorities for
explosives detection technologies at airport checkpoints as well as for
other transportation modes, and has met periodically since then.
According to TSA officials, the Capstone IPT has enabled TSA to
establish a clear understanding with S&T of TSA's needs for technology
solutions that meet stringent detection thresholds and throughput
requirements to support the aviation sector. Additionally, the
officials stated that the Capstone IPT has given TSA a better
collective understanding of the technology needs of other DHS
components, which will help DHS identify technology solutions that can
be combined to benefit multiple users. Finally, to follow through on
the priorities established by the Capstone IPT for Explosives
Prevention, S&T officials stated that they established project-level
IPTs, including one for airport checkpoints and one for homemade
explosives. S&T officials stated that they are working with TSA on
these project-level IPTs to try to meet the needs identified by the
Capstone IPT. TSA officials further stated that they have PSP IPTs or
working groups to coordinate on technology projects, establish program
goals and objectives, and develop requirements and time lines. These
groups meet on a weekly basis, according to TSA officials. In April
2008, S&T dissolved the IPT for explosives detection and replaced it
with two separate IPTs, a transportation security IPT, chaired by TSA
and a counter-IED IPT, chaired by the Office of Bombing Prevention
within the National Protection and Programs Directorate and the United
States Secret Service.
Coordination and collaboration efforts between S&T and TSA have helped
in identifying checkpoint screening solutions. For example, S&T and TSA
officials collaborated on a hand-held vapor detection unit called the
Fido PaxPoint. After the August 2006, discovery of the alleged plot to
detonate liquid explosives on board commercial air carriers bound for
the United States from the United Kingdom, S&T and TSA worked together
to identify, develop, and test screening technologies to address this
threat. According to TSA officials, S&T learned that the Department of
Defense had developed a handheld unit that could detect vapors from
explosives. S&T modified the Department of Defense handheld unit,
resulting in the Fido PaxPoint unit to screen liquids and gels at
airport checkpoints for explosives, and S&T helped TSA test and
evaluate the device.[Footnote 53]
Although S&T and TSA have taken steps to coordinate and collaborate
with one another, inconsistent communication and a lack of an
overarching test and evaluation strategy have contributed to
coordination and collaboration challenges. Specifically, communication
between S&T and TSA related to S&T's basic and applied research efforts
and TSA's efforts to modify commercially available technologies has
been lacking at times. For example, TSA officials stated that early in
the TSL's transition to S&T (that is, during fiscal year 2006), TSA did
not receive information from S&T regarding which of TSA's research and
development needs S&T would fund, which projects related to airport
checkpoint technologies were underway at the TSL, or the time frames to
complete those projects. TSA officials stated that, without this
information, TSA was unable to determine whether its work on modifying
commercially available technologies for screening passengers and carry-
on items unnecessarily duplicated S&T's research and development
efforts, although TSA officials were not aware of any duplication that
occurred. An S&T official further stated that TSA had not consistently
fulfilled its responsibility to provide clearly defined functional
requirements for the equipment to be developed by S&T and tested by the
TSL, nor has TSA consistently given sufficient notice to the TSL of TSA
testing requests. Under the S&T and TSA MOU, TSA has retained
responsibility to establish requirements for equipment certification
and qualification and acceptance testing. Specifically, an S&T official
at the TSL stated that TSA had inadequately defined the functional
requirements and allowed too little time for testing several checkpoint
screening technologies, including the Advanced Technology Systems,
Enhanced Metal Detector II, and Bottled Liquids Scanner. A TSL official
acknowledged that when the TSA was responsible for the TSL, the agency
had not consistently developed requirements prior to testing or
certification of equipment as required by the DHS guidance.[Footnote
54]
In another example, as previously mentioned in this report, TSA is
developing new certification standards and functional requirements for
screening technologies, and is working with national laboratories to
validate data on aircraft vulnerabilities and generate new computer
models to help TSA develop requirements for explosives detection.
According to the TSA Chief Technology Officer in 2007, the TSL has
custody of the aircraft vulnerability data, but TSL officials had
refused to release the data to the national laboratories as requested
by TSA. Although the TSL later provided 32 of the 46 requested reports,
TSA officials estimated that the TSL's refusal to release all of the
reports had delayed the effort to develop new certification standards
and technology requirements by about 1 month. The officials added that
most of TSA's requests to S&T and the TSL had involved similar problems
and that, although the MOU provides a framework for coordination, these
types of problems are related to day-to-day operations and will have to
be resolved as situations arise.
According to S&T and TSA officials, senior-level management turnover at
S&T and TSA contributed to these communication difficulties, as well as
an S&T reorganization which began in August 2006 with the arrival of a
new Under Secretary for Science and Technology. S&T officials further
stated that, prior to the establishment of the PSP working groups,
there was no mechanism for S&T and TSA to communicate information about
priorities, funding, or project timelines. However, through the working
groups, S&T officials stated that S&T and TSA are beginning to achieve
regular communication and interaction at the working level, which
allows for information to be shared in a mutually beneficial way. S&T
and TSA officials also stated that communication with each other has
improved since the MOU was signed in August 2006 and, in particular
since the summer of 2007, although officials from both organizations
stated that further improvement is needed. According to S&T officials,
the TSL's independent test and evaluation division and TSA have
developed an effective working relationship for several programs,
including the Whole Body Imager and Advanced Technology Systems. In
addition, S&T officials stated that TSA has come to better understand
the processes involving the Capstone IPT and identifying capability
needs. According to TSA officials, the agency is in the process of
determining whether a position within its Office of Security Technology
should be established as a liaison with S&T to improve coordination
between S&T and TSA. If the position is created, the TSA liaison would
coordinate and collaborate with S&T officials on technology projects by
assessing the science that supports the technologies.
The MOU specifies that S&T and TSA will coordinate activities,
including developing an integrated, overarching test and evaluation
strategy for projects to ensure that test and evaluation functions are
not duplicative, adequate resources are outlined and secured for these
functions, and activities are scheduled to support the overall project
master schedule. However, an overarching test and evaluation strategy
for checkpoint technologies has not been developed. The lack of this
strategy has presented coordination and collaboration challenges
between S&T and TSA, and has resulted in the delay of some
technologies. For example, a TSL official stated that the TSL could not
accommodate TSA's request to test the Advanced Technology Systems, in
part, because TSA officials had not provided sufficient advance notice
of their testing needs. TSA officials said they were working with S&T
to develop a project master schedule for the Advanced Technology
Systems. S&T and TSA officials stated that they plan to develop a test
and evaluation strategy to define a coordinated technology transition
process from S&T to TSA by outlining key responsibilities and criteria
to initiate field evaluations of technologies, but officials could not
tell us when the test and evaluation strategy would be completed.
DHS Is Using Several Approaches to Strengthen Coordination and
Collaboration with Airport Operators, Technology Vendors, and Other
Federal Agencies:
DHS, through S&T and TSA, coordinates with airport operators, private
sector partners, such as technology vendors, and other federal agencies
on matters related to research and development efforts. This
coordination and collaboration between TSA and airport operators and
technology vendors is important because the agency relies on airport
operators to facilitate the deployment of equipment for testing and day-
to-day operations, and on vendors to develop and manufacture new
screening equipment.[Footnote 55] However, TSA does not have a
systematic process to coordinate with external stakeholders related to
checkpoint screening technologies, but TSA officials stated that the
agency has developed a draft communications plan, which is being
reviewed.
Although TSA does not have a systematic process to coordinate with
technology vendors, airport operators, and other stakeholders related
to the RDT&E, procurement, and deployment of checkpoint screening
technologies, agency officials stated that they plan to develop and
implement such a process. Specifically, TSA officials stated that they
have developed a draft communications plan, which is being reviewed,
that will document the communications process. However, TSA could not
provide an expected completion date for the plan. Although such a plan
should help in providing consistency to the agency's coordination
efforts, without knowing the specific activities the plan will include
or when it will be implemented, we cannot determine the extent to which
the plan may strengthen coordination. In addition, in September 2007,
TSA hired an Industry Outreach Manager within its Office of Security
Technology to improve relationships with airport operators and
communication with internal TSA stakeholders related to screening
technologies, including checkpoint technologies. In general, the
Industry Outreach Manager is the communications liaison for the Office
of Security Technology stakeholders and customers to exchange ideas,
information, and operational expertise in support of the office's
mission and goals, and to provide cutting-edge technologies in the most
efficient and cost-effective means possible. In addition to these
steps, in January 2007, S&T created a Corporate Communications Division
to coordinate on a wide variety of science and technology efforts with
public and private sector stakeholders. This office is in the process
of developing a tool to assess the effectiveness of its outreach
efforts to industry stakeholders.
The AMS guidance recommends that TSA coordinate with airport operators
to work out all equipment installation issues prior to deployment.
According to TSA officials, the role of the airport operator is
essential in ensuring that solutions under development are suitable for
use in an airport environment, taking into consideration all logistical
and operational constraints and possibilities. As described earlier,
provisions of the Homeland Security Act address the need to coordinate
research and development efforts to further homeland security missions,
and reinforce the importance of coordinating and collaborating with
airport operators. TSA sponsors monthly conference calls with airport
operators to discuss issues of general interest and, according to S&T
officials, S&T has conducted pilot studies with airport operators.
However, according to many of the 33 airport operators we interviewed,
[Footnote 56] TSA's coordination on the priorities for and deployment
of checkpoint screening technologies has been inconsistent.
Specifically, of the 33 airport operators we interviewed, 8 had only
positive comments about TSA's coordination and 16 expressed only
concerns regarding TSA's coordination efforts, while 9 expressed both
positive comments and concerns. Eleven of the 33 airport operators told
us that TSA had not shared information with them regarding checkpoint
technology needs and priorities. For example, an airport operator
stated that TSA provided specifications for new screening technologies
with sufficient lead time for the airport, which was building a new
checkpoint at the time, and that TSA had numerous coordination meetings
with airport officials to determine space constraints, power
requirements, and other factors. However, this same airport operator
expressed a desire for more coordination by TSA in the agency's
selection of the technologies to be pilot tested at this airport.
Another airport operator stated that, when TSA asks for volunteers to
participate in checkpoint screening technology pilot programs, it is
difficult to agree to participate because TSA does not clearly
communicate the program's goals or the capabilities of the technology
in the pilot program.
According to airport operators at another airport, TSA officials told
them that they would have the latitude to select the ETP from either of
two vendors on the TSA contract for purchase. According to the airport
officials, after they selected equipment from one of the vendors
because it would fit into the physical layout of the airport's
checkpoint, TSA told the airport officials that particular ETP vendor
was no longer under contract with TSA. As a result, airport officials
stated that they had to redesign the checkpoint, including raising the
ceiling, to accommodate the other vendor's ETP. Senior officials in
TSA's Office of Operational Process and Technology, the office
responsible for the development and implementation of security
technologies across several modes of transportation, subsequently
agreed that coordination with airport managers and other stakeholders
could be improved.
According to TSA officials, coordinating with technology vendors is
essential in order to determine what technology platform would be
appropriate and capable of providing the required detection and
throughput capabilities. S&T and TSA have conducted outreach efforts to
coordinate with technology vendors. For example, S&T officials stated
that they have hosted forums known as industry days and attended
conferences to discuss types of technologies needed to be developed and
the department's priorities for research and development. S&T officials
also stated that they make presentations at technology-related
conferences, symposia, and exhibits, highlighting the work conducted by
S&T. At every industry day and conference, officials said, airport
security and checkpoint screening technologies have been discussed. In
addition, TSA has coordinated with technology vendors through industry
days, individual meetings, and conferences. For example, TSA officials
stated that TSA held industry days with technology vendors to provide a
forum to communicate information to potential vendors on specific
technology testing and procurement efforts, and to allow vendors to ask
questions regarding technology projects and TSA expectations.
Despite these outreach efforts, of the seven vendors we interviewed who
had contracted with TSA to provide checkpoint screening technologies,
officials from five vendors expressed concerns about the agency's
ability to coordinate with them on current or future needs for
checkpoint technologies. Officials from four of the seven vendors
stated that TSA had not communicated a strategic vision for screening
technologies that will be needed at the checkpoint in the future, and
that TSA did not effectively and clearly communicate standards and
requirements for technologies to vendors. For example, just as TSL
officials commented that TSA did not always provide clear and
quantifiable requirements to conduct tests of screening technologies,
vendors stated that TSA had not communicated effectively about its
future needs, such as the operational requirements for an advanced,
integrated checkpoint screening system.[Footnote 57] Therefore, a
vendor official stated that some of them had taken the initiative to
develop integrated screening technologies in the hope that TSA will
eventually request this type of integrated system. TSA did not express
an opinion regarding the specific concerns raised by the technology
vendors, but a senior TSL official stated that TSA should sponsor
better briefings for vendors after the agency announces its intentions
to develop new technologies. The official stated that these briefings
could provide vendors with an opportunity for open dialogue with TSA
and clarification of TSA's needs for new technologies. According to a
vendor, without adequate coordination and communication from TSA, the
vendors' ability is limited in deciding how best to invest their
resources to develop new checkpoint screening technologies.
In addition to coordinating and collaborating with airport operators
and technology vendors, S&T and TSA coordinate and collaborate on the
department's RDT&E efforts with other federal agencies through
participation in the Technical Support Working Group, which is co-
chaired by the Departments of Defense and State. The Technical Support
Working Group is the U.S. national forum that identifies, prioritizes,
and coordinates interagency research and development of technologies to
combat terrorist acts, including explosives detection technologies. S&T
also coordinates with the national laboratories on homeland security
research.[Footnote 58] Specifically, S&T's Office of National
Laboratories coordinates homeland security-related activities and
laboratory-directed research conducted within the Department of
Energy's national laboratories. According to an S&T senior official,
S&T has worked with the national laboratories to supplement S&T's
research and development of explosives detection technologies by
tasking the national laboratories to conduct basic research on the
characteristics of homemade explosives.
Conclusions:
Researching, developing, testing and evaluating, procuring, and
deploying checkpoint technologies capable of detecting ever-changing
threats to the commercial aviation system is a daunting task. Although
TSA has recently produced a strategic plan that identified a strategy
for the PSP, neither the plan nor the agency's strategy for
researching, developing, and deploying checkpoint technologies was
informed by some key risk management principles, including a risk
assessment, cost-benefit analysis, and performance measures. Without
conducting a risk assessment that includes all three elements of risk-
-threat, vulnerability, and consequence--and completing a cost-benefit
analysis to guide the PSP strategy, TSA has limited assurance that its
strategy targets the most critical risks and that it invests in the
most cost-effective new technologies or other protective measures.
Further, without developing performance measures that assess the extent
to which checkpoint screening technologies achieve the PSP's security
goals and thereby reduce or mitigate the risk of terrorist attacks, TSA
is limited in its ability to determine the success of its strategy and
make needed adjustments. Even though TSA has not implemented a risk-
informed strategy to ensure that its investments target the most
pressing security needs, the agency has moved forward in investing in
new checkpoint screening technologies.
Despite limited progress in the RDT&E, procurement, and deployment of
new checkpoint screening technologies during the first few years that
S&T and TSA had responsibilities related to these technologies, more
recently, the organizations have made progress as reflected by the
number of technologies for which procurement and deployment has been
initiated. TSA faced challenges with the first new technology that it
procured and deployed--the ETP. In the interest of protecting the
homeland, it is understandable that TSA may, at times, not follow all
established guidance in an effort to deploy technologies quickly to
address urgent threats and vulnerabilities. However, deploying the ETP
despite unresolved performance concerns identified during testing of
earlier ETP models, as well as failing to ensure that ETP models that
were ultimately deployed had passed operational testing, increased the
risk that the machines would not perform as intended, resulting in a
questionable security benefit. TSA did not follow AMS guidance that
recommended operational testing of a new technology prior to deployment
because it is more cost effective to resolve performance issues then.
While TSA deployed the ETPs to provide a much-needed capability to
automatically screen higher risk passengers at airport checkpoints,
relying on the ETPs could have resulted in airport checkpoints being
more vulnerable given the ETPs' performance problems and lack of
operational testing. Also, relying on the ETPs to screen these
particular passengers instead of existing screening procedures may not
enhance airport checkpoint security because TSA does not know if ETP
screening provides an improved detection capability compared to
existing screening procedures. Moreover, it is risky to substitute any
new technology for existing screening procedures before the technology
has been proven to be effective through operational testing. Although
TSA is trying to deploy new technologies to address immediate threats,
the problems associated with the development and deployment of the ETPs
may be repeated with other technologies unless TSA adheres to testing
guidance and makes decisions using a knowledge-based acquisition
approach. Finally, it is not clear whether it is worthwhile to continue
to use the ETPs currently deployed to airports due to the costs
associated with maintaining the machines in good, operational
condition.
Recommendations for Executive Action:
To help ensure that DHS's Science and Technology Directorate (S&T) and
Transportation Security Administration (TSA) take a comprehensive, risk-
informed approach to the RDT&E, procurement, and deployment of airport
passenger checkpoint screening technologies, and to increase the
likelihood of successful procurements and deployments of such
technologies, in the restricted version of this report, we recommended
that the Assistant Secretary for TSA take the following eight actions:
* Conduct a complete risk assessment, including threat, vulnerability,
and consequence assessments, which would apply to the PSP.
* Develop cost-benefit analyses to assist in prioritizing investments
in new checkpoint screening technologies.
* Develop quantifiable performance measures to assess the extent to
which investments in research, development, and deployment of
checkpoint screening technologies achieve performance goals for
enhancing security at airport passenger checkpoints.
* After conducting a complete risk assessment and completing cost-
benefit analyses and quantifiable performance measures for the PSP,
incorporate the results of these efforts into the PSP strategy as
determined appropriate.
* To the extent feasible, ensure that operational tests and evaluations
have been successfully completed before deploying checkpoint screening
technologies to airport checkpoints.
* Evaluate whether TSA's current passenger screening procedures should
be revised to require the use of appropriate screening procedures until
it is determined that existing emerging technologies meet their
functional requirements in an operational environment.
* In the future, prior to testing or using all checkpoint screening
technologies at airports, determine whether TSA's passenger screening
procedures should be revised to require the use of appropriate
screening procedures until the performance of the technologies has been
validated through successful testing and evaluation.
* Evaluate the benefits of the Explosives Trace Portals that are being
used in airports, and compare the benefits to the costs to operate and
maintain this technology to determine whether it is cost-effective to
continue to use the machines in airports.
Agency Comments and Our Evaluation:
We provided a draft of our restricted report to DHS for review and
comment. On April 7, 2009, DHS provided written comments, which are
presented in Appendix II. In commenting on our report, DHS stated that
it agreed with our recommendations and identified actions planned or
underway to implement them. While DHS is taking steps to address our
first and second recommendations related to conducting a risk
assessment, the actions DHS reported TSA had taken or plans to take do
not fully address the intent of the remaining six recommendations.
In its comments, DHS stated that it concurred with our first
recommendation that a risk assessment should be developed for the PSP
and that TSA has two efforts currently underway to do so. Completion of
TSA's first effort--the Air Domain Risk Analysis (ADRA)--is expected in
the winter of 2009. DHS commented that TSA's second effort is the Risk
Management and Analysis Toolset (RMAT), a model to simulate the
potential of some technologies to reduce the risk of certain threat
scenarios which will apply specifically to the passenger screening
process. DHS reported that it expects initial results from RMAT to be
available during the second quarter of 2009. DHS further stated that
TSA has made resource allocation and technology decisions that were
informed by consideration of risk (including threat, vulnerability, and
consequence), although not by comparative assessments of these three
elements. However, as we reported, TSA has not conducted a risk
assessment for the PSP, and it is unclear to what extent the ADRA would
provide risk information needed to support the PSP. Until such a risk
assessment is developed and integrated into TSA's strategy for the PSP,
TSA continues to invest in checkpoint technologies without the benefit
of a risk-informed strategy and increases the possibility that its
investments will not address the highest-priority security needs.
DHS also concurred with our second recommendation that it develop cost-
benefit analyses. DHS commented that TSA is developing an approach for
selecting cost-effective technologies by developing life-cycle cost
estimates and using the RMAT tool to determine how technologies balance
risk (based on current threats) with cost. TSA's decision to collect
cost and benefit information is a positive first step. Irrespective of
how TSA collects data on the costs and benefits of technologies, it is
important, as we reported, that TSA conduct cost-benefit analysis of
each checkpoint technology that it invests in that weighs the costs and
benefits of technologies relative to the costs and benefits of other
solutions. Such analysis is important because it helps decision-makers
determine whether investments in technologies or in other security
programs will provide the greatest mitigation of risk for the resources
that are available.
DHS concurred with our third recommendation that TSA develop
quantifiable performance measures to assess the extent to which TSA's
investments in checkpoint screening technologies make the checkpoint
more secure, the key mission of the program. DHS commented that it
currently collects quantifiable performance attributes for all
potential acquisitions with regards to metrics, such as detection,
false alarm rate, and operational availability and plans to use
information on machines' attributes as measures of the PSP's overall
effectiveness as a program. However, these actions will not fully
address our third recommendation. First, information collected on
potential acquisitions prior to their deployment may not reflect their
performance in an operational environment; consequently, relying on
information about technologies' attributes rather than measuring the
effectiveness of deployed technologies to secure the checkpoint will
likely have limited value in terms of measuring the effectiveness of
the PSP as a program. Second, as we reported, the ETP example
illustrates that TSA did not collect information on the ETP's
performance attributes such as operational availability during
laboratory testing prior to procurement and did not collect data on the
ETP's detection capabilities during tests in an operational
environment. This raises questions about the completeness of data TSA
collects on technologies prior to acquisition and deployment. We could
not verify that TSA collects such information on other technologies
because TSA did not provide documentation to support this comment. As
TSA moves forward in developing performance measures, it is important
that these measures reflect not only efficiency of the technologies to
process passengers but the effectiveness of technologies and other
countermeasures to make the checkpoint more secure and thereby reduce
the risks posed by those most pressing threat scenarios that will be
identified once TSA completes its risk assessment.
In addition, DHS concurred with our fourth recommendation that it
develop a PSP strategic plan that reflects the risk assessment, cost
benefit analysis, and performance measures. DHS commented that TSA
plans to combine results from the RMAT tool and lifecycle cost
estimates for possible technology solutions that strike a balance
between risk and efficient use of funding. DHS also stated it will use
RMAT to develop proxy measures and general "what-if" analysis and risk
insights. However, these actions alone will not satisfy the intent of
this recommendation. While it is possible that proxy measures could be
developed to assess the extent to which TSA's investments in the
research and development of technologies have achieved program goals of
making the checkpoint more secure, to fully address this
recommendation, TSA must also conduct a risk assessment that addresses
the PSP, develop quantifiable measures that clearly assess the PSP's
progress towards its security goals, and revise its strategic plan
accordingly.
DHS concurred with our fifth recommendation that before deploying
technologies to airport checkpoints, the technologies should
successfully complete testing and evaluation and stated that TSA is
taking action to implement a formal testing process. DHS commented that
TSA has prepared a Test and Evaluation Master Plan (TEMP) that
describes a new formal testing process that is consistent with DHS's
new acquisition directive. However, the TEMP does not address the
intent of this recommendation. We deleted from this public report our
evaluation of why the TEMP does not address the intent of this
recommendation, because TSA determined our evaluation to be sensitive
security information.
Further, DHS agreed with our sixth and seventh recommendations that TSA
evaluate whether its screening procedures should be revised to require
the use of appropriate procedures until it can be determined that
emerging technologies or future technologies that may be developed meet
all of their requirements in an operational environment. However, DHS's
comments suggest that it does not intend to implement these
recommendations. DHS commented that the performance of machines is
always measured and confirmed in the laboratory setting prior to
operational field testing. However, we disagree that laboratory testing
is sufficient to address this recommendation. We deleted from this
public report our evaluation of why laboratory testing alone does not
address the intent of this recommendation, because TSA determined our
evaluation to be sensitive security information.
DHS stated that TSA implemented our eighth recommendation that the
agency evaluate the benefits of the ETP, such as its effectiveness, and
conduct a cost-benefit analysis to determine whether the technologies
should remain in use at airports. However, we disagree that TSA has
implemented this recommendation. DHS commented that two actions
fulfilled this recommendation: TSA's current program management reviews
in which costs are periodically discussed with vendors and the
laboratory testing of the ETP's detection capabilities. To fully
address this recommendation, a cost-benefit analysis and tests of the
ETP's effectiveness to detect explosives in an operational environment
are required. As we reported, TSA has not conducted cost-benefit
analyses, which, as noted earlier, should compare costs and benefits of
alternative solutions. Discussions of maintenance costs with vendors on
a periodic basis do not constitute a cost-benefit analysis.
Based on DHS's written comments, we deleted a reference to the 2004 OMB
PART review in a footnote because of updated information from OMB's
2008 PART review. DHS also provided us with technical comments, which
we considered and incorporated in the report where appropriate. In
particular, we clarified the wording of a recommendation which
originally stated that TSA should develop quantifiable performance
measures to assess the extent to which investments in research,
development, and deployment of checkpoint screening technologies have
mitigated the risks of a terrorist attack. We altered the wording to
state that performance measures should be developed to assess progress
towards security goals.
As agreed with your offices, unless you publicly announce the contents
of this report, we plan no further distribution for 45 days from the
report date. At that time, we will send copies of this report to the
Secretary of Homeland Security, the Assistant Secretary of the
Transportation Security Administration, and appropriate congressional
committees.
If you or your staffs have any questions about this report, please
contact me at (202) 512-8777 or LordS@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. GAO staff who made major contributions to
this report are listed in appendix III.
Signed by:
Stephen M. Lord:
Director Homeland Security and Justice:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
This report addresses the following questions: (1) To what extent has
the Transportation Security Administration (TSA) developed a risk-
informed strategy to prioritize investments in the research and
development of passenger checkpoint screening technologies? (2) What
new passenger checkpoint screening technologies has the Department of
Homeland Security (DHS) researched, developed, tested and evaluated,
procured, and deployed since its creation, and why did TSA halt the
first technology deployment that it initiated--the Explosives Trace
Portal (ETP)? (3) To what extent has DHS coordinated the research,
development, test and evaluation (RDT&E), procurement, and deployment
of passenger checkpoint screening technologies internally and with key
stakeholders, such as airport operators and technology vendors?
To determine the extent to which TSA has developed a risk-informed
strategy to prioritize investments in the research and development of
passenger checkpoint screening technologies, we analyzed program
documents, TSA's August 2008 strategic plan for checkpoint
technologies, TSA's September 2007 report on the development of a
strategic plan, technology project plans, and funding. We also compared
TSA's strategic plan and DHS's responses regarding their efforts to
manage their research and development investments, with DHS's guidance
from the National Infrastructure Protection Plan on how to utilize risk
management principles to target funding.
To determine the extent to which DHS researched, developed, tested and
evaluated, procured, and deployed new checkpoint screening technologies
since its creation, and to identify why TSA halted deployment of the
ETP, we analyzed TSA's strategic plan for checkpoint technologies,
TSA's Passenger Screening Program (PSP) documentation, including
information on the status of technologies being researched, developed,
tested and evaluated, procured, and deployed. Regarding the ETPs, we
analyzed the functional requirements for the system, contracts with
General Electric and Smiths Detection, and test reports for acceptance
tests, regression tests, and operational tests. We also reviewed ETP
deployment schedules and documentation on operational availability and
mean time between critical failure, and interviewed TSA officials about
the reasons that the ETP deployment was halted. We also compared the
ETP test approach used by S&T and TSA to the Acquisition Management
System (AMS) guidance and knowledge-based acquisition best practices.
[Footnote 59] We also interviewed TSA and S&T officials to obtain
information on current investments in the research, development, and
deployment of checkpoint technologies, and conducted site visits to the
Transportation Security Laboratory in Atlantic City, New Jersey, and
Tyndall Air Force Base, Florida, to observe testing of new checkpoint
technologies. We visited the TSL because that is where S&T tests and
evaluates technologies, including checkpoint screening technologies. We
visited Tyndall Air Force Base because technologies to detect bottled
liquids explosives were being tested there. Additionally, we analyzed
TSA's passenger screening standard operating procedures and interviewed
various TSA headquarters officials, 29 Federal Security Directors, 1
Deputy Federal Security Director, and 5 Assistant Federal Security
Directors for Screening, and visited nine airports where the ETPs had
been or were to be deployed or new checkpoint screening technologies
were undergoing pilot testing. We chose these officials because they
are the senior official at the airport in charge of security and manage
TSA's role in deploying new technologies at the airport. We selected
these nine locations based on the technologies that had been deployed
or were being tested, their geography, size, and proximity to research
and development laboratories. Of the nine airports we visited, the ETPs
had been or were to be deployed to seven of them, and other new
checkpoint screening technologies were undergoing pilot demonstrations
or testing at two of them. We visited four airports on the east coast,
and three airports on the west coast, and two airports located in the
west and southwestern regions of the United States. To determine
whether the ETP's requirements had been tested prior to procuring and
deploying them, we selected a non-probability sample of 8 out of the
157 total requirements. We selected the 8 requirements because they
were related to some of the ETP's key functionality requirements,
including operational effectiveness, operational suitability, and
passenger throughput.
To determine the extent to which DHS has coordinated and collaborated
on the RDT&E, procurement, and deployment of passenger screening
technologies internally and with key stakeholders, we analyzed program
documents, including an August 2006 memorandum of understanding between
TSA and S&T for the management of the Transportation Security
Laboratory (TSL). Additionally, we interviewed Department of State
officials, TSA and S&T officials, seven checkpoint technology vendors,
and airport operators[Footnote 60] and other officials at airports
where ETPs were initially deployed. Because we selected nonprobability
samples of airports to visit and officials to interview, we cannot
generalize the results of what we learned to airports nationwide.
However, the information we gathered from these locations and officials
provided us with insights and perspectives on DHS's efforts to
operationally test and evaluate, and deploy checkpoint technologies
that could only be obtained from officials stationed at locations where
the technologies had been tested or deployed. We reviewed the
Acquisition Management System, the Aviation and Transportation Security
Act, the Homeland Security Act of 2002, and the Intelligence Reform and
Terrorism Prevention Act and identified requirements and guidance for
coordination and collaboration among S&T, TSA, and other stakeholders.
We also reviewed S&T's and TSA's coordination activities and compared
them to TSA program guidance and GAO's recommended coordination
practices regarding agency coordination with external stakeholders.
[Footnote 61]
We conducted this performance audit from June 2006 through April 2009
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.
[End of section]
Appendix II: Comments from the Department of Homeland Security:
U.S. Department of Homeland Security:
Washington, DC 20528:
October 6, 2009:
Ms. Cathleen A. Berrick:
Managing Director, Homeland Security and Justice Team:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Ms. Berrick:
Thank you for the opportunity to comment on the draft report: DHS and
TSA Have Researched, Developed, and Begun Deploying Passenger
Checkpoint Screening Technologies, but Continue to Face Challenges (GAO-
09-21). The Transportation Security Administration (TSA) appreciates
the U.S. Government Accountability Office's (GAO) work in planning,
conducting, and issuing this report.
In its report, GAO recommends that TSA improve its analysis of
technologies prior to deployment through increased use of risk-
management principles and cost-benefit analyses. TSA agrees that there
are opportunities for improvement in the deployment of passenger
checkpoint technologies and will continue to take steps to advance the
testing, deployment, ongoing performance measurement, and stakeholder
outreach as each relates to these technologies.
The Passenger Screening Program has embraced the challenge to design
and engineer passenger screening as an integrated network of
technologies, human factors, and processes to minimize the risk of
terrorist attack, using an approach that accounts for the strategic,
tactical, and operational realities of the airport environment. As a
strategy, the proactive approach builds upon lessons learned from hard
intelligence, airport security operations, and TSA organizational
experience.
The goals of TSA's security initiatives are to improve explosives
detection capability by combining new technology with advanced
procedures, establish a cadre of trained behavior detection officers,
and incorporate unpredictability into screening measures to thwart
terrorist plans. In each operational environment, be it at the
checkpoint, in the queues, or in conversations with passengers, TSA
created approaches designed to detect and contain risks to minimize
catastrophic loss of life or property.
Today, TSA implements passenger checkpoint screening through a risk-
informed approach that integrates a variety of sensors, devices, and
techniques into a threat detection and response network. This
integrated, network-centric approach considers operational requirements
to maximize passenger throughput, minimize the number of operating
personnel, and protect privacy, all within the constrained footprint of
the checkpoint. This proactive approach relies on an optimized,
integrated, mix of networked systems to deter known threats through the
passenger screening checkpoint.
TSA's Risk-Informed Strategy:
While TSA recognizes that additional risk-informed management
initiatives should and will be undertaken, current security technology
initiatives have been developed based on an assessment of what TSA
believes to be the most cost-effective way of reducing risk across the
entire aviation system.
TSA has made resource allocation and technology decisions that were
informed by considerations of risk (including threat, vulnerability,
and consequence), although not by explicit formal comparative
assessment of risk, including threat, vulnerability, and consequence
(TVC). This approach was appropriate at TSA's previous level of
maturity, and through this process, the Agency gained important
insights that have become guiding principles for risk management
decisions.
These principles include a deep appreciation for uncertainty and for
the limits of what can be known. TSA cannot predict every attack vector
in the aviation domain; therefore, our strategy reflects a bias against
rigid prioritization and narrow, inflexible solutions that can be
overcome by dynamic, adaptive adversaries. TSA continues to focus on
developing capabilities that are flexible and adaptable, that cut
across risks and threats, and that contain elements of randomness or
unpredictability to inject uncertainty into adversaries' planning
process. These principles have informed many of TSA's operational and
technology decisions in the last few years.
As this work has progressed, TSA has also pursued a formal TVC risk
analysis as important decision inputs into future strategy development.
TSA has nearly completed a range of formal, comparative, scenario-based
TVC risk analyses. The results will be used”along with other factors,
such as cost, feasibility, statutory requirements, and privacy”to
inform decisions about strategy, operations, and technology development
across TSA's aviation security responsibilities, including the
Passenger Screening Program (PSP).
At the strategic level, TSA is updating the initial Air Domain Risk
Analysis (ADRA), an action item stemming from Homeland Security
Presidential Directive 16, with final completion expected in November
2009 and approval expected in the fourth quarter of calendar year 2009.
The initial ADRA will compare high-level risks to and through the
checkpoint with other high-level risks (e.g., risks involving aviation
infrastructure and general aviation) across the domestic U.S. air
domain. High-level countermeasures are also evaluated to identify their
aggregate, cross-cutting risk-reduction potential across the portfolio
of risks. The ADRA will also include a view of risks that originate
internationally.
At a more granular level, TSA has been developing the Risk Management
and Analysis Toolset (RMAT). RMAT is an agent-based, high-detail
simulation model of the commercial aviation security regime, and can be
used to model specific risk scenarios (for example, specific types and
sizes of explosives) and the risk-reduction potential (across the
portfolio of risks) of specific technologies or other countermeasures.
The model accounts for dynamic, adaptive adversaries by incorporating
intelligence analysis to model the adversary's decision processes and
using that as the threat input, rather than using a direct, static
estimation of a threat vector.
This initial set of scenarios is keyed to the actionable decision-
support needs of TSA leadership in the near term. The initial RMAT
outputs are expected to be available to offer insights to TSA leaders
in the second quarter of calendar year 2009. As RMAT data sets and
capabilities grow, a larger range of high-detail risks can be modeled;
and the effectiveness of a greater range of potential countermeasures
can be evaluated.
These improved formal tools for understanding risk in a comprehensive
and comparative way”explicitly considering comparative TVC”will offer
additional insights and support to TSA leaders making decisions in
aviation security, including in the PSP. These tools, and formal risk
analysis generally, offer the additional effect of challenging accepted
assumptions and surfacing hidden assumptions, providing another benefit
to the decision process. Finally, formal comparative TVC risk analyses
do not replace, but rather complement and enhance, TSA's principles of
risk management described previously. Together they will enable TSA
leaders to make decisions on aviation security that are more effective,
more flexible, more traceable, and more defendable.
Measuring Performance:
TSA conducts a range of threat and vulnerability assessments relative
to the checkpoint to select, prioritize, and determine the
effectiveness of operational and technology investments. At the field
level, Federal Security Directors (FSDs) conduct various types of
testing and assessment of operations of their checkpoints. TSA also
conducts Joint Vulnerability Assessments (JVAs) in collaboration with
the Federal Bureau of Investigation (FBI) at airports on a regular
basis. The results of JVAs help provide understanding of common
vulnerabilities and trends. On a national level, TSA's Office of
Inspection conducts covert testing of checkpoint operations (Red Team
testing). This method of measuring the checkpoint's effectiveness, both
operationally and technologically, against a simulated attack can
identify vulnerabilities that may be specific to one airport or common
across the system. Covert tests serve both as a measure of
vulnerability that can help inform investment decisions and a reality-
based measure of effectiveness of existing operational and technology
investments. TSA collaborates with GAO to review Red Team results based
on threat injects conducted at checkpoints. In addition, TSA
continually monitors field equipment performance data such as Mean Down
Time (MDT), Operational Availability (OA), Mean Time to Repair (MTTR),
and throughput.
TSA recognized that it needed a more systematic, nationwide framework
to assess the effectiveness of the screening process and to identify
areas to focus our resources in training and technology. To this end,
TSA instituted a comprehensive program to measure screening performance
called the Aviation Screening Assessment Program (ASAP). ASAP is
aggressively focused on improving recognition of Improvised Explosive
Devices (IEDs), and TSA has performed thousands of covert assessments
at airports across the country. Through ASAP, we are assessing our
performance every day in every aspect of the screening process.
Findings from ASAP are reported directly to TSA leadership, who use
these performance metrics to make strategic decisions within the
screening environment. These decisions include the type of equipment
TSA purchases to the type of training TSA delivers to our
Transportation Security Officers (TSOs).
In addition, TSA recognizes the value of measuring program progress by
developing core program metrics. Through TSA's participation in the
Office of Management and Budget (OMB) Program Assessment Rating Tool
(PART), the Agency measures both long-term and annual performance
metrics. Along with the Reliability, Maintainability, Availability
(RMA) metrics, PSP calculates the Operational Availability of Passenger
Screening Equipment, the Cost per Bag Screened, and the Ratio of Bags
Screened to Passenger Screener Full-Time Equivalent (FTE) Expended. In
addition, TSA includes key performance parameters in all technology
specifications and tests the technology against those measures. These
measures reflect different facets of the purpose of PSP. Contrary to
footnote 43 of the draft GAO report, the most recent review (2008) of
PSP by OMB through the PART ranked the program as "Effective."
Testing and Evaluation Strategy for New Technologies:
TSA is in the process of improving the already robust Testing and
Evaluation (T&E) paradigm to ensure that operational effectiveness and
suitability of candidate security technology systems are evaluated
prior to deployment. Employing the concept of independent and
integrated T&E in support of acquisition decisions and other program
reviews, this process leverages data from multiple developmental and
operational testing sources, accredited vendor data, modeling and
simulation, and other special analyses (as required), in accordance
with T&E and systems engineering principles and best practices, to
streamline T&E requirements while still providing a credible and
comprehensive evaluation product. The system-specific integrated T&E
strategy addresses technical and operational requirements, considering
the contributions of people, processes, and technologies, to provide a
single portrait of anticipated mission capabilities for decision
makers. TSA is also active in the U.S. Department of Homeland Security
(DHS) T&E Council, which will lead to implementation of best practices
for T&E across DHS.
TSA has prepared a Test and Evaluation Master Plan (TEMP) for the PSP
program and is implementing a formal testing process specified in the
TEMP, consistent with DHS's new Acquisition Directive 102. The TEMP
establishes a framework that provides an overview of the testing
processes followed for all PSP technologies to ensure products meet our
specifications, are safe, and operationally effective. The test and
evaluation strategy is consistent with the program acquisition
strategy. All PSP technology projects follow this testing process,
which includes, at a minimum, qualification test and evaluation (QT&E)
conducted by the DHS Directorate for Science & Technology (S&T) and
operational test and evaluation (OT&E) conducted by TSA. While QT&E
tests equipment in a lab setting to validate its operational
effectiveness, OT&E tests the product in an airport setting to validate
its operational suitability.
Recognizing that Initial Operational Testing and Evaluation (IOT&E)
principles mandate examining system detection performance in the field
environment, TSA's Office of Security Technology (OST) recently
implemented a process to better coordinate realistic threat portrayal
with our developmental/technical testing partners. Starting with a
common threat baseline, we have developed a process to evaluate system
performance (considering operators, systems under test, and concepts of
operations) against both active threat agents in a laboratory
environment, as well as threat surrogates covertly inserted in the
stream of commerce, to better understand the system detection
performance envelope. Threat surrogates employed for OT&E purposes have
undergone a rigorous verification, validation, and accreditation (VV&A)
process to ensure that stimulants not only generate appropriate threat
signatures (and equivalent responses) relative to the technologies of
interest, but also can be strongly correlated back to the active threat
and appropriate segment of the threat space for the system of interest.
As much of our operational testing is conducted in active environments,
in situations where security decisions are being made with systems
under test (to most realistically demonstrate operational performance),
test design and execution is structured to ensure that test conduct
does not degrade the current security posture within the venue.
Depending on the specific test, TSA may employ redundant processes and
technologies to minimize potential security risks prior to operational
test execution. It should also be noted that detection effectiveness,
as per TSA testing strategy, is always measured and confirmed in the
laboratory setting prior to operational testing in the field.
Transportation Security Integration Facility Testing (TSIF):
TSA has completed the construction of a state-of-the-art facility
permitting emerging technologies to be tested for extended periods of
time while simulating a variety of operational conditions. This
facility will allow TSA to evaluate future maintenance needs and the
most efficient operational configurations of new equipment while
avoiding disruption to the flow of passengers at checkpoints. The TSIF
began operations in January 2009.
Explosives Trace Portal (ETP) Testing and Deployment:
In response to a growing concern that terrorists may try to destroy an
airplane by packing explosives on their person, TSA tested, procured,
and deployed ETP technology from 2004 to 2006. TSA followed its
standard technology development process for the ETP, which includes
requirements development, testing, procurement, and deployment.
TSA and the DHS S&T followed a formalized testing process, including
laboratory and field testing, from April 2004 to January 2006, to
validate the effectiveness and suitability of ETPs prior to full-scale
deployment. QT&E testing of two vendors' submissions was completed by
the Transportation Security Laboratory (TSL) in 2004, confirming the
ETP technology was effective in detecting explosives in accordance with
TSA's technical detection standards. TSA proceeded with airport
operational assessments by fielding five General Electric (GE) ETP
systems in 2004 During the field evaluations, TSA assessed the
suitability of the system and the Portal Quality Control (PQC) patches,
to evaluate the quality of the patches and the application process in a
checkpoint environment. As the GAO report noted, the PQC patch
performed unfavorably in the February 2005 tests. The test report also
indicated that a combination of factors, including the method of
testing, environmental conditions, the undetermined shelf life of PQC
patches, and the possible variance between levels of explosive
materials in the PQC patch, which could have affected detection
performance. It is important to note that detection effectiveness was
established at the TSL in 2004 and that the PQC assessment demonstrated
that the airport assessments did not accurately represent the ETP
detection performance, since the reliability of the quality control
items was unconfirmed in the field.
The ETPs were further tested by S&T at Idaho National Engineering and
Environmental Laboratory (INEEL) in 2005 using methods that were not
available to the TSL at that time (live explosives). The testing at
INEEL was presented to TSA by S&T as part of developmental data
collection effort for research and development purposes. The INEEL test
administrator who conducted the test was not authorized to perform
qualification testing on the ETPs, nor was it considered as part of the
detection effectiveness assessment. The performance of the ETPs during
this testing was not assessed against the established standards.
S&T and TSA proceeded to further assess modified ETP systems from
December 2004 to February 2005, reviewing software modifications
performed by the vendor. The February 2005 review concluded that the
requirements for detection were met by the modified ETP systems. TSA
proceeded with another round of airport operational assessments from
April to May 2005 to further validate operational suitability. Field
test results demonstrated satisfactory performance, indicating the
equipment was ready for full-scale deployment. In April 2006, TSA began
deploying ETPs to airports.
Additional Testing:
In 2006, TSA initiated another round of laboratory testing of the ETP
to evaluate its operational effectiveness. During April and May of
2006, INEEL conducted testing on both vendor submissions that revealed
deficiencies in reliable performance. Once these test results were
received, along with exhibited performance issues with the fielded
units, TSA's Office of Acquisitions formally notified the ETP vendor in
June 2006 that TSA would not deploy any additional ETPs until the
performance issues were addressed. Remaining delivery units were
diverted to the TSA warehouse until improvements could be completed and
verified. TSA received a reasonable level of assurance from the vendor
that these issues could be successfully addressed.
After working with vendors for several months, it was determined that
the ETP technology could not be enhanced, and vendors chose not to make
additional upgrades. Consequently, TSA did not purchase any further
units. TSA determined it was beneficial to keep the existing fielded
units in place, since they were effective at detecting explosives when
the performance reliability issues did not interfere.
Reliance on New Technologies:
As has been described, TSA follows a strict testing process for all
technologies before they are fully deployed. It should also be noted
that detection ability, as per TSA testing strategy, is always measured
and confirmed in the laboratory setting prior to operational testing in
the field. For example, in the case of Whole Body Imagers (WBIs), the
TSL Independent Test and Evaluation Division (IT&E) has conducted a
series of laboratory tests evaluating their performance. The WBI is
currently being evaluated against established criteria of threat
detection, passenger throughput, and operational availability. In
contrast to the ETP, the WBI provides the capability to detect a wider
range of threats, including metallic and nonmetallic explosives
threats. IT&E has performed five lab assessment-quality test series of
WBIs with various devices and/or operational procedures starting in
February 2007 and continuing to the present. These tests compare TSO
performance while using WBI devices to that of a manual pat-down
process. In the event of a WBI alarm, current Standard Operating
Procedures (SOPs) require a minimum of a pat-down. All test results
indicate the WBI technology is an effective alternative to the pat-down
process.
Stakeholder Coordination and Collaboration:
TSA appreciates GAO's conclusion that collaboration with S&T; our
external stakeholders, including technology vendors; and our airport
staff is an essential element to the successful testing, evaluation,
and deployment of checkpoint technologies. TSA also values the
relationships that have been fostered with industry stakeholders
throughout the aviation and security technology communities. While TSA
has taken measures to increase coordination with S&T through the
development of a Memorandum of Understanding (MOU), the Capstone
Integrated Product Team (IPT), and the associated PSP working group,
the findings in GAO's report further highlight the need to increase the
level of formal coordination with S&T.
TSA has participated in several American Association of Airport
Executives conferences. TSA also hosted an airport symposium in the
fall of 2007, in which it shared its vision for the future with airport
operators and the public. TSA's OST regularly participates in outreach
conferences to discuss the technologies available for passenger and
baggage screening. To better focus its efforts in this regard, TSA has
established a new position on checkpoint stakeholder outreach. In
addition, TSA partners with DHS to host technical interchange meetings
and industry days where Original Equipment Manufacturers are invited to
discuss future requirements for various technologies. These efforts
should ultimately reduce costs and development time as vendors work to
meet TSA's screening requirements.
TSA also makes every effort to effectively coordinate and collaborate
with our airport field staff. During deployment of passenger screening
technologies, TSA meets weekly with airports to discuss plans, status,
and any issues that arise. TSA has also initiated a TSO focus group in
which OST periodically meets with a select group of TSOs to gather
their input on new technology and process requirements for passenger
screening. As part of TSA's OT&E testing conducted for all equipment,
TSA gathers input from airport operators and TSOs to ensure the
equipment is functioning effectively in an operational environment.
Additional outreach for feedback and input will be implemented for the
FSD and their staff as appropriate.
General Conclusion:
To protect the security of the traveling public, TSA must be flexible
and able to adapt quickly to changes in terrorist tactics. This
overarching objective is reflected in every research and development
(R&D) and technology deployment decision made by the Agency. TSA will
continue to strive toward optimizing technological investments based on
thorough R&D analysis and risk-management principles, as well as the
collaborative testing and evaluation of new technologies.
Recommendation 1: Conduct a complete risk assessment, including threat,
vulnerability, and consequence assessments, which would apply to the
PSP.
Concur: TSA concurs and has initiated approaches that identify risks at
both the strategic and detailed level. At the strategic level, TSA is
updating the initial ADRA. Final completion is expected in November
2009 and approval is expected in fourth quarter of calendar year 2009.
At a more detailed level, TSA has been developing the RMAT, an agent-
based, high-detail simulation model that can be used to model specific
risk scenarios (for example, specific types and sizes of explosives).
The RMAT also can be used to model the risk-reduction potential (across
the portfolio of risks) of specific technologies or other
countermeasures (for example, a specific piece of equipment deployed in
a given percentage of locations). The RMAT tool will be used to
generate reports that apply specifically to the passenger screening
process and will allow PSP to perform analysis at a level not
previously available while not disrupting real world checkpoint
operations.
Recommendation 2: Develop cost-benefit analyses to assist in
prioritizing investments in new checkpoint screening technologies.
Concur: TSA concurs and is incorporating an approach that combines
those risks identified in the RMAT tool with detailed Life Cycle Cost
Estimates (LCCEs) to make an informed decision on screening
technologies that balances levels of risk (based on current threats)
with cost-effective procurement projections of available technologies.
In addition, as multiple technologies are implemented and overlapped at
the checkpoint, the RMAT scenario can be revisited to model the
possible effect of the new systems.
Recommendation 3: Develop quantifiable performance measures to assess
the extent to which investments in research, development, and
deployment of checkpoint screening technologies achieve performance
goals for enhancing security at airport passenger checkpoints.
Concur: TSA understands the importance of developing quantifiable
performance measures to ensure that investments support goals and
enhance security at the checkpoint. TSA already collects quantifiable
performance attributes for all potential acquisitions with regard to
metrics, such as detection, false alarm rate, and operational
availability. These attributes serve as the baseline against which new
technology submissions are evaluated and tested. TSA will work to
compare and integrate technology performance measures with those
included in the PART and to determine what measures improve checkpoint
security when overlapped with multiple systems.
Recommendation 4: After conducting a complete risk assessment and
completing cost-benefit analyses and quantifiable performance measures
for the PSP, incorporate the results of these efforts into the PSP
strategy as determined appropriate.
Concur: TSA concurs and will, upon availability, combine results from
the RMAT with LCCEs for possible technology solutions that strike a
balance between both risk and efficient use of funding. TSA believes
that the RMAT process provides highly valuable insights into capability
and risk priorities. However, because of the fundamental uncertainty
associated with a dynamic terrorist risk, RMAT and the use of proxy
measures cannot be assumed to represent risk reduction measures and
will instead be used to provide proxy measures and general "what-if'
analysis and risk insight. The projected performance of these
technologies will be compared against the long term performance metrics
for PSP as established by the most recent PART (2008) in order to
ensure linkage between goals and technology implementations.
Recommendation 5: To the extent feasible, ensure that operational tests
and evaluations have been successfully completed before deploying
checkpoint screening technologies to airport checkpoints.
Concur: TSA has prepared a Test and Evaluation Master Plan (TEMP) for
the PSP program and is implementing a formal testing process specified
in the TEMP, consistent with DHS's new Acquisition Directive 102. The
TEMP establishes a framework for incorporating phased-oriented test and
evaluation activities that facilitate the acquisition process. All PSP
technology projects follow this testing process, which includes, at a
minimum, QT&E conducted by the TSL and OT&E conducted by TSA. OT&E
tests the product in an airport setting to validate its operational
suitability. TSA has established a robust T&E paradigm to ensure that
candidate security technology systems are evaluated for operational
effectiveness and suitability prior to deployment.
Recommendation 6: Evaluate whether TSA's passenger screening procedures
should be revised to require use of appropriate screening procedures
such as pat downs where emerging technologies, including the ETPs and
whole body imagers, are currently being used at airports, until it is
determined that the machines can meet all of their explosives detection
requirements in an operational environment.
Concur: TSA was informed by the TSL that all Atlantic-1 requirements
for the ETP were met prior to the pilot testing of machines in the
field. The use of the ETP at the checkpoint does not prevent nor
prohibit the TSO from exercising judgment and performing additional
screening procedures, such as a pat-down, when they feel that it is
warranted. In addition, as per current SOP guidance, a pat-down is
required for ETP alarm resolution. TSA will complete a review of the
explosives trace portals to determine if they are cost-effective and
operationally feasible to continue using in airports. For WBI, the TSL
IT&E has conducted a series of laboratory tests evaluating the
performance of whole body imagers. IT&E performed more than five lab
assessment quality test series of WBIs (with various devices and/or
operational procedures) starting in February 2007 and continuing to
present. These tests have compared TSO performance while using WBI
devices to that of a manual pat-down process. Also, in the event of a
WBI alarm, current SOPs require a minimum of a pat-down. All results
indicate the WBI technology is an effective alternative to the pat-down
process.
Recommendation 7: Evaluate all future checkpoint screening technologies
prior to testing or using them at airports to determine whether
appropriate passenger screening procedures, such as pat downs, should
remain in place until the performance of the technologies has been
validated through operational test and evaluation.
Concur: TSA will continue to follow its formal test and evaluation
process to ensure all checkpoint screening technologies are
operationally effective prior to testing or using them in airport
settings.
Recommendation 8: Evaluate the benefits, such as the explosives
detection effectiveness and deterrent effect, of the ETPs that are
being used in airports, and compare the benefits to the costs to
operate and maintain this technology to determine whether it is cost-
effective to continue to use the machines in airports.
Concur: TSA has already implemented this recommendation through a
current and ongoing process of evaluation for the ETP that includes
periodic Program Management Reviews (PMRs) with the vendor during which
cost-related items are discussed. The detection effectiveness of the
ETP has been established in multiple lab tests, but TSA does not feel
that deterrence is a measurable quality. TSA has already begun making
progress implementing GAO's recommendations. This progress demonstrates
our commitment to continual improvement to ensure the security of the
traveling public.
Thank you for the opportunity to provide comments to the draft report.
Sincerely yours,
Signed by:
Jerald E. Levine:
Director:
Departmental GAO/0IG Liaison Office:
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Stephen M. Lord, (202) 512-8777 or LordS@gao.gov:
Staff Acknowledgments:
In addition to the contact named above, Robert Goldenkoff, Acting
Director; E. Anne Laffoon and Steve Morris, Assistant Directors; and
Joseph E. Dewechter, Analyst-in-Charge, managed this assignment.
Carissa Bryant, Chase Cook, Orlando Copeland, Neil Feldman, and Ryan
MacMaster made significant contributions to the work. Charles Bausell,
Jr., Richard Hung, and Stanley Kostyla assisted with design,
methodology, and data analysis. Michele Mackin assisted with
acquisition and contracting issues. Sally Williamson, Linda Miller, and
Kathryn Godfrey provided assistance in report preparation, and Thomas
Lombardi provided legal support.
[End of section]
Footnotes:
[1] TSA screeners are known as Transportation Security Officers and
perform a variety of duties related to security and protection of air
travelers, airports, and aircraft. TSA further oversees the operations
of private sector screeners at airports participating in TSA's
Screening Partnership Program.
[2] GAO, Aviation Security: Screener Training and Performance
Measurement Strengthened, but More Work Remains, [hyperlink,
http://www.gao.gov/products/GAO-05-457] (Washington, D.C.: May 2,
2005); and Aviation Security: Risk, Experience, and Customer Concerns
Drive Changes to Airline Passenger Screening Procedures, but Evaluation
and Documentation of Proposed Changes Could Be Improved, [hyperlink,
http://www.gao.gov/products/GAO-07-57SU] (Washington, D.C.: March 7,
2007).
[3] In this report, we define basic research as including all
scientific efforts and experimentation directed towards increasing
knowledge and understanding; applied research as including all efforts
directed toward--the solution of specific problems; advanced
development as including all efforts directed toward projects that have
moved into the development of hardware; and operational testing as
verification that new systems are operationally effective, supportable,
and suitable.
[4] GAO, Transportation Security R&D: TSA and DHS Are Researching and
Developing Technologies, but Need to Improve R&D Management,
[hyperlink, http://www.gao.gov/products/GAO-04-890] (Washington, D.C.:
September 30, 2004).
[5] GAO, Transportation Security: Efforts to Strengthen Aviation and
Surface Transportation Security Are Under Way, but Challenges Remain,
[hyperlink, http://www.gao.gov/products/GAO-08-140T] (Washington, D.C.:
October 16, 2007).
[6] In the April 2009 restricted version of this report, we reported
that, as of September 2008, TSA officials could not provide an expected
completion date or identify the extent to which the ADRA would address
risks to the checkpoint. In this public report on pages 7, 18, and 22,
we updated this information and stated that, as of September 2009, TSA
officials expected the ADRA to be completed by the end of 2009, but
could not identify the extent to which the ADRA would address risks to
the checkpoint. Also, in the restricted version of this report, we
reported that the NIPP was issued in 2006. In this public report on
page 15, we updated this information and stated that DHS issued a new
version of the plan in 2009. Furthermore, in the restricted version of
this report, we reported that, as of April 2009, TSA had 90 ETPs at
airports and 116 ETPs in storage. In this report on page 39, we updated
this information and stated that, as of September 2009, 22 ETPs were at
airports and no ETPs were in storage.
[7] A Federal Security Director is the ranking TSA authority
responsible for the leadership and coordination of TSA security
activities at TSA-regulated airports.
[8] TSA defines "airport operator" as any person who operates an
airport serving an aircraft operator or foreign air carrier required to
have a security program under 49 C.F.R. parts 1544 or 1546. See 49
C.F.R. § 1540.5.
[9] We selected the eight requirements because they were related to
some of the ETP's key functionality requirements, including operational
effectiveness, operational suitability, and passenger throughput.
[10] GAO, Results-Oriented Government: Practices That Can Help Enhance
and Sustain Collaboration among Federal Agencies, [hyperlink,
http://www.gao.gov/products/GAO-06-15] (Washington, D.C.: October 21,
2005).
[11] TSA deployed the ETPs from January to June 2006. Since June 2006,
TSA removed 11 ETPs from airports due to maintenance issues and placed
the ETPs in a warehouse for storage.
[12] We interviewed 46 airport operators, but 13 of them did not
express an opinion about whether TSA had shared or solicited
information regarding research and development needs and priorities for
checkpoint technologies.
[13] See [hyperlink, http://www.gao.gov/products/GAO-05-457] and
[hyperlink, http://www.gao.gov/products/GAO-07-57SU]. We found that TSA
had initiated actions designed to enhance screener training; however,
screeners sometimes encountered difficulty accessing and completing
training due to technological and staffing constraints. We also found
that TSA had implemented and strengthened efforts to collect screener
and checkpoint performance data through covert testing and a screener
recertification program. We further reported that TSA modified standard
operating procedures based on risk information, airport staff
experiences, and complaints and concerns made by the traveling public,
but that TSA could strengthen data collection and analysis to assist in
determining whether proposed procedures would achieve their intended
purpose.
[14] Sterile areas are generally located within the terminal where
passengers are provided access to boarding aircraft and access is
controlled in accordance with TSA requirements. Access is controlled by
screeners--either Transportation Security Officers employed by TSA or
nonfederal screeners at airports participating in TSA's Screening
Partnership Program--at checkpoints where screening is conducted of
individuals and carry-on baggage for weapons, explosives, and other
prohibited items. Screeners must deny passage beyond the screening
location to any individual or property that has not been screened or
inspected in accordance with measures and procedures in place at that
checkpoint. If an individual refuses inspection or the inspection of
any item, that person or item may not be allowed to enter the sterile
area or to board an aircraft.
[15] A nonselectee passenger who alarms the walk-through metal detector
on the first pass is offered a second pass. If the passenger declines
the second pass through, the passenger must proceed to additional
screening. If the nonselectee passenger accepts the second pass and the
machine does not alarm, the passenger may generally proceed without
further screening.
[16] Passengers are also screened by Behavior Detection Officers under
the Screening of Passengers by Observation Techniques (SPOT) program
and by Travel Document Checkers. SPOT is an additional layer of
security using behavior observations and analysis techniques to
identify potentially high-risk individuals based on deviations from
environmental baselines. Behavior Detection Officers are tasked with
detecting individuals exhibiting behaviors that indicate they may be a
threat to aviation and/or transportation security. Travel Document
Checkers are specially trained screeners who are positioned in front of
the checkpoint to check passengers' boarding passes and identification
in order to determine the authenticity of these documents.
[17] See Pub. L. No. 107-71, 115 Stat. 597 (2001).
[18] See Pub. L. No. 107-296, § 302, 116 Stat. 2135, 2163-64 (2002).
[19] See Pub. L. No. 108-458, § 4013(a), 118 Stat. 3638, 3719-20 (2004)
(codified at 49 U.S.C. § 44925(a)).
[20] Developmental testing is conducted to assist in the development
and maturation of a system or subsystem to verify the status of
technical progress and certify readiness to enter initial operational
testing. Operational testing verifies that new systems are
operationally effective, supportable, and suitable before deployment.
[21] DHS undertook to coordinate and integrate most of its research,
development, demonstration, testing, and evaluation activities in
accordance with section 302(12) of the Homeland Security Act.
[22] Acceptance testing consists of testing conducted to determine
whether a system or, in this case, technology satisfies its acceptance
criteria, such as specification requirements, and to enable the
customer to determine whether to accept the system or technology.
[23] According to TSA officials, depending on the requirements of the
sought-after technology and how it will be used, S&T and TSA first try
to identify commercial-off-the-shelf (COTS) equipment that meets
identified requirements without having to modify it. If COTS equipment
is identified but must be modified to meet TSA's needs, it would only
be used if it could be modified within a reasonable cost and time
frame. If COTS equipment cannot be identified or cannot be modified to
meet TSA's requirements within a reasonable cost or time frame, S&T
would try to develop a new technology for TSA.
[24] DHS, National Infrastructure Protection Plan (Washington, D.C.:
June 2006). In 2009, DHS issued an updated plan that replaced the one
issued in 2006.
[25] DHS has adopted an all-hazards mission, which includes both
natural disasters and terrorism. The department uses the NIPP to assess
risk for both; however, in the context of this report, we are focusing
on terrorism. The NIPP provides that for some critical infrastructure
sectors, assessing system risk is more appropriate.
[26] Passenger Checkpoint Screening Program Strategic Plan, Aviation
Security, Report to Congress in Response to Conference Report 109-699
to the Fiscal Year 2007 Department of Homeland Security Appropriations
Bill, August 2008.
[27] According to TSA, Checkpoint Evolution is a new security approach
that involves many different elements to secure the checkpoint
including continuously adapting security procedures to improve
passenger security.
[28] See 49 U.S.C. § 44925(b).
[29] See Pub. L. No. 110-53, § 1607, 121 Stat. at 483.
[30] The strategic plan mandated by the Intelligence Reform and 9/11
Commission Acts was to be submitted to the Senate Commerce, Science,
and Transportation Committee and the House of Representatives
Transportation and Infrastructure Committee.
[31] See Pub. L. No. 110-161, Div. E, 121 Stat. at 1844, 2053 (2007)
(referencing H.R. Conf. Rep. No. 109-699, at 138 (Sept. 28, 2006),
which had initially directed TSA to develop and submit this plan to the
committees).
[32] See 49 U.S.C. § 44925(b)(2).
[33] [hyperlink, http://www.gao.gov/products/GAO-04-890].
[34] The Threat Image Projection (TIP) system places images of threat
objects on the X-ray screen during actual operations and records
whether screeners identify the threat object. TIP is designed to test
screeners' detection capabilities by projecting threat images,
including guns and explosives, into bags as they are screened.
Screeners are responsible for positively identifying the threat image
and calling for the bag to be searched. Once prompted, TIP identifies
to the screener whether the threat is real and then records the
screener's performance in a database that could be analyzed for
performance trends. Low performance makes the screening process
vulnerable to terrorist attempts to smuggle such materials onto
aircraft.
[35] TSA manages the PSP, in part, through an Integrated Product Team
(IPT), which is led by the PSP, but draws its members from across TSA,
including the Office of Security Operations and the Office of
Acquisitions.
[36] The work related to the tests conducted by GAO contains classified
material and the results of these tests are not publicly available.
[37] GAO, Transportation Security: TSA Has Developed a Risk-Based
Covert Testing Program, but Could Better Mitigate Aviation Security
Vulnerabilities Identified Through Covert Tests, [hyperlink,
http://www.gao.gov/products/GAO-08-958] (Washington, D.C.: August 8,
2008).
[38] The ADRA is part of TSA's efforts to meet the requirements of
Homeland Security Presidential Directive 16 (HSPD-16), which requires
the DHS Secretary, in coordination with the Secretaries of State,
Defense, Commerce, and Transportation, the Attorney General, and the
Director of National Intelligence, to prepare a National Strategy for
Aviation Security that provides an overarching national strategy to
optimize and integrate governmentwide aviation security efforts. The
national strategy and its supporting plans are to use a risk-based
approach to ensure that national resources are allocated to security
efforts with the greatest potential to prevent, detect, defeat, or
minimize the consequence of an attack, taking into consideration
threat, vulnerabilities, and probable consequences of an attack. The
Secretaries of Homeland Security and Transportation are also to lead,
in conjunction with the Secretaries of State, Defense, and Energy, and
the Attorney General, an interagency effort, in consultation with
appropriate industry representatives, to develop and execute a risk-
based implementation plan for the continued reduction of
vulnerabilities within the Aviation Transportation System.
[39] According to the NIPP, investments in protective programs should
be prioritized based on a cost benefit analysis that weighs the cost,
time, and other characteristics of potential solutions, along with the
potential that these various investments in countermeasures will reduce
or mitigate threat, vulnerability, or consequence of an attack.
[40] The role of the DHS Joint Requirements Council is, among other
things, to manage investment portfolios and review projects to identify
cross-functional requirements and applications.
[41] In April 2008, S&T dissolved the IPT for explosives detection and
replaced it with two separate IPTs, a transportation security IPT,
chaired by TSA and a counter-IED IPT, chaired by the Office for Bombing
Prevention within the National Protection and Programs Directorate and
the United States Secret Service.
[42] Descriptive measures are used to understand sector resources and
activities, such as the number of facilities in a jurisdiction.
Process/output measures are used to measure whether specific activities
were performed as planned, tracking the progression of a task, or
reporting on the output of a process, such as inventorying assets.
Outcome measures track progress towards a strategic goal by beneficial
results rather than level of activity. In addition to the NIPP, the
Government Performance and Results Act of 1993 provides, among other
things, that federal agencies establish program performance measures,
including the assessment of relevant outputs and outcomes of each
program activity. According to the Office of Management and Budget
(OMB), performance goals are target levels of performance expressed as
a measurable objective, against which actual achievement can be
compared. Performance goals should incorporate measures (indicators
used to gauge performance); targets (characteristics that tell how well
a program must accomplish the measure), and time frames.
[43] In [hyperlink, http://www.gao.gov/products/GAO-04-890], we
recommended that the Secretary of Homeland Security and the Assistant
Secretary for TSA complete strategic plans containing measurable
objectives for DHS's and TSA's transportation security research and
development programs. DHS stated that it had completed a strategic plan
and that TSA was developing a strategic plan that outlined measurable
objectives. However, TSA has not yet completed a risk-based plan that
outlines measureable objectives. TSA's August 2008 strategic plan for
the PSP states that each technology is assessed in the laboratory and
in the field using key performance measures, which are reported to
senior management, so a decision about whether to acquire the
technology can be made. However, these measures apply to the
performance of individual, specific technologies against their
functional requirements before they are deployed, whereas the NIPP
guidance refers to performance measures that assess the effectiveness
of a program as a whole to mitigate risk and improve security.
[44] Some of the technologies that have initiated deployments or
procurements are continuing in research and development to do follow-on
work. For example, the Bottled Liquids Scanner and Advanced Technology
Systems continue to be enhanced.
[45] Commercial-off-the-shelf technology is a product or service that
has been developed for sale, lease, or license to the general public
and is currently available at a fair market value. The product or
service can sometimes be modified, which can save time and money
compared to researching, developing, and producing a product from
scratch.
[46] TSA submitted a strategic plan for the PSP to congressional
committees in September 2008. In the plan TSA identified several new
technologies that the agency had not previously identified to us.
Because we did not receive this strategic plan until toward the end of
our review in September 2008, we did not conduct detailed assessments
of these particular technologies.
[47] According to TSA, the specific methods and results of testing of
the ETPs during the research and development phase are sensitive
security information protected from disclosure pursuant to 49 C.F.R. §
1520.5(b). As a result, the relevant sections are described in the
restricted version of this report.
[48] TSA originally deployed 101 ETPs to airport checkpoints, and had
90 ETPs at airports and 116 ETPs in storage at the time we issued our
restricted April 2009 report. After issuance of our restricted report,
TSA stated that 22 ETPs were at airports and no ETPs were in storage as
of September 2009.
[49] GAO, Best Practices: Using a Knowledge-Based Approach to Improve
Weapon Acquisition, [hyperlink,
http://www.gao.gov/products/GAO-04-386SP] (Washington, D.C.: January
2004).
[50] According to TSA, our evaluation of TSA's use and validation of
airport screening technologies is sensitive security information
protected from disclosure pursuant to 49 C.F.R. § 1520.5(b)(9)(v). As a
result, the relevant sections are described in the restricted version
of this report.
[51] S&T is responsible for conducting basic and applied research, and
advanced development, including developmental test and evaluation. TSA
is responsible for conducting operational test and evaluation,
operational integration, procurement and deployment of new
technologies, including checkpoint screening technologies.
[52] In accordance with provisions of the Homeland Security Act and
ATSA, the S&T's Explosives Division and TSA should coordinate with one
another and other stakeholders, including the commercial aviation
community and DHS components, to facilitate the research, development,
and deployment of checkpoint screening technologies. TSA should also
coordinate countermeasures to protect civil aviation with appropriate
federal departments and agencies. See 49 U.S.C. § 114(f)(4). The S&T
Explosives Division develops technical capabilities to detect,
interdict, and lessen impacts of nonnuclear explosives used in
terrorist attacks against mass transit, civil aviation, and critical
infrastructure.
[53] Even though TSA officials stated that the Fido PaxPoint was
determined to be effective, the data collection process for it has been
extended to ascertain its operational suitability, specifically, how
sustainable and maintainable it is. TSA could not provide information
to us on the status of this data collection process.
[54] DHS Investment Review Process Management Directive 1400. This
directive was replaced in November 2008 by an Interim Acquisition
Directive (102-01).
[55] We focused our work on TSA's coordination and collaboration with
airport operators and technology vendors, and not on S&T's coordination
and collaboration with these external stakeholders, because TSA is
responsible for procuring and deploying checkpoint technologies.
[56] We selected a nonprobability sample of 40 airports and obtained
the views of 46 operators at these airports regarding coordination with
TSA. Thirteen of the 46 airport operators did not express an opinion
about coordination for and deployment of checkpoint screening
technologies. See appendix I for more information on how we selected
these airports.
[57] Section 4014 of the Intelligence Reform Act required TSA to
"develop and initiate a pilot program to deploy and test advanced
airport checkpoint screening devices and technology as an integrated
system at not less than 5 airports in the United States" by March 31,
2005. See Pub. L. No. 108-458, § 4014, 118 Stat. at 3720. According to
TSA, the only advanced checkpoint screening technology available to TSA
at that time was the ETP, and TSA initially conducted pilot tests at
five airports and later expanded the tests to 16 airports. TSA
officials stated that the agency submitted a strategic report to
Congress on August 9, 2005, Detection Equipment at Airport Screening
Checkpoints, in satisfaction of this requirement in the Act.
[58] The Homeland Security Act addresses the need for DHS to work with
federal laboratories and the private sector, among others, to develop
innovative approaches to produce and deploy the best available
technologies for homeland security missions. See Pub. L. No. 107-296, §
102(f)(5), 116 Stat. at 2143-44.
[59] See [hyperlink, http://www.gao.gov/products/GAO-06-257T].
[60] TSA defines an "airport operator" as any persons, who operates an
airport serving an aircraft operator or foreign air carrier required to
have a security program under 49 C.F.R. parts 1544 or 1546. See 49
C.F.R. § 1540.5.
[61] GAO, Results-Oriented Government: Practices That Can Help Enhance
and Sustain Collaboration among Federal Agencies, [hyperlink,
http://www.gao.gov/products/GAO-06-15] (Washington, D.C.: October 21,
2005).
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
