Aviation Security
Efforts to Validate TSA's Passenger Screening Behavior Detection Program Underway, but Opportunities Exist to Strengthen Validation and Address Operational Challenges Gao ID: GAO-10-763 May 20, 2010To enhance aviation security, the Transportation Security Administration (TSA) began initial testing in October 2003 of its Screening of Passengers by Observation Techniques (SPOT) program. Behavior Detection Officers (BDO) carry out SPOT's mission to identify persons who pose a risk to aviation security by focusing on behavioral and appearance indicators. GAO was asked to review the SPOT program. GAO analyzed (1) the extent to which TSA validated the SPOT program before deployment, (2) implementation challenges, and (3) the extent to which TSA measures SPOT's effect on aviation security. GAO analyzed TSA documents, such as strategic plans and operating procedures; interviewed agency personnel and subject matter experts; and visited 15 SPOT airports, among other things. Although the results from these visits are not generalizable, they provided insights into SPOT operations.
Although the Department of Homeland Security (DHS) is in the process of validating some aspects of the SPOT program, TSA deployed SPOT nationwide without first validating the scientific basis for identifying suspicious passengers in an airport environment. A scientific consensus does not exist on whether behavior detection principles can be reliably used for counterterrorism purposes, according to the National Research Council of the National Academy of Sciences. According to TSA, no other large-scale security screening program based on behavioral indicators has ever been rigorously scientifically validated. DHS plans to review aspects of SPOT, such as whether the program is more effective at identifying threats than random screening. Nonetheless, DHS's current plan to assess SPOT is not designed to fully validate whether behavior detection can be used to reliably identify individuals in an airport environment who pose a security risk. For example, factors such as the length of time BDOs can observe passengers without becoming fatigued are not part of the plan and could provide additional information on the extent to which SPOT can be effectively implemented. Prior GAO work has found that independent expert review panels can provide comprehensive, objective reviews of complex issues. Use of such a panel to review DHS's methodology could help ensure a rigorous, scientific validation of SPOT, helping provide more assurance that SPOT is fulfilling its mission to strengthen aviation security. TSA is experiencing implementation challenges, including not fully utilizing the resources it has available to systematically collect and analyze the information obtained by BDOs on passengers who may pose a threat to the aviation system. TSA's Transportation System Operations Center has the resources to investigate aviation threats but generally does not check all law enforcement and intelligence databases available to it to identify persons referred by BDOs. Utilizing existing resources would enhance TSA's ability to quickly verify passenger identity and could help TSA to more reliably "connect the dots." Further, most BDOs lack a mechanism to input data on suspicious passengers into a database used by TSA analysts and also lack a means to obtain information from the Transportation System Operations Center on a timely basis. TSA states that it is in the process of providing input capabilities, but does not have a time frame for when this will occur at all SPOT airports. Providing BDOs, or other TSA personnel, with these capabilities could help TSA "connect the dots" to identify potential threats. Although TSA has some performance measures related to SPOT, it lacks outcome-oriented measures to evaluate the program's progress toward reaching its goals. Establishing a plan to develop these measures could better position TSA to determine if SPOT is contributing to TSA's strategic goals for aviation security. TSA is planning to enhance its evaluation capabilities in 2010 to more readily assess the program's effectiveness by conducting statistical analysis of data related to SPOT referrals to law enforcement and associated arrests.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Stephen M. Lord Team: Government Accountability Office: Homeland Security and Justice Phone: (202) 512-4379GAO-10-763, Aviation Security: Efforts to Validate TSA's Passenger Screening Behavior Detection Program Underway, but Opportunities Exist to Strengthen Validation and Address Operational Challenges
This is the accessible text file for GAO report number GAO-10-763
entitled 'Aviation Security: Efforts to Validate TSA's Passenger
Screening Behavior Detection Program Underway, but Opportunities Exist
to Strengthen Validation and Address Operational Challenges' which was
released on May 20, 2010.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Ranking Member, Committee on Transportation and
Infrastructure, House of Representatives:
United States Government Accountability Office:
GAO:
May 2010:
Aviation Security:
Efforts to Validate TSA's Passenger Screening Behavior Detection
Program Underway, but Opportunities Exist to Strengthen Validation and
Address Operational Challenges:
GAO-10-763:
GAO Highlights:
Highlights of GAO-10-763, a report to the Ranking Member, Committee on
Transportation and Infrastructure, House of Representatives.
Why GAO Did This Study:
To enhance aviation security, the Transportation Security
Administration (TSA) began initial testing in October 2003 of its
Screening of Passengers by Observation Techniques (SPOT) program.
Behavior Detection Officers (BDO) carry out SPOT‘s mission to identify
persons who pose a risk to aviation security by focusing on behavioral
and appearance indicators. GAO was asked to review the SPOT program.
GAO analyzed (1) the extent to which TSA validated the SPOT program
before deployment, (2) implementation challenges, and (3) the extent
to which TSA measures SPOT‘s effect on aviation security. GAO analyzed
TSA documents, such as strategic plans and operating procedures;
interviewed agency personnel and subject matter experts; and visited
15 SPOT airports, among other things. Although the results from these
visits are not generalizable, they provided insights into SPOT
operations.
What GAO Found:
Although the Department of Homeland Security (DHS) is in the process
of validating some aspects of the SPOT program, TSA deployed SPOT
nationwide without first validating the scientific basis for
identifying suspicious passengers in an airport environment. A
scientific consensus does not exist on whether behavior detection
principles can be reliably used for counterterrorism purposes,
according to the National Research Council of the National Academy of
Sciences. According to TSA, no other large-scale security screening
program based on behavioral indicators has ever been rigorously
scientifically validated. DHS plans to review aspects of SPOT, such as
whether the program is more effective at identifying threats than
random screening. Nonetheless, DHS‘s current plan to assess SPOT is
not designed to fully validate whether behavior detection can be used
to reliably identify individuals in an airport environment who pose a
security risk. For example, factors such as the length of time BDOs
can observe passengers without becoming fatigued are not part of the
plan and could provide additional information on the extent to which
SPOT can be effectively implemented. Prior GAO work has found that
independent expert review panels can provide comprehensive, objective
reviews of complex issues. Use of such a panel to review DHS‘s
methodology could help ensure a rigorous, scientific validation of
SPOT, helping provide more assurance that SPOT is fulfilling its
mission to strengthen aviation security.
TSA is experiencing implementation challenges, including not fully
utilizing the resources it has available to systematically collect and
analyze the information obtained by BDOs on passengers who may pose a
threat to the aviation system. TSA‘s Transportation System Operations
Center has the resources to investigate aviation threats but generally
does not check all law enforcement and intelligence databases
available to it to identify persons referred by BDOs. Utilizing
existing resources would enhance TSA‘s ability to quickly verify
passenger identity and could help TSA to more reliably ’connect the
dots.“ Further, most BDOs lack a mechanism to input data on suspicious
passengers into a database used by TSA analysts and also lack a means
to obtain information from the Transportation System Operations Center
on a timely basis. TSA states that it is in the process of providing
input capabilities, but does not have a time frame for when this will
occur at all SPOT airports. Providing BDOs, or other TSA personnel,
with these capabilities could help TSA ’connect the dots“ to identify
potential threats.
Although TSA has some performance measures related to SPOT, it lacks
outcome-oriented measures to evaluate the program‘s progress toward
reaching its goals. Establishing a plan to develop these measures
could better position TSA to determine if SPOT is contributing to
TSA‘s strategic goals for aviation security. TSA is planning to
enhance its evaluation capabilities in 2010 to more readily assess the
program‘s effectiveness by conducting statistical analysis of data
related to SPOT referrals to law enforcement and associated arrests.
What GAO Recommends:
GAO recommends that TSA, among other things, use an independent panel
of experts to assist in validating SPOT, enhance SPOT data collection
and analysis, fully utilize TSA resources to identify possible
threats, and establish a plan to develop more outcome-oriented
measures for SPOT. DHS reviewed a draft of this report and generally
concurred with our recommendations although its plans do not fully
address one of our recommendations.
View [hyperlink, http://www.gao.gov/products/GAO-10-763] or key
components. For more information, contact Stephen M. Lord at (202) 512-
4379 or lords@gao.gov.
[End of section]
Contents:
Letter:
Background:
DHS Is Taking Action to Validate the Scientific Basis of TSA's SPOT
Program but Opportunities Exist to Help Inform Future Program
Decisions:
More Fully and Consistently Utilizing Available Information Technology
Could Enhance TSA's Ability to Identify Threats to the Aviation System:
TSA Lacks Program Effectiveness Measures for SPOT but Is Taking Steps
to Improve Evaluation Capabilities:
TSA Developed and Deployed SPOT Training but Further Action Could
Enhance Its Effectiveness:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: DHS Comments:
Appendix III: GAO Contacts and Staff Acknowledgments:
GAO Contact:
Staff Acknowledgments:
Obtaining Copies of GAO Reports and Testimony:
Tables:
Table 1: Summary of Desirable Characteristics for Developing a
Strategic Plan:
Table 2: Reasons for Arrests from SPOT Referrals, May 29, 2004,
through August 31, 2008:
Table 3: SPOT Instructor Evaluation Ratings, 2006 to September 2008,
and March 2009:
Table 4: TSA Training Standards and Evaluation Branch Recommendations
for Improving SPOT Training and TSA Actions on the Recommendations:
Figures:
Figure 1: TSA's Layers of Aviation Security:
Figure 2: The First Step in the SPOT Process: BDOs Observing
Passengers About to Go Through Checkpoint Magnetometer:
Figure 3: Budget and Personnel Growth in the SPOT Program, Fiscal
Years 2007 through 201027:
Figure 4: Passenger Boardings at SPOT Airports, May 29, 2004, through
August 31, 200844:
Abbreviations:
AMRA: Aviation Modal Risk Assessment:
BDO: Behavior Detection Officer:
CBP: U.S. Customs and Border Protection:
DEA: Drug Enforcement Agency:
DHS: Department of Homeland Security:
FAMS: Federal Air Marshal Service:
FBI: Federal Bureau of Investigation:
ICE: U.S. Immigration and Customs Enforcement:
LEO: Law Enforcement Officer:
NCIC: National Crime Information Center:
NIPP: National Infrastructure Protection Plan:
OMB: Office of Management and Budget:
SOP: Standard Operating Procedures:
SPOT: Screening of Passengers by Observation Techniques:
S&T: Science and Technology Directorate:
TSA: Transportation Security Administration:
TSO: Transportation Security Officer:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
May 20, 2010:
The Honorable John L. Mica:
Ranking Member:
Committee on Transportation:
and Infrastructure House of Representatives:
Dear Mr. Mica:
The terrorist attacks of September 11, 2001, highlighted the need to
improve security within the nation's civil aviation system to deter
persons seeking to repeat similar attacks on the nation's critical
infrastructure. In October 2003, the Transportation Security
Administration (TSA) of the Department of Homeland Security (DHS)
conducted an operational test of the use of behavior detection
techniques to screen passengers in an airport environment, and
subsequently began training certain Transportation Security Officers
(TSO)--TSA employees responsible for screening passengers and their
property--in these techniques. These TSOs performed behavior
observation as a collateral duty. Beginning in fiscal year 2007, TSA
created separate Behavior Detection Officer (BDO) positions as part of
the Screening of Passengers by Observation Techniques (SPOT) program.
[Footnote 1] According to TSA, the SPOT program is a derivative of
other behavioral analysis programs that have been successfully
employed by law enforcement and security personnel both in the United
States and around the world, particularly that of Israel's airline, El
Al.[Footnote 2]
TSA designed SPOT to provide BDOs with a means of identifying persons
who may pose a potential security risk at TSA-regulated airports
[Footnote 3] by focusing on behaviors and appearances that deviate
from an established baseline, and that may be indicative of stress,
fear, or deception. Passengers in an airport terminal, including those
waiting in security checkpoint lines, are observed by the BDOs to
determine if their behavioral and appearance indicators--which are
assigned varying points by SPOT--have (in combination) exceeded a
predetermined numerical threshold. In cases where the passenger
exceeds the threshold, the passenger is referred for additional
screening by BDOs and a TSO. During this referral screening, if the
passenger exhibits behaviors that exceed another numerical threshold,
they are to be referred to a law enforcement officer (LEO) for further
investigation. In addition to observing passengers at airport
checkpoints, BDOs may patrol throughout an airport terminal, and
sometimes participate in other activities, such as TSA's Visible
Intermodal Prevention and Response team operations. These teams are
responsible for periodically augmenting security at air and ground
transportation facilities around the country.[Footnote 4]
As of March 2010, TSA deployed about 3,000 BDOs at an annual cost of
about $212 million; this force increased almost fifteen-fold between
March 2007 and July 2009. BDOs have been selectively deployed to 161
of the 457 TSA-regulated airports in the United States at which
passengers and their property are subject to TSA-mandated screening
procedures.[Footnote 5] The conference report accompanying the fiscal
year 2010 DHS appropriations act provided that $211.9 million of
aviation security funding was for the SPOT program.[Footnote 6] The
administration has requested $232 million for SPOT for fiscal year
2011, a $20.2 million (9.5 percent) increase over the current funding
level. This increase would support a workforce increase from about
3,000 to 3,350 BDOs. If this funding request is approved and
maintained, SPOT would cost about $1.2 billion over the next 5 years.
You asked us to address SPOT's development and implementation. This
report addresses the following questions:
1. To what extent did TSA determine whether SPOT had a scientifically
validated basis for identifying passengers before deploying it and
utilize recognized best practices during SPOT's development?
2. What management challenges, if any, have emerged during the
implementation of SPOT at the nation's airports?
3. To what extent has TSA measured SPOT's effect on aviation security?
4. To what extent has TSA incorporated the attributes of an effective
training program into the training for SPOT?
This report is a public version of the restricted report (GAO-10-
157SU) that we provided to you on May 14, 2010. DHS and TSA deemed
some of the information in the restricted report as sensitive security
information, which must be protected from public disclosure.
Therefore, this report omits this information. Although the
information provided in this report is more limited in scope, it
addresses the same questions as the restricted report. Also, the
overall methodology used for both reports is the same.
To determine the extent to which TSA determined whether SPOT had a
scientifically validated basis for identifying passengers who may pose
a risk to aviation security before deploying it, we reviewed
literature on behavior analysis by subject matter experts, and
analyzed relevant reports and books on the topic. These included a
2008 study by the National Research Council of the National Academy of
Sciences that included a discussion section on deception and
behavioral surveillance, as well as other issues related to behavioral
analysis.[Footnote 7] We interviewed seven recognized experts in the
field, and an expert on emergency responses to terror attacks and
mathematical models in operations management.[Footnote 8] Although the
views of these experts cannot be generalized across all experts on
behavior analysis, because we selected these individuals based on
their publications on behavioral analysis or related topics, their
recognized accomplishments and expertise, and, in some cases, TSA's
use of their work or expertise to design and review the SPOT program's
behaviors, they provided us with an understanding of the fundamentals
of behavior analysis, and its use in airports. We also interviewed
cognizant officials from other U.S. government agencies that utilize
behavior analysis in their work, including U.S. Customs and Border
Protection (CBP), the U.S. Secret Service, the Federal Air Marshall
Service (FAMS), and the Federal Bureau of Investigation (FBI).
[Footnote 9] To better understand how SPOT incorporated expertise on
behavior analysis for aviation security, we also interviewed current
and retired officials of Israel's El Al Airlines, whose security
processes TSA cites as providing part of the basis of the SPOT
program.[Footnote 10]
To determine to what extent TSA utilized best practices during SPOT's
development--including carrying out a comprehensive risk assessment, a
cost-benefit analysis, and a strategic plan--we interviewed program
officials and reviewed related program documentation, including
briefings used in the course of developing and fielding SPOT,
strategic plans, and standard operating procedures.[Footnote 11] We
compared these documents to DHS's 2006 Cost Benefit Analysis
Guidebook,[Footnote 12] Office of Management and Budget (OMB)
guidance,[Footnote 13] and DHS's 2006 and 2009 National Infrastructure
Protection Plans (NIPP), which set forth a risk management framework
to guide security decision making and resource allocation decisions,
and our previous work on the characteristics of an effective strategic
plan.
To identify any challenges that emerged during implementation of the
SPOT program, we conducted field site visits to 15 TSA-regulated
airports with SPOT that represent almost 10 percent of the 161 TSA-
regulated airports with SPOT to observe operations and meet with key
program personnel.[Footnote 14] We chose airports with high, medium,
and low passenger volume; airports with BDOs who are TSA (i.e.,
government) employees and an airport with BDOs employed by contractors
as part of the TSA Screening Partnership Program; and airports with
LEOs who were identified by TSA as having received some form of
behavior detection training and airports where they were not known to
have received such training.[Footnote 15] We also selected airports on
the basis of TSA's assessment of which ones are at highest risk of
attack by terrorists, including the 2 that ranked the highest, as
reported in TSA's Current Airport Threat Assessment.[Footnote 16]
Since the airports we selected range broadly in terms of passenger
volume, physical size and layout, geographic location, and potential
value as a target for terrorism, among other things, the results from
these visits are not generalizable to other airports. However, these
visits provided helpful insights into the operation of SPOT at
airports. In addition, to determine whether challenges emerged in
implementing SPOT, we compared TSA's approach for implementing and
managing SPOT to our Standards for Internal Control in the Federal
Government[Footnote 17] and to risk management principles we had
previously identified.[Footnote 18] In reviewing TSA's approach to
developing and implementing SPOT, we considered relevant laws,
regulations, and other materials, including those related to privacy,
such as TSA's Privacy Impact Assessments. To obtain comparative data
on how SPOT had been implemented at different airports across the
nation, we conducted a survey of all Federal Security Directors
responsible for security operations at TSA-regulated airports with
SPOT.[Footnote 19] (This accounted for all 161 TSA-regulated airports
with SPOT because a single Federal Security Director may be
responsible for several airports.) We obtained a 100 percent response
rate. This survey asked, among other things, about the relationship
between LEOs and the airport authority and BDOs. In addition, to
understand the interaction of BDOs and LEOs, as well as other SPOT
implementation issues, at each of the 15 TSA-regulated airports we
visited we spoke with BDO managers, Federal Security Directors,
Assistant Federal Security Directors, 1 or 2 BDOs, and 1 or 2 LEOs.
To determine the extent to which TSA has measured SPOT's effect on
aviation security, we obtained and analyzed the TSA SPOT referral
database,[Footnote 20] which aims to record all incidents in which
passengers who have passed through the checkpoint are sent to SPOT
referral screening for additional questioning and screening of
property and person. The database also maintains records of instances
where passengers were referred by a BDO to a LEO for questioning. We
assessed the reliability of the SPOT referral data by (1) performing
electronic testing of required data elements, (2) reviewing existing
information about the data and the system that produced them, and (3)
interviewing agency officials knowledgeable about the data. We found a
number of problems related to how the data were collected and recorded
that are discussed later in this report. As a result, we were unable
to use the SPOT referral data to assess whether any behavior or
combination of SPOT behaviors could be used to reliably predict the
final outcome of an incident involving the use of SPOT. However, with
the stated limitations in mind, and after resolving certain
contradictions and anomalies in the database, we utilized the SPOT
referral data to provide examples of information used by TSA to report
on the program's performance, including a count of arrests and the
reasons for those arrests. In addition, to determine if individuals
who were later charged with or pleaded guilty to terrorism-related
offenses had transited SPOT airports and whether TSA could obtain
information from these transits to enhance its understanding of
terrorist behaviors, we reviewed CBP and Department of Justice
information to (1) identify individuals who were charged with or
pleaded guilty to terrorism-related offenses and (2) determine if
these individuals had, prior to being charged, transited airports
where SPOT had been deployed. Further, we used our survey of Federal
Security Directors at SPOT airports to determine the extent to which
video surveillance cameras, which could make video recordings of
terrorists transiting airports, are present at checkpoints.
To assess the extent that SPOT training incorporates the attributes of
an effective training program, we had TSA training experts complete a
training assessment tool that we developed using guidance we prepared
in our previous work for assessing training courses and curricula.
[Footnote 21] To better understand how other entities train their
employees in behavior detection, and what their curricula include, we
conducted site visits to the Secret Service, CBP, FAMS, and the FBI,
and also interviewed nongovernmental experts on aspects of behavior
detection training. We interviewed BDOs and BDO managers about the
SPOT training. In addition, we interviewed El Al officials with regard
to how El Al trains and tests its personnel in behavior recognition
and analysis. Appendix I contains additional details about our scope
and methodology.
We conducted this performance audit from May 2008 through May 2010 in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.
Background:
The Aviation and Transportation Security Act established TSA as the
federal agency with primary responsibility for securing the nation's
civil aviation system, which includes the screening of all passenger
and property transported by commercial passenger aircraft.[Footnote
22] TSA currently has direct responsibility for, or oversees the
performance of, security operations at approximately 457 TSA-regulated
airports in the United States implementing security requirements in
accordance with TSA-approved security programs and other TSA
direction.[Footnote 23] At TSA-regulated airports, prior to boarding
an aircraft, all passengers, their accessible property, and their
checked baggage are screened pursuant to TSA-established procedures,
which include, for example, passengers passing through security
checkpoints where they and their identification documents are checked
by TSOs and Travel Document Checkers, or by Screening Partnership
Program employees.
TSA uses multiple layers of security to deter, detect, and disrupt
persons posing a potential risk to aviation security. These layers
include three principal types of screening employees at airport
checkpoints--Travel Document Checkers, who examine tickets, passports,
and other forms of identification; TSOs, who examine property,
including checked baggage, and persons using x-ray equipment and
magnetometers, as well as other devices; and BDOs, using SPOT to
assess passenger behaviors and appearance.[Footnote 24] BDOs are the
only type of TSA screening employees not deployed to all TSA-regulated
airports and all checkpoints within the airports where it is deployed
on a regular basis. TSA deployed SPOT as an added layer of security to
help deter terrorists attempting to exploit TSA's focus on prohibited
items and other potential security weaknesses. Other security layers
cited by TSA include intelligence gathering and analysis; passenger
prescreening; random canine team searches at airports; federal air
marshals; reinforced cockpit doors; federal flight deck officers; the
passengers themselves; as well as other measures both visible and
invisible to the public. Figure 1 shows TSA's 20 aviation security
layers.
Figure 1: TSA's Layers of Aviation Security:
[Refer to PDF for image: illustration]
Terrorist paths: through the following layers:
Intelligence:
Customs and Border Protection:
Joint Terrorism Task Force:
No-fly List and Passenger Pre-screening[A]:
Crew Vetting:
VIPR:
Canines:
[Start gray bar, screening layers applied to passengers and their
property:
Behavior Detection Officers:
Travel Document Checker:
Checkpoint/Transportation Security Officers:
Checked Baggage[B];
End gray bar, screening layers applied to passengers and their
property]
Transportation Security Inspectors:
Random Employee Screening:
Bomb Appraisal Officers:
Federal Air Marshal Service:
Federal Flight Deck Officers:
Trained Flight Crew:
Law Enforcement Officers:
Hardened Cockpit Door:
Passengers:
Source: TSA.
[A] The No-Fly List is used to identify individuals who should be
prevented from boarding an aircraft; it contains applicable records
from the FBI's Terrorist Screening Center consolidated database of
known or suspected terrorists.
[B] The four layers inside the gray bar are screening layers of
security applied to passengers and their property.
[End of figure]
The gray area in figure 1 highlights four layers that apply to
passengers and their property as they seek to board an aircraft.
Airport LEOs, another layer of security cited by TSA, do not report to
TSA and may not maintain a physical presence at smaller TSA-regulated
airports. According to TSA, each one of these layers alone is capable
of stopping a terrorist attack. In combination, TSA states that their
security value is multiplied, creating a much stronger system, and
that a terrorist who has to overcome multiple security layers in order
to carry out an attack is more likely to be preempted, deterred, or to
fail during the attempt.
SPOT Uses Behavior Detection Techniques to Assess Passenger Behaviors
and Appearances:
The SPOT program utilizes behavior observation and analysis techniques
to identify potentially high-risk passengers. Individuals who exhibit
suspicious behaviors, including both physical and appearance
indicators, may be required to undergo additional screening. Field
agents and law enforcement officers of other federal agencies and
entities--such as the FBI, the Secret Service, CBP, and FAMS--utilize
elements of behavior detection analysis as a part of their work. In
addition, some foreign entities, such as Israel's El Al airlines, use
behavior detection and analysis techniques as part of their security
efforts. However, TSA emphasized to us that the SPOT program is unique
among these entities because it uses a point system to help identify
suspicious persons on the basis of their behavior and appearance and
because behavior detection and analysis are the central focus of SPOT.
Officials from the other agencies stated that their field personnel
incorporate behavior detection as one of many skills used in their
work; in contrast, behavior detection is the primary element of the
BDOs' work.
SPOT trains BDOs to look for and recognize facial expressions, body
language, and appearance that indicate the possibility that an
individual is engaged in some form of deception and fears discovery.
These behaviors and appearances are listed on a SPOT score sheet used
in SPOT training.
Passenger behavior and appearance are to be compared by the BDOs--who
typically work in two-person teams. BDOs are expected to "walk the
line"--that is, to initiate casual conversations with passengers
waiting in line, particularly if their observations led them to
question someone exhibiting behaviors or appearances on the SPOT
checklist. As the BDOs walk the line, and the passenger with SPOT
indicators is reached, a casual conversation is used to determine if
there is a basis for observed behaviors or appearances on the
checklist. In most instances, these conversations provide information
to the BDOs that permits them to consider the issue resolved, and
hence not a security concern. Figure 2 below illustrates the first
step of the three-step SPOT process, the BDO-passenger interaction at
a checkpoint prior to the passenger passing through a magnetometer.
Figure 2: The First Step in the SPOT Process: BDOs Observing
Passengers About to Go Through Checkpoint Magnetometer:
[Refer to PDF for image: illustration]
Step 1:
1. BDOs scan the passengers in line and occasionally initiate casual
conversation.
2. BDOs identify person(s) who exhibit clusters of suspicious
behaviors that meet a given threshold.
3. BDOs identify passengers exhibiting behaviors that exceed SPOT
numerical threshold for referral questioning.
Sources: GAO (analysis), ArtExplosion (clip art), TSA (data).
Note: Circle around passenger shows a person who is exhibiting a
cluster of suspicious behaviors.
[End of figure]
As shown in figure 2, passenger behavior and appearance are observed
by the BDOs as passengers wait in line for screening at a security
checkpoint. Even if the checkpoint is busy, the BDOs must attempt to
visually scan all the passengers waiting in line, as well as persons
near the checkpoint, to determine if any are showing behaviors or
appearances on the SPOT checklist. According to TSA, on average a BDO
has approximately 30 seconds to assess each passenger while the
passenger waits in line. For passengers exhibiting indicators above
baseline conditions, the BDOs are to (mentally) add up the points
assigned to each indicator they observe. Both BDO team members must
agree that observed indicators have exceeded the predetermined
numerical threshold, although they do not have to identify the same
indicators the passenger exhibited. In instances when a passenger's
SPOT indicators place them above the numerical threshold, and the
passenger has placed their property on the conveyor belt for x-raying,
and has walked through the magnetometer or equivalent screening device
for passengers, he or she will be directed to the second step of SPOT,
referral screening. This involves additional questioning and physical
search of their person and property by BDOs and TSOs. This referral
screening occurs in the checkpoint area.
If the passenger's behavior escalates further--accumulating more
points based on the SPOT checklist--the BDOs are to refer the
passenger to a LEO. A referral to a LEO is a potential third step in
the SPOT process. BDOs are not LEOs--they do not conduct criminal
investigations, carry weapons, or make arrests.
After a passenger has been referred by the BDOs to a LEO, the LEO is
then expected to independently determine, through additional
investigation, such as questioning the passenger and, if appropriate,
by conducting an identity verification and background check through
the FBI's National Crime Information Center (NCIC), whether sufficient
grounds exist to take further action, such as detaining or arresting
the passenger. TSA officials who are LEOs also have access to NCIC,
such as an airport's Assistant Federal Security Director for Law
Enforcement or federal air marshals. NCIC is the FBI's computerized
index of criminal justice information (i.e., criminal record history
information, fugitives, stolen properties, and missing persons),
available to federal, state, and local law enforcement and other
criminal justice agencies at all times.[Footnote 25] Similarly, other
federal LEOs also have such access, including CBP, and Drug
Enforcement Agency (DEA) personnel. However, since both local and
federal LEOs have other responsibilities, and may not be present at
each operating checkpoint, BDOs may have to seek them out to request
an NCIC check. According to TSA, aside from requiring that an airport
maintain a law enforcement presence,[Footnote 26] it exercises no
jurisdiction over the law enforcement activities of non-TSA officers
or entities at an airport; thus, it cannot require LEOs to conduct an
NCIC check or to provide BDOs with information about the ultimate
disposition of cases referred by them to LEOs.
Once the LEO concludes his or her investigation and determines whether
the passenger will be arrested or detained, TSA officials are to
evaluate the security concerns to determine whether to allow the
passenger to proceed to the boarding gate. (In some instances, a LEO
might choose not to arrest or detain a passenger; TSA would then
decide whether the infraction was sufficiently serious to necessitate
barring the passenger from boarding.) After a referral incident has
been resolved, BDOs are to enter information about the incident into
TSA's SPOT referral database. The data entered are to include time,
date, location of the incident, behaviors witnessed, prohibited items
found (if any), and information on the LEO's response (if applicable),
such as whether the LEO questioned the passenger, arrested the
individual, or released the passenger. The SPOT referral database
contains no personal identifying information about passengers.
SPOT Has Been Deployed in Phases:
The SPOT program began with pilot tests in 2003 and 2004 at several
New England airports, in which TSA began using uniformed BDOs at
airport checkpoints. After some initial pilot projects and test
deployments, 644 BDOs were deployed to 42 airports in the first phase
of the program from November 2006 through June 2007. As of March 2010,
about 3,000 BDOs utilizing SPOT were deployed at 161 of 457 TSA-
regulated airports.[Footnote 27]
BDO eligibility is restricted to TSOs with at least 12 months of TSO
experience, or others with related security experience. Applicants
must apply and be accepted into the BDO training program. The training
includes 4 days of classroom courses, followed by 3 days of on-the-job
training. BDOs must memorize all of the behaviors and appearances on
the SPOT checklist, as well as the point value assigned to each, in
order to be able to add these up to determine if a passenger should be
sent to SPOT referral screening. BDO applicants must also pass a job
knowledge test at the conclusion of the training. The test includes
related multiple choice questions, true or false statements, and case-
based scenarios.
DHS Is Taking Action to Validate the Scientific Basis of TSA's SPOT
Program but Opportunities Exist to Help Inform Future Program
Decisions:
Although DHS is in the process of validating the way in which the SPOT
program utilizes the science of behavior detection in an airport
environment, TSA deployed SPOT nationwide before first determining
whether there was a scientifically valid basis for using behavior and
appearance indicators as a means for reliably identifying passengers
as potential threats in airports. TSA reported that it deployed SPOT
before a scientific validation of the program was completed in
response to the need to address potential threats to the aviation
system that would not necessarily be detected by existing layers of
aviation security. TSA stated that no other large-scale U.S. or
international screening program incorporating behavior-and appearance-
based indicators has ever been rigorously scientifically validated.
While TSA deployed SPOT on the basis of some risk-related factors,
such as threat information and airport passenger volume, it did not
use a comprehensive risk assessment to guide its strategy of
selectively deploying SPOT to 161 of the nation's 457 TSA-regulated
airports. TSA also expanded the SPOT program over the last 3 years
without the benefit of a cost-benefit analysis of SPOT. Additionally,
TSA's strategic plan for SPOT could be improved by the inclusion of
desirable characteristics identified in our prior work, such as risk
assessment information, cost and resources analysis, and a means for
collaboration with other key entities.
TSA Is in the Process of Validating the Scientific Basis Used to
Identify Passengers with SPOT Behaviors:
TSA proceeded with deploying SPOT on a nationwide basis before
determining whether the list of passenger behaviors and appearances
underpinning the SPOT program were scientifically validated, and
whether these techniques could be applied for counterterrorism
purposes in an airport environment. In 2008, a report issued by the
National Research Council of the National Academy of Sciences noted
that behavior and appearances monitoring might be able to play a
useful role in counterterrorism efforts but stated that a scientific
consensus does not exist regarding whether any behavioral surveillance
or physiological monitoring techniques are ready for use in the
counterterrorist context given the present state of the science.
[Footnote 28] The report also stated that the scientific evidence for
behavioral monitoring is preliminary in nature.[Footnote 29] According
to the report, an information-based program, such as a behavior
detection program, should first determine if a scientific foundation
exists and use scientifically valid criteria to evaluate its
effectiveness before going forward. The report added that programs
should have a sound experimental basis and documentation on the
program's effectiveness should be reviewed by an independent entity
capable of evaluating the supporting scientific evidence. The report
also stated that often scientists and other experts can help
independently assess the scientific evidence on the effectiveness of a
program. A contributor to the National Research Council report also
stated that no conclusive research has been conducted to determine if
behavior detection can be reliably used on a larger scale, such as in
an airport setting, to identify persons intending to cause harm to the
aviation system.
While TSA and DHS's Science and Technology (S&T) Directorate officials
agreed that SPOT was deployed before its scientific underpinnings were
fully validated, they stated that no large-scale U.S. or international
operational screening program incorporating behavior-and appearance-
based indicators has been rigorously scientifically validated. These
officials also questioned the findings of the National Research
Council report and stated that the study lacked sufficient information
for its conclusions because it did not consider recent findings from
unpublished DHS, defense, and intelligence community studies.[Footnote
30] However, National Research Council officials stated that an agency
should be cautious about relying on the results of unpublished
research that has not been peer reviewed, such as that generated by
DHS and the defense and intelligence community, and using unpublished
work as a basis for proceeding with a process, method, or program.
[Footnote 31] Moreover, we have previously reported that peer review
is widely accepted as an important quality control mechanism that
helps prevent the dissemination of potentially erroneous information.
[Footnote 32]
In addition to the unpublished research, TSA told us that the SPOT
program was based on operational best practices from law enforcement,
defense, and the intelligence communities. According to TSA officials,
the agency based its choice of SPOT behavior, appearance, and
deception indicators on existing research and training programs. For
example, TSA cited research on emotions and their behavior indicators
by Dr. Paul Ekman,[Footnote 33] interviewing and interrogation by Stan
Walters,[Footnote 34] and nonverbal indicators by Dr. David Givens
[Footnote 35] and Dr. Mark Frank[Footnote 36] as support for the
choice of several of the behavior indicators. According to TSA, its
development of the SPOT program was based on related DHS research and
information from the training curricula of other federal agencies,
such as the Federal Transit Administration and the Bureau of Alcohol,
Tobacco, Firearms, and Explosives.[Footnote 37]
As with the SPOT behavior indicators, TSA told us that it sought input
in creating the SPOT point scoring system from subject matter experts
and from participants in TSA's SPOT working group, which consisted of
law enforcement officials from agencies such as FBI, DEA, and local
law enforcement officials.[Footnote 38] While TSA officials said that
they coordinated with relevant subject matter experts, such as Dr.
Ekman, and based the SPOT scoring system on existing research and
training programs, no validation of the behavior, appearance, and
deception indicators was conducted prior to the deployment of SPOT in
November 2006. According to TSA officials, they used professional
judgment in developing the SPOT point system and stated that the
purpose of developing the scoring system was to increase the
objectivity of the SPOT process.
Dr. Ekman stated that, in his opinion, and after reviewing the scoring
system and observing the program in operation, it was not clear
whether the SPOT behaviors and appearances, and the related point
system, could be used effectively in an airport environment because no
credible validation research on this issue had been conducted. He
noted, for example, that research is needed to identify how many BDOs
are required to observe a given number of passengers moving at a given
rate per day in an airport environment, or the length of time that
such observation can be conducted before observation fatigue affects
the effectiveness of the personnel. He commented that observation
fatigue is a well-known phenomenon among workers whose work involves
intense observation, and that it is essential to determine the
duration of effective observation and to ensure consistency and
reliability among the personnel carrying out the observations.
DHS has recognized the need to conduct additional research to
scientifically validate the use of the SPOT behavioral indicators in
an airport environment. DHS's S&T Directorate began research in 2007
to determine if there is a statistically significant correlation
between the SPOT behaviors exhibited by airport passengers and finding
airport passengers with prohibited items (such as weapons), false
documents, and illegal drugs or who pose a potential risk to aviation
security. According to S&T, this research is expected to be completed
in fiscal year 2011 and is to include three key elements. First, the
study's purpose is to assess the reliability of the SPOT program by
analyzing TSA's SPOT database to determine patterns of BDO scoring to
measure consistency across BDOs, teams, locations, and other
variables. Second, the study aims to compare the current
implementation of SPOT to random passenger screening. Specifically,
according to S&T officials, 130,000 passengers are to be randomly
selected for additional SPOT referral screening. The study's design
states that data collected about these passengers will be compared to
data for passengers screened through the normal SPOT process. S&T
officials expect that the results of this element of the study will
provide a better understanding of how SPOT compares to random
selection, as well as providing a baseline of each indicator present
in the traveling public. Third, the study also aims to utilize live
and video data, as available, to measure SPOT score ratings by BDOs of
behaviors exhibited by passengers against ratings of the same
passengers by subject matter experts. This element of the study could
help determine whether BDOs are using, or are continuing to use, the
SPOT score sheet correctly as time passes after their initial
training. According to S&T officials, the study is to form the basis
for BDO performance and training requirements.
The S&T Directorate reported some preliminary findings associated with
this research in February 2008. The Directorate reported that although
some of the existing literature supported the possibility of using
behavioral and physiological cues, the results are not
methodologically strong enough to support standardized applications in
an operational setting.[Footnote 39] The preliminary findings also
noted that it is not known whether behavioral and physiological cues
linked to deception in planning a hostile action will be the same or
different as those indicators linked to deception by an individual
after they have already engaged in a hostile action. However, an S&T
program director stated that although early literature can be
characterized as methodologically weak, more recent unpublished
research sponsored by DHS, the Department of Defense, and the
intelligence community is promising in that it has demonstrated some
linkages between behavioral and physiological indicators and
deception.[Footnote 40]
In March 2009, the Under Secretary (Acting) for DHS's S&T Directorate
testified that the Directorate had performed an initial validation of
the behavior indicators used by BDOs.[Footnote 41] The Under Secretary
stated that this analysis provided statistically significant support
that persons demonstrating select behavioral indicators are more
likely to possess prohibited items and that behaviors can distinguish
deceptive from nondeceptive individuals. According to S&T, this
validation was the result of statistical analyses performed by S&T
using operational data from the SPOT program database. However, we
identified weaknesses in TSA's process for maintaining these data. For
example, controls over the SPOT database to help ensure the
completeness and accuracy of the data were missing. Specifically, the
SPOT database did not have computerized edit checks built into the
system to review the format, existence, and reasonableness of data.
For example, we found that discrepancies existed between the number of
passengers arrested by local law enforcement at the screening
checkpoints and the number of screened passengers recorded as
arrested. In another example, we found that the total number of LEO
referrals differed from the number of passenger records with
information on the reasons for LEO referral. Internal control
standards state that controls should be installed at an application's
interfaces with other systems to ensure that all inputs are received
and are valid and that outputs are correct and properly distributed.
[Footnote 42] TSA officials explained these issues as data anomalies
and planned to change instructions to staff entering data to reduce
these problems. Although TSA is taking steps to update the SPOT
database, which are discussed later in this report, the data used by
S&T to conduct its preliminary validation of related behaviors lacked
such controls. In addition, BDOs could not input all behaviors
observed in the SPOT database because the database limits entry to
eight behaviors, six signs of deception, and four types of prohibited
items per passenger referred for additional screening. Because of
these data-related issues, meaningful analyses could not be conducted
to determine if there is an association between certain behaviors and
the likelihood that a person displaying certain behaviors would be
referred to a LEO or whether any behavior or combination of behaviors
could be used to distinguish deceptive from nondeceptive individuals.
As a result, TSA lacks assurance that the SPOT data can be used
effectively to determine that the person poses a risk to aviation
security. S&T has recognized weaknesses in the procedures for
collecting data on passengers screened by SPOT and plans to more
systematically collect data during its study by, for example,
requiring BDOs to record more complete and accurate information
related to a passenger referral immediately following resolution.
The S&T study is an important step to determine whether SPOT is more
effective at identifying passengers who may be threats to the aviation
system than random screening. However, S&T's current research plan is
not designed to fully validate whether behavior detection and
appearances can be effectively used to reliably identify individuals
in an airport terminal environment who pose a risk to the aviation
system. For example, research on other issues, such as determining the
number of individuals needed to observe a given number of passengers
moving at a given rate per day in an airport environment or the
duration that such observation can be conducted by BDOs before
observation fatigue affects effectiveness, could provide additional
information on the extent to which SPOT can be effectively implemented
in airports. In another example, Dr. Ekman told us that additional
research could help determine the need for periodic refresher training
since no research has yet determined whether behavior detection is
easily forgotten or can be potentially degraded with time or lack of
use. While S&T officials agree on the need to validate the science of
behavior detection programs, they told us that some of these other
issues could be examined in the future but are not part of the current
plan due to time and budgetary constraints. According to S&T, some
additional analysis is underway to inform the current BDO selection
process. This analysis is intended to provide information on the
knowledge, skills, abilities, and other characteristics of successful
BDOs. Since the analysis is scheduled for completion in May 2010, it
remains unclear to what extent the findings will help to validate the
science related to SPOT. While we recognize the potential benefits of
these efforts, we believe that an assessment by an independent panel
of experts of the planned methodology of DHS's study could help DHS
assess the costs and benefits associated with a more comprehensive
methodology designed to fully validate the science related to SPOT.
Our prior work has recommended the use of such independent panels for
comprehensive, objective reviews of complex issues.[Footnote 43] In
addition, according to the National Research Council, an independent
panel could provide an objective assessment of the methodology and
findings of DHS's study to better ensure that SPOT is based on
validated science. Thus, an independent panel of experts could help
DHS develop a comprehensive methodology to determine if the SPOT
program is based on valid scientific principles that can be
effectively applied in an airport environment for counterterrorism
purposes.
SPOT Was Deployed Nationwide on Basis of Threat, but Without a
Comprehensive Risk Assessment:
According to DHS's National Infrastructure Protection Plan (NIPP),
risk assessments are to be documented, reproducible (so that others
can verify the results), defensible (technically sound and free of
significant errors), and complete. The NIPP states that comprehensive
risk assessments are necessary for determining which assets or systems
face the highest risk, for prioritizing risk mitigation efforts and
the allocation of resources, and for effectively measuring how
security programs reduce risks. For a risk assessment to be considered
complete, the NIPP states that it must specifically assess threat,
vulnerability, and consequence;[Footnote 44] after these three
components have been assessed, they are to be combined to produce a
risk estimate.[Footnote 45]
According to TSA, SPOT was deployed to TSA-regulated airports on the
basis of threat information in TSA's Current Airport Threat Assessment
list.[Footnote 46] TSA deployed SPOT to 161 of 457 TSA-regulated
airports. TSA officials told us that this selective deployment creates
unpredictability for persons seeking to cause harm to the aviation
system because they would not know which airports had BDO teams and
because BDOs are occasionally sent out to the smaller airports that do
not have BDOs on a permanent basis. Although TSA's selective
deployment of SPOT was based on threat information, TSA did not
conduct vulnerability and consequence assessments to inform the
deployment of BDOs. As a result, it could not combine the results to
conduct a comprehensive risk assessment to inform the deployment of
BDOs to those airports with the highest risks.
TSA officials told us that while they have not completed a
comprehensive risk assessment for airport security, they have prepared
and are currently reviewing a draft of a comprehensive, scenario-based
Aviation Modal Risk Assessment--known as the AMRA--which is to serve
as a comprehensive risk assessment for aviation security.[Footnote 47]
According to TSA officials, the AMRA is to address all three elements
of risk for domestic commercial aviation, general aviation, and air
cargo.[Footnote 48] Although TSA planned to release the AMRA in
February 2008, it now expects to finalize the AMRA in 2010. According
to TSA, the AMRA may help provide information for the prioritization
of BDO deployment within airports, but could not provide specifics on
how it would do so. Further, TSA officials noted that information from
AMRA would inform BDO deployment in conjunction with other TSA
priorities not related to SPOT.[Footnote 49] Since the AMRA is not yet
complete, it is not clear whether it will provide the risk analysis--
including assessments of vulnerability and consequence--needed to
inform TSA's decisions and planning for any revisions or future
deployment of SPOT. If AMRA lacks information relevant to the
deployment of SPOT and further research determines that SPOT has a
scientifically validated basis for using behavior detection for
counterterrorism purposes in the airport environment, then conducting
a comprehensive risk assessment of airports could strengthen TSA's
ability to establish priorities and make cost-effective resource
decisions regarding the deployment of BDOs to those airports deemed to
have the highest priority risks.
TSA Deployed SPOT Nationwide Without Conducting a Cost-Benefit
Analysis but Such an Analysis Could Help Inform Program Decisions
Moving Forward:
DHS and other federal guidance recommend conducting a cost-benefit
analysis before implementing new programs to avoid unnecessary costs
and identify the best way to achieve goals at the lowest costs among
potential alternatives. Our prior work has also supported the use of
cost-benefit analyses during retrospective reviews to validate the
agency's original assumptions regarding costs and benefits.[Footnote
50] In addition, the DHS February 2006 Cost-Benefit Analysis Guidebook
and OMB guidance both recommend the use of cost-benefit analysis, both
in the planning stage for a program, and when significant milestones
or financial options are to be assessed.[Footnote 51] The DHS
Guidebook states that a cost-benefit analysis is designed to identify
optimal financial solutions among competing alternatives. OMB guidance
also identifies cost-benefit analysis as one of the key principles to
be considered when making capital expenditures, that expected benefits
of proposed actions should be explained, and that a baseline should be
identified discussing costs and benefits in comparison with clearly
defined alternatives. DHS's 2006 and 2009 NIPPs also state that
priority is to be given to those protective measures that provide the
greatest mitigation of risk for the resources that are available. The
DHS NIPPs add that effective protective programs seek to use resources
efficiently by focusing on actions that offer the greatest mitigation
of risk for any given expenditure. In addition, measuring cost
effectiveness of SPOT was a key TSA goal in an October 2005 version of
the SPOT strategic plan.
Although the DHS and OMB guidance recommend that a cost-benefit
analysis be conducted prior to deploying a program nationwide--and
potentially incurring substantial costs--TSA did not conduct such an
analysis of SPOT to inform its pilot testing prior to full-scale
nationwide deployment. In early 2003, TSA began conducting a pilot
test of the SPOT program at Boston Logan airport to better understand
the benefits of the program. According to Boston Logan's Federal
Security Director, the primary purpose of this pilot test was to
understand the potential of the program, not to validate its success.
[Footnote 52] TSA officials stated that the program had several
benefits, one of which was its "negligible cost." However, TSA did not
analyze the pilot test results to determine if SPOT was more cost
effective than other alternatives, such as random screening of
passengers. In October 2004, TSA implemented additional pilot programs
in Providence, Rhode Island and Portland, Maine with the goal of
providing Federal Security Directors with an additional layer of
security to identify high-risk passengers for additional screening
using behavior detection techniques. TSA concluded that the pilot
program was successful and cited several security benefits of these
pilots. For example, TSA personnel in Providence identified two
individuals in possession of illegal drugs, who were then arrested.
Law enforcement also arrested another individual referred to them for
providing a fraudulent passport. In another example, BDOs in Portland
discovered a passenger with multiple passports and a hidden luggage
compartment. The passenger was interviewed by LEOs and later released.
TSA determined that these initial pilot tests at three airports were
successful without comparing pilot test data to other possible
security alternatives. For example, the results of random screening of
passengers at the pilot airports could have provided TSA with
objective baseline data. Specifically, these data could have been
compared to data collected during the SPOT pilots to determine if SPOT
was more effective than random screening in detecting passengers who
pose a potential risk to aviation security. TSA concluded that the
pilot tests were successful because pilot airports were able to easily
incorporate SPOT into their security program, train personnel in SPOT,
and implement procedures for an additional layer of security according
to TSA.
TSA conducted additional pilot tests at the Minneapolis-St. Paul,
Minnesota and Bangor, Maine airports in October 2005. TSA also
deployed the program to nine additional airports in response to TSA's
holiday preparedness plan in December 2005 to further operationally
test the program. Senior SPOT program officials explained that TSA did
not conduct an analysis of the pilot testing because the program was
in its infancy and officials were focused on deploying SPOT to
additional airports. Since that time, TSA has not conducted a cost-
benefit analysis, which could help the agency establish the value of
the program relative to other layers of aviation security. Moreover, a
cost-benefit analysis could also be useful considering recent program
growth. For example, from fiscal year 2007 through fiscal year 2009,
TSA allotted about $383 million for SPOT. During this period, SPOT's
share of TSA's total screening operations budget increased from 1
percent to 5 percent.[Footnote 53] The conference report accompanying
the fiscal year 2010 DHS appropriations act designates $212 million of
the appropriated aviation security funding for the SPOT program.
[Footnote 54] A cost-benefit analysis could have provided TSA
management with analysis on whether this allocation was a prudent
investment, as well as whether this level of investment in SPOT is
appropriate. Figure 3 shows the growth in the budget and personnel
numbers for SPOT from fiscal years 2007 through 2010.
Figure 3: Budget and Personnel Growth in the SPOT Program, Fiscal
Years 2007 through 2010:
[Refer to PDF for image: 2 vertical bar graphs]
Fiscal year: 2007;
Total actual cost: $14 million;
Actual BDO allocation: 589.
Fiscal year: 2008;
Total actual cost: $144 million;
Actual BDO allocation: 2,011.
Fiscal year: 2009;
Total actual cost: $198 million;
Actual BDO allocation: 2,860.
Fiscal year: 2010;
Total appropriated: $212 million;
Total BDO allocation: 2,986.
Source: GAO analysis of TSA data.
Note: The actual BDO allocation for fiscal year 2009 is as of June
2009. The appropriated amount for SPOT for fiscal year 2010 is the
amount reflected in the conference report accompanying the fiscal year
2010 DHS appropriations act. The appropriated amounts prior to fiscal
year 2010 cannot be determined because funding was appropriated as a
lump sum with funding for other screeners and the relevant conference
reports did not allocate a specific amount for SPOT. BDO allocation
figures are full-time equivalents.
[End of figure]
SPOT's Strategic Plan Could be Strengthened by Addressing Key
Characteristics of a Successful Strategy:
Our previous work,[Footnote 55] and the Government Performance and
Results Act,[Footnote 56] set forth several key elements of a
strategic plan. Such plans can guide agencies in planning and
implementing an effective government program. Table 1 summarizes the
desirable characteristics of an effective strategic plan, as
identified in our prior work. In April 2009, we reported that these
characteristics are the starting point for developing a strategic
plan.[Footnote 57]
Table 1: Summary of Desirable Characteristics for Developing a
Strategic Plan:
Desirable characteristic: Purpose, scope, and methodology;
Description: Addresses why the plan was produced, the scope of its
coverage, and the process by which it was developed.
Desirable characteristic: Problem definition and risk assessment;
Description: Addresses the particular problems and threats the plan is
directed towards.
Desirable characteristic: Goals, subordinate objectives, activities,
and performance measures;
Description: Addresses what the plan is trying to achieve, steps to
achieve those results, as well as the priorities, milestones, and
performance measures to gauge results.
Desirable characteristic: Resources, investments, and risk management;
Description: Addresses what the plan will cost, the sources and types
of resources and investments needed, and where resources and
investments should be targeted based on balancing risk reductions with
cost.
Desirable characteristic: Organizational roles, responsibilities, and
coordination;
Description: Addresses who will implement the plan, what their roles
will be compared to others, and mechanisms for them to coordinate
their efforts.
Desirable characteristic: Integration and implementation;
Description: Addresses how the plan relates to the agency's other
goals, objectives, and activities, to other federal and nonfederal
entities involved in implementation or coordination, and their plans
to implement the strategic plan.
Source: GAO analysis based on GAO-09-369 and GAO-04-408T.
[End of table]
TSA officials at Boston Logan airport told us that they completed the
first strategic plan for SPOT in 2006. The strategic plan was last
updated in March 2007. The March 2007 plan includes some of the
desirable characteristics described above, such as an overall purpose.
However, incorporating additional characteristics of an effective
strategic plan could enhance the plan's usefulness in program
management and resource allocation decisions to effectively manage the
deployment of SPOT if TSA determines that the program has a
scientifically valid basis. TSA officials stated that they believed
the plan was sufficiently comprehensive to develop a national program,
such as SPOT. However, these officials told us that the plan was not
updated after TSA expanded the program in 2008 and 2009. They also
stated that the program's focus remained on deploying SPOT to
additional airports. Our assessment of the extent to which the SPOT
strategic plan addresses these characteristics is presented below.
Purpose, scope, and methodology: The SPOT strategic plan addresses why
the plan was developed (i.e., purpose) and the scope of its coverage.
Specifically, the plan describes a strategy to utilize behavior
detection screening as an additional layer of security. The plan also
notes that the primary focus is to expand SPOT in the aviation
environment while also developing a capability to deploy BDOs to
support security efforts in all modes of transportation. However, the
plan does not discuss the process by which it was developed (i.e.,
methodology). According to TSA, officials responsible for developing
the plan received input from relevant stakeholders at Boston Logan
airport and TSA headquarters. We believe incorporating the methodology
into the plan could make the document more useful to TSA and other
organizations, such as local law enforcement, responsible for
implementing the plan.
Problem definition and risk assessment: The plan addresses the
particular threat it is directed towards. Specifically, the plan
describes the need to implement SPOT to counter terrorist activities,
improve security, and incorporate additional layers of protection
within aviation security. However, the plan does not incorporate risk
assessment information to identify priorities or guide program
implementation because TSA has not conducted a comprehensive risk
assessment related to the deployment of SPOT.[Footnote 58] Using
available risk assessment information to inform the development of a
strategic plan would help ensure that clear priorities are established
and focused on the areas of greatest need. Specifically, incorporating
the results of a risk assessment in the program's strategic plan could
help inform TSA's decisions such as whether to deploy SPOT to
additional TSA-regulated airports, to shift SPOT teams from one
airport to another, or to remove SPOT at airports where the benefit of
addressing the risk does not outweigh the costs, as well as to
identify and communicate the risks to aviation security if SPOT was
not deployed to all TSA-regulated airports.
Goals, subordinate objectives, activities, and performance measures:
The plan outlines several goals, objectives, and activities for the
SPOT program to achieve. For example, the plan outlines a goal to
develop multimodal partnerships, including at the local level, to
support SPOT. An associated objective for this goal includes
identifying and fostering advocates within each mode of transportation
by developing transportation, intelligence, and law enforcement
working groups with relevant officials to share information and foster
cooperation. The plan also includes a goal to develop and implement
performance measures for SPOT. However, the plan did not include
performance measures for SPOT. Incorporating performance measures into
the plan could help TSA officials measure progress in implementing the
plan's goals, objectives, and activities.
Resources, investments, and risk management: The plan does not
identify the costs and resources needed to achieve program objectives
discussed in the plan. Incorporating information about cost and
resources would facilitate TSA's ability to allocate resources across
programs according to priorities and constraints, track costs and
performance, and shift such investments and resources as appropriate.
Organizational roles, responsibilities, and coordination: The SPOT
program relies on a close partnership with law enforcement officers at
airports. TSA provides briefings to law enforcement on the SPOT
program, and TSA officials conduct outreach efforts to local law
enforcement as needed. The SPOT SOP guidance and SPOT training include
guidance about ensuring that LEOs receive complete and accurate
information about each SPOT referral. However, while the strategic
plan identifies TSA officials and offices as responsible parties for
implementing the strategic plan, it does not provide guidance on how
to effectively link the roles, responsibilities, and capabilities of
federal, state, and local officials providing program support.
Moreover, although SPOT SOP guidance discusses the need for BDOs to
coordinate with other TSA personnel, such as TSOs and TDCs, TSA does
not identify their roles and responsibilities in regards to the SPOT
program in the program's strategic plan. Integrating these elements
into the strategic plan could help to clarify the relationships
between these various implementing parties, which would thereby
increase accountability and improve the effectiveness of
implementation.
Integration and implementation: The SPOT strategic plan does not
discuss how its scope complements, expands upon, or overlaps with
other related strategic documents. For example, TSA's April 2008
Office of Security Operations Organizational Business Plan for Fiscal
Year 2010 describes how its goals--including those for SPOT--relate to
DHS and TSA strategic goals.[Footnote 59] However, TSA does not link
goals in the SPOT strategic plan with other related strategic
documents, such as the Aviation Implementation Plan of DHS's
Transportation Systems Sector-Specific Plan[Footnote 60] and the
Passenger Checkpoint Screening Program Strategic Plan.[Footnote 61] By
linking goals in its SPOT strategic plan to other TSA efforts, TSA
could better ensure that the program's objectives are integrated with
other TSA security programs and that resources are used effectively by
minimizing any unnecessary duplication with these other actions.
More Fully and Consistently Utilizing Available Information Technology
Could Enhance TSA's Ability to Identify Threats to the Aviation System:
Inconsistencies in the use of available information technology to aid
in the collection and recording of data on passengers by BDOs during
referrals to LEOs, lack of guidance on, or a mechanism for, BDOs to
request the TSA's Transportation Security Operations Center to run the
names of passengers exhibiting suspicious behaviors against law
enforcement and intelligence databases, and the Center's not checking
all of the databases available to it--have limited TSA's ability to
identify potential terrorist threats to the aviation system.[Footnote
62] Among other information, these databases include terrorism-related
watch lists.
Systematic Collection of Data on Passengers Identified Through the
SPOT Program Could be Improved to Better Identify Activity Potentially
Harmful to the Aviation System:
TSA is not fully utilizing the resources it has available to
systematically collect the information obtained by BDOs on passengers
whose behaviors and appearances resulted in either SPOT referral
screening, or in a referral to LEOs, and who thus may pose a risk to
the aviation system. TSA's July 2008 Privacy Impact Assessment on the
TSA Transportation Security Operations Center, and its August 2008
Privacy Impact Assessment on SPOT, state that information may be
obtained by BDOs to check an individual's identity against
intelligence, terrorist, and law enforcement databases and to permit
intelligence analysts to conduct trend analysis.[Footnote 63]
The August 2008 SPOT Privacy Impact Assessment states that information
about a passenger who has exceeded the SPOT behavior threshold,
leading to LEO referral, may be collected and entered into DHS's
Transportation Information Sharing System.[Footnote 64] According to
the SPOT Privacy Impact Assessment, information collected may be
submitted to the Transportation Information Sharing System database
for analysis, and, through it to other linked intelligence databases
and the intelligence analysts who study them, to detect, deter, and
defeat a criminal or terrorist act in the transportation domain before
it occurs. The SPOT Privacy Impact Assessment notes that terrorist
acts that threaten transportation security are most vulnerable in the
planning stages and that the timely passage of SPOT referral
information may assist in identifying such efforts before they become
operational. A June 2008 Transportation Information Sharing System
Privacy Impact Assessment similarly states that one goal is to use the
system data to find trends and patterns that may indicate
preoperational terrorist or criminal activity--that is, to "connect
the dots" about a planned terrorist attack or criminal enterprise.
Information in TSA's Transportation Information Sharing System is
primarily activity or behavioral information but may also contain
personal information regarding the individuals identified by the BDO
through SPOT. According to TSA, BDOs do not analyze the data obtained
during referrals; if they have the appropriate training, they may
enter the data by computer into the Transportation Information Sharing
System, where they can be analyzed by intelligence analysts. Other
appropriately trained and officially designated TSA officials, such as
Federal Security Directors, may also enter data into the system.
According to TSA, a 2008 pilot program it conducted that involved BDOs
entering data into the Transportation Information Sharing System
database was "invaluable," in part because over 40 referrals have
since been passed on to other LEO organizations for further
investigation, most of which came from BDO input. A February 2006 TSA
memorandum describes the Transportation Information Sharing System as
"a critical element in the success of SPOT" because it provides the
necessary platform for the reporting of information obtained as a
result of SPOT referrals. TSA noted that through the use of the
Transportation Information Sharing System, two different BDO teams had
separately identified and selected the "same extremist" for secondary
questioning.[Footnote 65] TSA officials also told us about an incident
in which an individual sought to board an aircraft with a handgun on
two separate occasions, at two different airports. Although the
handgun was detected both times, the individual was released after
providing what seemed to be a credible explanation. After the second
incident, however, intelligence analysts who reviewed the system
information saw that this individual had tried twice in 2 weeks to
bring a weapon onto an aircraft. A LEO was dispatched to the person's
home, and an arrest was made. Without the data inputted into the
system both times, no pattern would have been detected by the
analysts, according to TSA. Although the pilot program illustrated the
benefits of BDOs entering data into the system, access to the system
was not expanded to all SPOT airports in 2008 or 2009.
Internal control standards call for management to develop policies,
procedures, and techniques to help enforce management directives. TSA
does not provide official guidance on how or when BDOs or other TSA
personnel should enter data into the Transportation Information
Sharing System or which data should be entered. Official guidance on
what data should be entered into the system on passengers could better
position TSA personnel to be able to consistently collect information
to facilitate synthesis and analysis in "connecting the dots" with
regard to persons who may pose a threat to the aviation system.
On March 18, 2010, TSA officials told us that TSA recognizes the value
of recording SPOT incidents for the purposes of intelligence
gathering. As a result, TSA decided that certain data would be entered
into the Transportation Information Sharing System, and would, in
turn, be analyzed as a way to potentially "connect the dots" with
other transportation security incidents.[Footnote 66]
TSA officials said that the Federal Security Director at each SPOT
airport has been given the discretion to decide which personnel should
have access to the Transportation Information Sharing System. However,
TSA has not developed a plan detailing how many personnel would have
access to the system, or when they would have access at SPOT airports.
TSA officials said that training is currently being provided to
personnel responsible for using the system at all SPOT airports
although they did not provide information on the number being trained.
Standard practices for defining, designing, and executing programs
include developing a road map, or program plan, to establish an order
for executing specific projects needed to obtain defined programmatic
results within a specified time frame. However, TSA stated that it has
not developed a schedule or milestones by which database access will
be deployed to SPOT airports, or a date by which access at all SPOT
airports will be completed. Setting milestones for expanding
Transportation Information Sharing System access to all SPOT airports,
and setting a date by which the expansion will be completed, could
better position TSA to identify threats to the aviation system that
may otherwise go undetected and help TSA track its progress in
expanding Transportation Information Sharing System access as
management intended.
Guidance on and a Mechanism for Running Names of Referred Passengers
Through the Databases Available to the Transportation Security
Operations Center Could Help Improve SPOT Practices:
Internal control standards state that policies, procedures,
techniques, and other mechanisms are essential to help ensure that
actions are taken to address program risks.[Footnote 67] The current
process makes the BDOs dependent on the LEOs with regard to the
timeliness that LEOs respond to BDO calls for service, as well as with
regard to whether the LEOs choose to question the passengers referred
to them or conduct a background check. Our analysis of the SPOT
referral database found a wide variation in the percent of times that
LEOs responded to calls for service at SPOT airports.[Footnote 68]
Moreover, if a local LEO decides to run a background check on a
passenger referred to them, they would be accessing the FBI's NCIC and
not other intelligence and law enforcement databases.
Although LEOs may not always respond to calls for service, question
passengers, or check passenger names against databases available to
TSA, TSA has not developed a mechanism allowing BDOs to send
information to the Transportation Security Operations Center about
passengers whose behavior indicates that they may be a possible threat
to aviation security. According to TSA's July 2008 Transportation
Security Operations Center Privacy Impact Assessment, passenger
information may be submitted to the Transportation Security Operations
Center to ascertain, as quickly as possible, the individual's
identity, whether they are already the subject of a terrorist or
criminal investigation, or to analyze suspicious behavior that may
signal some form of preoperational surveillance or activity.[Footnote
69]
Our survey of Federal Security Directors at SPOT airports found a
notable inconsistency in the rates at which BDOs at different airports
contacted the Transportation Security Operations Center.[Footnote 70]
Developing additional guidance in the SPOT operating procedures could
help improve consistency in the extent to which BDOs utilize
Transportation Security Operations Center resources. Given the range
of responses we received from SPOT airports about whether the BDOs
contact the Transportation Security Operations Center to verify
passenger identities and run their names against terrorist and
intelligence databases and the inconsistencies identified related to
LEO responses to BDO requests for service, developing a standard
mechanism and providing BDOs with additional guidance could help TSA
achieve greater consistency in the SPOT process. Such a mechanism
would provide designated TSA officials with a means of verifying
passenger identities and help them determine whether a passenger was
the subject of a terrorist or criminal investigation and thus posed a
risk to aviation security.
Standards for internal control state that effectively using available
resources, including key information databases, is one element of
functioning internal controls.[Footnote 71] In this connection, it is
widely recognized among intelligence entities and police forces that a
capability to "run" names against databases that contain criminal and
other records is a potentially powerful tool to both identify those
with outstanding warrants and to help discover an ongoing criminal or
security-related incident. Additionally, TSA recommended in an April
2008 Organizational Business plan for its Office of Security
Operations that the SPOT program should establish a mechanism and
policy for allowing real-time checks of federal records for
individuals whose behavior indicates they may be a threat to security.
[Footnote 72] The Office of Security Operations plan also states that
BDOs should communicate the data to U.S. intelligence centers, with
the purpose of permitting rapid communication of this information to
local LEOs to take action. However, TSA officials told us that because
of safety concerns, the Transportation Security Operations Center does
not provide information from database checks directly to BDOs because
BDOs are not LEOs, are unarmed, and do not have the training needed to
deal with potentially violent persons.[Footnote 73] If the mechanism
discussed in the Office of Security Operations business plan were
implemented, it would allow the Transportation Security Operations
Center to use BDO information to conduct real-time record checks of
passengers and communicate the results to LEOs for action. Such a
mechanism could increase the chances to detect ongoing criminal or
terror plans.
TSA's Transportation Security Operations Center Does Not Use All
Database Resources When Contacted:
The final report of the National Commission on Terrorist Attacks Upon
the United States (the "9/11 Commission Report") recommends that in
carrying out its goal of protecting aviation, TSA should utilize the
larger set of information maintained by the federal government, that
is, the entire Terrorist Screening Database--the U.S. government's
consolidated watch list that contains information on known or
suspected international and domestic terrorists--as well as other
government databases, such as intelligence or law enforcement
databases.[Footnote 74] However, the Transportation Security
Operations Center is not using all the resources at its disposal to
support BDOs in verifying potential risks to the aviation system. This
reduces the opportunities to "connect the dots" that would increase
the chances of detecting terrorist attacks in their planning stage,
which the SPOT Privacy Impact Assessment states is when they are the
most vulnerable.
According to TSA, the Transportation Security Operations Center has
access to multiple law enforcement and intelligence databases that can
be used to verify the identity of airline passengers; these include
among others:[Footnote 75]
1. the Selectee list, which identifies persons who must undergo
enhanced screening at the checkpoint prior to boarding;
2. the No-Fly list,[Footnote 76] which lists persons prohibited from
boarding aircraft; and:
3. the Terrorist Identity Datamark Environment terrorist list.
[Footnote 77]
TSA stated that the Transportation Security Operations Center checks
passenger names submitted to it against these three databases if the
passenger has been referred by a BDO to a LEO, but has not been
arrested. Of the three databases that the Transportation Security
Operations Center is to check in the case of a referral, passengers
would have already been screened against two--the Selectee and No-Fly
lists--in accordance with TSA passenger prescreening procedures when
purchasing a ticket. The third database checked--the Terrorist
Identity Datamark Environment--tracks terrorists but not persons
wanted for other crimes. The FBI's NCIC information system would
contain names of such persons, but is not among the three databases
checked for nonarrest referrals. If the passenger has been arrested,
the Transportation Security Operations Center will run the passenger's
name against the additional law enforcement and intelligence databases
available to it.
In addition, TSA told us that the Operations Center does not have
direct electronic access to the Terrorist Screening Database and must
call the FBI's Terrorist Screening Center to provide it with a name to
verify. TSA stated that this is done if a passenger's identity could
not be verified using the Operations Center databases. In effect, if a
passenger has been referred to a LEO, but not arrested, the Operations
Center is to check the three databases shown above to verify the
passenger's identity. If a passenger has been arrested, but the three
databases do not list the person, the Center can check the additional
databases available to it. If none of these databases can verify the
person's identity, the Operations Center can contact the Terrorist
Screening Center by telephone to request further screening.
For passengers who have risen to the level of a LEO referral at an
airport checkpoint, having the Transportation Security Operations
Center consistently check their names against all the databases
available to it could potentially help TSA identify threats to the
aviation system and aid in "connecting the dots." TSA indicated that
there are no obstacles to rapidly checking all databases rather than
the three listed. We did not analyze the extent to which the law
enforcement and intelligence databases available to TSA may contain
overlapping information.
TSA Lacks Program Effectiveness Measures for SPOT but Is Taking Steps
to Improve Evaluation Capabilities:
TSA has established some performance measures by tracking SPOT
referral and arrest data, but lacks the measures needed to evaluate
the effectiveness of the SPOT program and, as a result, has not been
able to fully assess SPOT's contribution to improving aviation
security. TSA emphasized the difficulty of developing performance
measures for deterrence-based programs, but stated that it is
developing additional measures to quantify the effectiveness of the
program. The SPOT program uses teams to assess BDO proficiency,
provide individual and team guidance, and address issues related to
the interaction of BDOs with TSA checkpoint personnel. However, TSA
does not systematically track the teams' recommendations or the
frequency of the teams' airport visits. TSA states that it is working
to address these issues and plans to do so by the end of fiscal year
2010.
TSA Has Taken Action to Collect Data for Some Performance Measures,
but Work Remains to Assess Progress Towards Achieving Strategic Goals:
TSA agreed that the SPOT program lacked sufficient performance
measures in the past, but stated that it has some performance measures
in place including tracking data on passengers referred for additional
screening and the resolution of this screening, such as if prohibited
items were found or if law enforcement arrested the passenger and the
reason for the arrest. TSA is also working to improve its evaluation
capabilities to better assess the effectiveness of the program. DHS's
NIPP, internal controls standards, and our previous work on program
assessment state that performance metrics and associated program
evaluations are needed to determine if a program works and to identify
adjustments that may improve its results.[Footnote 78] Moreover,
standard practices in program management for defining, designing, and
executing programs include developing a road map, or program plan, to
establish an order for executing specific projects needed to obtain
defined programmatic results within a specified time frame.[Footnote
79] Congress also needs information on whether and in what respects a
program is working well or poorly to support its oversight of agencies
and their budgets; and agencies' stakeholders need performance
information to accurately judge program effectiveness.[Footnote 80]
For example, in the Senate Appropriations Committee report
accompanying the fiscal year 2010 DHS appropriations bill,[Footnote
81] the committee noted that while TSA has dramatically increased the
size and scope of SPOT, resources were not tied to specific program
goals and objectives. In addition, the conference report accompanying
the fiscal year 2010 DHS appropriations act requires TSA to report to
Congress, within 60 days of enactment, on the effectiveness of the
program in meeting its goals and objectives, among other things.
[Footnote 82] This report was completed on March 15, 2010.
Although TSA tracks data related to SPOT activities including
prohibited items, law enforcement arrests related to SPOT referrals,
and reasons for the arrests (output measures), it has not yet
developed measures to gauge SPOT's effectiveness in meeting TSA
strategic goals (outcome measures), such as identifying individuals
who may pose a threat to the transportation system.[Footnote 83] OMB
encourages the use of outcome measures because they are more
meaningful than output measures, which tend to be more process-
oriented or means to an end.[Footnote 84] For example, TSA's Assistant
General Manager for the Office of Operation Process and Performance
Metrics[Footnote 85] told us that SPOT staffing levels are currently
used as one performance metric. The official said that since the SPOT
is an added layer of security, additional SPOT staffing would add to
security effectiveness. While staffing levels may help gauge how fast
the program is growing, they do not measure the effectiveness in
meeting strategic goals.
Similarly, TSA also cited the number of prohibited items discovered by
BDOs in SPOT metrics reports as a measure of program success.[Footnote
86] However, TSA told us that possession of a prohibited item is often
an oversight and not an intentional act; moreover, other checkpoint
screening layers are intended to find such items, such as the TSOs and
the property screening equipment.[Footnote 87] TSA also cited measures
of BDO job performance as some of the existing measures of program
effectiveness, but noted that these are "pass/fail" assessments of
individual BDOs, rather than overall program measures.
TSA notes that one purpose of the SPOT program is to deter terrorists,
but that proving that it has succeeded at deterring terrorists is
difficult because the lack of data has presented challenges for the
SPOT program office when developing performance measures. We agree
that developing performance measures, especially outcome measures, for
programs with a deterrent focus is difficult. Nevertheless, such
measures are an important tool to communicate what a program has
accomplished and provide information for budget decisions. TSA uses
proxy measures--indirect measures or indicators that approximate or
represent the direct measure--to address deterrence, other security
goals, or a combination of both. For example, TSA tracks the number of
prohibited items found and individuals arrested as a result of SPOT
referrals. According to OMB, proxy measures are to be correlated to an
improved security outcome, and the program should be able to
demonstrate--such as through the use of modeling--how the proxies tie
to the eventual outcome.[Footnote 88]In using a variety of proxy
measures, failure in any one of the identified measures could provide
an indication on the overall risk to security. However, developing a
plan that includes objectives, milestones, and time frames to develop
outcome-based performance measures could better position TSA in
assessing the effectiveness of the SPOT program.
With regard to more readily quantifiable output performance measures,
such as the number of referrals by BDOs, or the ratio of arrests to
referrals, TSA was limited in its ability to analyze the data related
to these measures. The SPOT database includes information on all
passengers referred by BDOs for additional SPOT screening including
the behaviors of the passengers that led to the additional screening,
as well as the resolution of the screening process (e.g., no further
action taken, law enforcement notification, law enforcement
investigation, arrested, and reason for arrest). However, TSA reported
that any analysis of the data had to be done manually.[Footnote 89]
In March 2010, TSA migrated the SPOT referral data to its Performance
Management Information System, allowing for more statistical and other
analyses. According to TSA, migrating the SPOT referral database will
enhance the SPOT program's analytic capabilities. For example, TSA
stated that it would be able to conduct trend analyses, better
segregate data, and create specific reports for certain data. This
includes better tracking of performance data at specific airports,
analyzing by categories of airports (threat or geographic location),
and tracking the performance data of individual BDOs, such as number
of referrals, number of arrests, arrest to referral ratios, and other
analyses. However, since these changes to the database were not
complete at the time of our audit, we could not assess whether the
problems we identified with the database had been corrected.
Over 4 Years, SPOT Resulted in About 1,100 Arrests Out of Almost
14,000 Referrals to Law Enforcement:
The SPOT referral database records the total number of SPOT referrals
since May 29, 2004, how many were resolved, how many passengers BDOs
referred to LEOs, the recorded reasons for the referral, and how many
referrals led to arrests, among other things. As shown in figure 4, we
analyzed the SPOT referral data for the period May 29, 2004, to August
31, 2008.
Figure 4: Passenger Boardings at SPOT Airports, May 29, 2004, through
August 31, 2008:
[Refer to PDF for image: illustration]
* Approximately 2 billion passengers boarded aircraft at SPOT airports:
* Approximately 152,000 SPOT secondary referrals:
* Approximately 14,000 passengers referred to LEOs:
* Approximately 1,100 SPOT related arrests:
Note: Figure 4 is not drawn to scale.
[End of figure]
Figure 4 shows that approximately 2 billion passengers boarded
aircraft at SPOT airports from May 29, 2004, through August 31, 2008.
[Footnote 90] Of these, 151,943 (less than 1/100TH of 1 percent) were
sent to SPOT referral screening, and of these, 14,104 (9.3 percent)
were then referred to LEOs. These LEO referrals resulted in 1,083
arrests, or 7.6 percent of those referred, and less than 1 percent of
all SPOT referrals (0.7 percent of 151,943).
We also analyzed the reasons for arrests resulting from SPOT
referrals, for the May 29, 2004, through August 31, 2008, period.
Table 2 shows, in descending order, the reasons for the arrests.
Table 2: Reasons for Arrests from SPOT Referrals, May 29, 2004 through
August 31, 2008:
Reasons for arrest: Illegal alien;
Number: 427;
Percentage: 39%.
Reasons for arrest: Outstanding warrants;
Number: 209;
Percentage: 19%.
Reasons for arrest: Possession of fraudulent documents;
Number: 166;
Percentage: 15%.
Reasons for arrest: Other;
Number: 128;
Percentage: 12%.
Reasons for arrest: Possession of suspected drugs;
Number: 125;
Percentage: 12%.
Reasons for arrest: No reason given;
Number: 16;
Percentage: 1%.
Reasons for arrest: Undeclared currency;
Number: 8;
Percentage: 1%.
Reasons for arrest: Suspect documents;
Number: 4;
Percentage: 0.
Reasons for arrest: Total;
Number: 1,083;
Percentage: 99%[A].
Source: TSA, SPOT referral database from period of May 29, 2004,
through August 31, 2008.
[A] Total does not add to 100 percent due to rounding.
[End of table]
While SPOT personnel did not determine a specific reason for arrest
for 128 cases categorized as "other" or 16 other cases categorized as
"no reason given," our analysis of the SPOT database found that a
specific reason for arrest could have been determined for these cases
by using the LEO resolution notes included in the database. For
example, we identified 43 additional arrests related to fraudulent
documents, illegal aliens, and suspect documents, among others. The
remaining 101 arrests originally characterized as "other" or "no
reason given" included arrests for reasons such as intoxication,
unruly behavior, theft, domestic violence, and possession of
prohibited items. Many of the arrests resulting from BDO referrals
would typically fall under the jurisdiction of various local, state,
and federal agencies and are not directly related to threats to
aviation security. For example, the 427 individuals arrested as
illegal aliens, and the 166 arrested for possession of fraudulent
documents, are subject to the enforcement responsibilities shared by
U.S. Immigration and Customs Enforcement (ICE) and CBP. Although
outstanding warrants and the possession of fraudulent or suspect
documents could be associated with a terrorist threat, TSA officials
did not identify any direct links to terrorism or any threat to the
aviation system in any of these cases.
According to TSA, anecdotal examples of BDO actions at airports show
the value added by SPOT to securing the aviation system. However,
because the SPOT program has not been scientifically validated, it
cannot be determined if the anecdotal results cited by TSA were better
than if passengers had been pulled aside at random, rather than as a
consequence of being identified for further screening by BDOs. Some of
the incidents cited by TSA include the following.
* A BDO referred two passengers who were traveling together to
referral screening due to suspicious behavior. During secondary
screening, one passenger presented fraudulent travel documents. The
other could not produce any documentation of his citizenship and it
was determined he was in the United States illegally. ICE responded
and interviewed both passengers. ICE stated one passenger was also in
possession of $10,000 dollars which alarmed positive for narcotics
when swept by a K-9 team. ICE arrested one passenger on a federal
charge of possession of fraudulent identification documents and entry
without inspection. ICE stated charges are still pending for the
possession of $10,000. The second passenger was charged with a federal
charge of entry without inspection.
* A BDO referred a passenger to referral screening for exhibiting
suspicious behavior. Port Authority of Portland (Oregon) Police
responded and interviewed the passenger who did not give a statement.
LEOs conducted an NCIC check which revealed that there was an
outstanding warrant for the failure to appear for a theft charge. LEOs
arrested the passenger on a state charge for an outstanding warrant
for the failure to appear for theft.
* A BDO referred a passenger for referral screening due to suspicious
behavior. During the referral, the passenger admitted that he was
unlawfully present in the United States. The Orlando (Florida) Police
Department and CBP responded and interviewed the passenger who stated
he had $100,000 in his checked baggage, which was confirmed by CBP.
The passenger was arrested on a federal charge of illegal entry.
Because these are anecdotal examples, they cannot be used to reliably
generalize about the SPOT program's overall effectiveness or success
rate. Our analysis of the SPOT referral database found that the
referral data do not indicate if any of the passengers sent to
referral screening, or those arrested by LEOs after being referred to
them, intended to harm the aircraft, its passengers, or other
components of the aviation system. Additionally, SPOT officials told
us that it is not known if the SPOT program has ever resulted in the
arrest of anyone who is a terrorist, or who was planning to engage in
terrorist-related activity.
Reviewing Airport Video Recordings of Individuals Later Arrested or
Who Pleaded Guilty for Engaging in Terrorism-Related Activities Could
Help TSA Better Identify Terrorist-Related Behaviors:
Studying airport video recordings of the behaviors exhibited by
persons waiting in line and moving through airport checkpoints and who
were later charged with or pleaded guilty to terrorism-related
offenses could provide insights about behaviors that may be common
among terrorists or could demonstrate that terrorists do not generally
display any identifying behaviors. TSA officials agreed that examining
video recordings of individuals who were later charged with or pleaded
guilty to terrorism-related offenses, as they used the aviation system
to travel to overseas locations allegedly to receive terrorist
training or to execute attacks, may help inform the SPOT program's
identification of behavioral indicators. In addition, such images
could help determine if BDOs are looking for the right behaviors or
seeing the behaviors they have been trained to observe.
Using CBP and Department of Justice information, we examined the
travel of key individuals allegedly involved in six terrorist plots
that have been uncovered by law enforcement agencies.[Footnote 91] We
determined that at least 16 of the individuals allegedly involved in
these plots moved through 8 different airports where the SPOT program
had been implemented.[Footnote 92] Six of the 8 airports were among
the 10 highest risk airports, as rated by TSA in its Current Airport
Threat Assessment. In total, these individuals moved through SPOT
airports on at least 23 different occasions. For example, according to
Department of Justice documents, in December 2007 an individual who
later pleaded guilty to providing material support to Somali
terrorists boarded a plane at the Minneapolis-Saint Paul International
Airport en route to Somalia to join terrorists there and engage in
jihad. Similarly, in August 2008 an individual who later pleaded
guilty to providing material support to Al-Qaeda boarded a plane at
Newark Liberty International Airport en route to Pakistan to receive
terrorist training to support his efforts to attack the New York
subway system.
Our survey of Federal Security Directors at 161 SPOT airports
indicated most checkpoints at SPOT airports have surveillance cameras
installed. As we previously reported, best practices for project
management call for conducting feasibility studies to assess issues
related to technical and economic feasibility, among other things.
[Footnote 93] In addition, Standards for Internal Control state that
effectively using available resources is one element of functioning
internal controls.[Footnote 94] TSA may be able to utilize the
installed video infrastructure at the nation's airports to study the
behavior of persons who were later charged with or pleaded guilty to
terrorism-related offenses, and determine whether BDOs saw the
behaviors. The Director of Special Operations in TSA's Office of
Inspection told us that video recordings could be used as a teaching
tool to show the BDOs which behaviors or activities they did or did
not observe. In addition, TSA indicated that although the airports may
have cameras at the security screening checkpoints, the cameras are
not owned by TSA, and in many cases, they are not accessible to TSA.
However, TSA officials lack information on the scope of these
potential limitations because prior to our work TSA did not have
information on the number of checkpoints equipped with video
surveillance. We obtained this information as part of our survey of
Federal Security Directors at SPOT airports. While TSA officials noted
several possible limitations of the use of the existing video
surveillance equipment, these images provide TSA a means of acquiring
information about terrorist behaviors in the checkpoint environment
that is not available elsewhere. If current research determines that
the SPOT program has a scientifically validated basis for using
behavior detection for counterterrorism purposes in the airport
environment, then conducting a study to determine the feasibility of
using images captured by video cameras could better position TSA in
identifying behaviors to observe.
Standardization Teams Assess BDO Proficiency in SPOT Activities and
Provide Guidance and Mentoring to BDOs:
TSA sends standardization teams to SPOT airports on a periodic basis
to conduct activities related to quality control. Teams observe SPOT
operations at an airport for several days, working side by side with
the BDOs, on multiple shifts, observing their performance, offering
guidance, and providing training when required. According to TSA, the
purpose of a standardization team visit is to provide operational
support to the BDOs, which includes additional training, mentoring,
and guidance to help maintain a successful SPOT program.
The standardization teams are comprised of at least two G-Band, or
Expert[Footnote 95] BDOs who have received an additional week of
training on SPOT behaviors and mentoring skills. SPOT officials stated
that the SPOT program uses its standardization teams to assess overall
BDO proficiency by observing BDOs, reviewing SPOT score sheet data,
and other relevant data. Standardization teams may also provide a
Behavior Observation and Analysis review class to refresh BDOs if the
team determines that such a class is needed. The SPOT program director
also said that the standardization teams aim to monitor the airport's
compliance with the SPOT program's Standard Operating Procedures. As
part of this mentoring approach, the standardization teams provide
individual and team guidance to the BDOs, offer assistance in program
management, and cover issues related to the interaction of BDOs with
other TSA checkpoint personnel.
TSA reported to us that it does not systematically track the
standardization teams' recommendations or the frequency of the teams'
airport visits. Standards for Internal Control state that programs
should have controls in place to assess the quality of performance
over time and ensure that the findings of audits and other reviews are
promptly resolved. Managers are to (1) promptly evaluate findings from
audits and other reviews, including those showing deficiencies and
recommendations reported by auditors and others who evaluate agencies'
operations; (2) determine proper actions in response to findings and
recommendations from audits and reviews; and (3) complete, within
established time frames, all actions that correct or otherwise resolve
the matters brought to management's attention.[Footnote 96] Although
the standardization teams may provide an airport Federal Security
Director with recommendations on how to improve SPOT operations, the
SPOT program director stated that Federal Security Directors are not
required to document whether they have implemented the team
recommendations. TSA officials told us that standardization teams can
follow up on recommendations made during previous visits. However, TSA
did not track whether corrective actions were implemented or the
frequency of the team's airport visits to ensure the implementation of
the airport's SPOT program. TSA officials stated that they are
currently examining ways to compile data to address this issue, and
expect to have a system in place in fiscal year 2010.
TSA Developed and Deployed SPOT Training but Further Action Could
Enhance Its Effectiveness:
Although TSA has taken steps to incorporate all four elements of an
effective training program by planning, designing, implementing, and
evaluating training for BDOs, further action could help enhance the
training's effectiveness. TSA initially consulted outside experts for
help in the training's development, which began as a half-day course
and has grown to include classroom, on-the-job, and advanced training.
TSA also has efforts underway to improve its training program, such as
the deployment of SPOT recurrent training. However, TSA evaluations of
SPOT program instructors found mixed quality among them, from 2006
onwards. Additionally, TSA has ongoing plans to evaluate the SPOT
training for effectiveness, but has not yet developed time frames and
milestones for completing the evaluation.
TSA Has Taken Actions to Develop and Deploy SPOT Training:
TSA's SPOT Training Evolved Over Time:
In 2003, TSA officials at Boston Logan International airport developed
the initial half-day training course for SPOT based on an existing
course developed for the Massachusetts State Police. Their goal was to
take the behavior detection program designed for law enforcement and
apply it to screeners at airport checkpoints. According to TSA
officials at Boston Logan, after they recognized that the lecture-
style course they originally designed was not effective, they tasked
an instructional system designer from TSA's Workplace Performance and
Training (the former name of TSA's Operational and Technical Training
Division)[Footnote 97] and an industrial psychologist from the Office
of Human Capital to redesign and expand the course, which was piloted
in 2005. The 2007 SPOT strategic plan included training objectives for
the SPOT program as follows:
* reviewing existing behavior observation training providers,
* establishing and prioritizing multimodal training and assistance
efforts based on threat assessments and critical infrastructure,
* establishing a Center of Excellence for Behavior Detection Program
training that would continually enhance the quantity and quality of
training to selected candidates, and:
* developing a recurrent training program designed to refresh and hone
skills needed for an effective Behavior Detection Program.
Since that time, the SPOT program implemented, or is in the process of
implementing, some of these objectives. For example, in 2008, as part
of its effort towards establishing a center for excellence in behavior
detection training (third objective), the SPOT program participated in
a meeting with behavior detection training officials from various DHS
components facilitated by DHS's Screening Coordination Office to
promote the sharing of information about behavior detection training
and foster future collaboration. Additionally, the SPOT program worked
with TSA's Operational and Technical Training Division to create a
recurrent training component for BDOs (fourth objective). For example,
in 2008, the SPOT program office added a course on detecting
microfacial expressions called Additional Behavior Detection
Techniques.[Footnote 98] This 3-day course builds on the behavior
detection skills taught in basic training, by teaching BDOs how to
detect microfacial expressions. After pilot testing, the course began
implementation nationwide in January 2009.
TSA Consulted with Some Experts on Developing SPOT Training:
In developing an effective training program, we previously reported
that consultation with subject matter experts and expert entities is a
core characteristic of the strategic training and development process.
[Footnote 99] TSA SPOT program staff told us that they consulted with
experts on behavior detection and observed existing behavior detection
courses before deploying the SPOT training program. According to SPOT
program officials, a TSA staff member from Boston Logan International
Airport attended other training programs offered by other federal
agencies and private training organizations to inform the design of
SPOT training.[Footnote 100] TSA officials told us that information
from the training courses was used to help develop the list of
behaviors or "stress elevators" for the program, and that the point
system used to identify passengers for referral screening was based in
part on consultations with several subject-matter experts.
TSA documentation also notes that a SPOT working group created in
February 2004 consulted with the FBI's Behavioral Science Unit.
[Footnote 101] The Behavioral Science Unit specializes in developing
and facilitating training, research, and consultation in the
behavioral sciences for the FBI, law enforcement, intelligence, and
military communities. While TSA officials from Boston Logan told us
that the FBI was included in this initial SPOT working group, these
officials agree that coordination with the FBI lapsed until June 2009
when the SPOT Program Office reengaged with the Behavioral Science
Unit, and held a meeting with the unit at the FBI Academy in Quantico,
Virginia. Since that meeting, a subject matter expert from the SPOT
Program Office has been invited to be a member of the Terrorism
Research and Analysis Project, which is an ongoing working group
sponsored by the unit.
In July 2008, DHS's Screening Coordination Office facilitated a
collaborative discussion on behavior detection that included TSA, CBP,
and Secret Service officials to better ensure that components within
DHS share information regarding their efforts in behavior detection
and provide a forum for components to have an informed and
collaborative discussion on current capabilities, best practices, and
lessons learned. According to TSA, no further contact has occurred
between the DHS Behavior Detection Working Group and the SPOT program.
Thus, the extent to which the working group's expertise will be used
to refine or augment SPOT training in the future is not yet clear.
SPOT Program Office Recently Deployed Recurrent Training:
Along with basic and remedial training required by the Aviation and
Transportation Security Act, TSA policy requires its screening force
to regularly complete recurrent (refresher) training. TSA recognized
that ongoing training of screeners on a frequent basis and effective
supervisory training are critical to maintaining and enhancing skills
learned during basic training. According to agency officials, TSA is
currently working with DHS S&T to determine the necessary frequency
for refresher training for each training course within the SPOT
program. Furthermore, TSA plans to place BDOs under TSA's Performance
and Accountability Standards System (PASS) beginning in fiscal year
2010. This will include a recertification module.
In 2008, the SPOT program office began the process for developing
recurrent SPOT training. Our internal control standards and training
assessment guidance suggest that such refresher training should be
considered integral to an effective training program from the start
because work conditions and environments can be expected to change
over time, and additional or updated training is essential to ensuring
that the program mission continues to be accomplished.[Footnote 102]
According to the SPOT program office, the recently deployed recurrent
training will be semiannual. TSA's Operational and Technical Training
Division initially planned to pilot test recurrent training in April
2009 followed by full implementation of the course in approximately
May 2009. Because the Operational and Technical Training Division
focus was shifted to completing the revisions for the SPOT basic
certification course, recurrent training was delayed until September
2009 when they released the training on TSA's Online Learning Center.
Instructor Evaluations Found Mixed Quality; Issues with Program
Management Led to Instructor Retraining:
Our previous work on elements of effective training states that
instructors must be both knowledgeable about the subject matter and
issues involved, as well as able to effectively transfer these skills
and knowledge to others.[Footnote 103] Moreover, internal control
standards state that all personnel need to possess and maintain a
level of competence that allows them to accomplish their assigned
duties.[Footnote 104] Management needs to identify appropriate
knowledge and skills needed for various jobs and provide needed
training, as well as to ensure that those teaching the skills are
themselves competent.
TSA conducted internal assessments of SPOT instructors episodically
from 2006 through March 2008. These assessments involved a few
instructors being rated at a time, and found a wide range of
competency among the instructors. In January 2009, TSA's Office of
Inspections and Investigations began an investigation of the SPOT
training manager, who resigned shortly thereafter. TSA investigators
determined that the training manager and other trainers had created a
hostile training environment that intimidated some trainees. To
address this problem, TSA stated that the program office reexamined
the SPOT training program nationally. This included recertifying 47 of
54 SPOT instructors in March 2009, which included evaluation by TSA's
Office of Human Capital, Quality Assurance assessors. Additionally, in
July 2009, TSA centralized SPOT training at five permanent, regional
training facilities in Orlando, Florida; Houston, Texas; Phoenix,
Arizona; Denver, Colorado; and Philadelphia, Pennsylvania.[Footnote
105] According to the SPOT program director, this will allow the SPOT
program office more oversight over training. Previously, training was
provided at individual airports.
After the March 2009 recertification training, ratings scores of SPOT
instructors showed less variation than did previous ratings. We
reviewed the quality assurance instructor evaluations of two SPOT
instructors conducted by TSA's Office of Human Capital, Training
Standards and Evaluation Branch, and the 167 SPOT program instructor
evaluations of 54 SPOT instructors conducted by the SPOT program
office and TSA's Operational and Technical Training Division since the
program started in October 2006.[Footnote 106] After the
recertification training, 93 percent of instructors were rated as
exceeding expectations, compared to 30 percent in the 2006 to
September 2008 ratings. Table 3 shows the ratings of instructors for
March 2009 compared to the period of 2006 to September 2008.[Footnote
107]
Table 3: SPOT Instructor Evaluation Ratings, 2006 to September 2008,
and March 2009:
2006 - Sept 2008;
Number of instructor evaluations: 73;
Unsatisfactory (0-74%): Number: 3;
Unsatisfactory (0-74%): Percent: 4%;
Needs improvement (75-84%): Number: 5;
Needs improvement (75-84%): Percent: 7%;
Meets expectations (85-94%): Number: 36;
Meets expectations (85-94%): Percent: 49%;
Exceeds expectations (95-100%): Number: 22;
Exceeds expectations (95-100%): Percent: 30%;
No numeric score given: Number: 7;
No numeric score given: Percent: 10%.
March 2009;
Number of instructor evaluations: 94;
Unsatisfactory (0-74%): Number: 0;
Unsatisfactory (0-74%): Percent: 0%;
Needs improvement (75-84%): Number: 1;
Needs improvement (75-84%): Percent: 1%;
Meets expectations (85-94%): Number: 6;
Meets expectations (85-94%): Percent: 6%;
Exceeds expectations (95-100%): Number: 87;
Exceeds expectations (95-100%): Percent: 93%;
No numeric score given: Number: 0;
No numeric score given: Percent: 0%.
Source: GAO analysis of TSA Quality Assurance Instructor Evaluations
for SPOT.
[End of table]
In addition to the variation in numeric scores and rating levels for
the 2006 to September 2008 period, as shown in table 3, we found
substantial variation in the comments about instructor competency for
the same period. For example, in 32 out of 74 instructor evaluation
forms that we reviewed where comments were made about the instructor
prior to 2009, the comments ranged from superb to needs more
experience as an instructor, as well as needs more time performing the
job as a BDO to be able to teach others. In the comments on an
instructor who was rated as "meets expectations," the instructor was
described as having "limited experience within the SPOT program," that
this was "a major concern," and it was recommended that the instructor
spend as much time as possible functioning as a BDO. In other cases,
however, SPOT instructors were described as competent, solid, and
outstanding. For example, one instructor who received a rating of
"exceeds expectations" was described as a superb instructor who "is a
valued member of the National Training Team." As noted above,
following the March 2009 recertification training, 93 percent of the
instructors received a rating of "exceeds expectations" with only 1
percent "needing improvement." Of the 94 instructor evaluations
completed in March 2009, 82 contained written comments. Of these,
multiple SPOT instructors were described as excellent, knowledgeable,
and effective. For example, an instructor who received a rating of
"exceeds expectations" was noted as demonstrating a high degree of
material knowledge and great presentation skills. TSA attributed the
increase in instructor ratings to two factors. The first is low
turnover among SPOT instructors, which allows instructors to hone both
their technical and instructor skills. The second factor cited by TSA
is that TSA conducted a 2-day instructor refresher training
immediately prior to the evaluations in March 2009. To ensure all
instructors were reevaluated within a specific time frame, evaluations
were scheduled and conducted in a controlled environment. Instructors
knew in advance they were going to be evaluated and delivered modules
of the BDO certification course to other BDO instructors.
TSA Has Taken Some Action, but Has Not Evaluated the SPOT Training
Program for Effectiveness:
We previously reported that evaluation is an integral part of training
and development efforts, and that agencies need to systematically plan
for and evaluate the effectiveness of training and development.
[Footnote 108] Employing systematic monitoring and feedback processes
can help by catching potential problems at an early stage, thereby
saving valuable time and resources that a major redesign of training
would likely entail. Similarly, in 2006, TSA's Operational and
Technical Training Division issued general evaluation standards for
training programs, stating that training programs should be
comprehensively evaluated on a periodic basis to identify program
strengths and weaknesses.[Footnote 109] Moreover, standard practices
in program management for defining, designing, and executing programs
include developing a road map, or program plan, to establish an order
for executing specific projects needed to obtain defined programmatic
results within a specified time frame.[Footnote 110]
The former SPOT training manager told us that the SPOT program
internally evaluates the effectiveness of SPOT training through the
job knowledge tests that BDO candidates must pass following the
classroom portion of the training and the SPOT Proficiency/On-the-Job
Training Checklist following the on-the-job portion of the training.
Furthermore, the former training manager told us that TSA knows that
the SPOT training is effective because BDOs are able to recognize
behaviors at the checkpoint, and because of BDOs' demonstrated ability
to identify criminals--such as drug couriers or people with
outstanding arrest warrants--through the screening process.
Although TSA has not conducted a comprehensive analysis of the
effectiveness of the SPOT training program, TSA's Office of Human
Capital, Training Standards and Evaluation Branch conducted training
evaluations to assess how students use what they were taught in the
SPOT basic training course. Specifically, from July through September
2008, the Training Standards and Evaluation Branch conducted
evaluations at 5 of the 161 airports where the SPOT program is
currently operating. Based on BDO feedback at the 5 airports, the
Training Standards and Evaluation Branch's final report contained a
series of recommendations for improving the SPOT training program.
These recommendations and TSA's actions to address them are summarized
in table 4.
Table 4: TSA Training Standards and Evaluation Branch Recommendations
for Improving SPOT Training and TSA Actions on the Recommendations:
Training Standards and Evaluation Branch recommendations: Ensure
training instructors adhere to a set of professional guidelines;
TSA action on recommendations: TSA sent 47 TSA Approved Instructors
for the SPOT program to recertification training in March 2009.
Training Standards and Evaluation Branch recommendations: Add local
policies and procedure as an addendum to the (SPOT) Training;
TSA action on recommendations: No action[A].
Training Standards and Evaluation Branch recommendations: Include more
role-playing and scenarios in the classroom training so all trainees
can practice casual conversation skills;
TSA action on recommendations: TSA added more role-playing scenarios
to their basic SPOT training.
Training Standards and Evaluation Branch recommendations: Develop
recurrent training that can be placed on the TSA Online Learning
Center;
TSA action on recommendations: TSA developed and deployed recurrent
training on the TSA Online Learning Center in September 2009.
Training Standards and Evaluation Branch recommendations: Develop
templates for writing reports;
TSA action on recommendations: TSA added an Incident Report Writing
course to the TSA Online Learning Center. Additionally, TSA has
developed templates for Incident Reports and After Action Reports. TSA
has also developed Online Learning Center training for completing SPOT
Referral Reports.
Training Standards and Evaluation Branch recommendations: Provide more
real world videos;
TSA action on recommendations: TSA revised the SPOT training videos in
late 2008.
Training Standards and Evaluation Branch recommendations: Provide
recurrent training of behaviors through online videos;
TSA action on recommendations: The video scenarios for recurrent
training will be available in the second quarter of fiscal year 2010.
Training Standards and Evaluation Branch recommendations: Add parts of
the Bomb Appraisal Officer task into the training;
TSA action on recommendations: No action[A].
Training Standards and Evaluation Branch recommendations: Provide
recurrent training outside of TSA (more Immigration and Customs
Enforcement, DEA, and CBP training);
TSA action on recommendations: No action[A].
Training Standards and Evaluation Branch recommendations: Have BDOs
spend more time with an On-the-Job-Training mentor;
TSA action on recommendations: No action[A].
Training Standards and Evaluation Branch recommendations: Validate the
training for course content and On-the-Job-Training;
TSA action on recommendations: In 2009, in coordination with DHS S&T,
TSA began the scientific analysis of the BDO position to empirically
derive and validate the knowledge, skills, and attributes that it
requires. The analysis is projected to be completed in fiscal year
2010.
Training Standards and Evaluation Branch recommendations: Clarify
SPOT's "Walk-the-Line" policy and communicate it to all BDO personnel;
TSA action on recommendations: TSA issued revised SPOT Standard
Operating Procedures to all BDOs in January 2009.
Source: TSA, Training Standards and Evaluation Branch, Office of Human
Capital, Memorandum For Operational and Technical Training, and
Behavior Detection and Document Validation Branch, Office of Security
Operations on Training Transfer (L3) of SPOT Training, October 30,
2008.
[A] According to TSA, the SPOT program office will determine if the
recommended action is appropriate after the BDO job task analysis and
training task analysis are completed.
[End of table]
Additionally, in conjunction with S&T, TSA conducted a training
effectiveness evaluation on the Additional Behavior Detection
Techniques course, which showed a statistically significant increase
in knowledge and skills following completion of the course.
S&T is currently conducting a BDO job task analysis, which may be used
to evaluate and update the SPOT training curriculum. Following the
completion of the job task analysis--anticipated in mid-May 2010--
TSA's Operational and Technical Training Division intends to conduct
an in-depth training gap analysis,[Footnote 111] which will take
approximately 2 months to complete. Following completion of the
training gap analysis, the agency will develop project plans,
including milestones for future development efforts, to address any
training concerns. However, to date, the agency does not have an
evaluation plan including time frames and milestones for completion.
According to the Operational and Technical Training Division, TSA will
conduct periodic evaluations as the BDO position evolves. By
conducting a comprehensive evaluation of the effectiveness of its
training program, TSA will be in a better position to determine if
BDOs are being taught the knowledge and skills they need to perform
their job. Furthermore, by developing milestones and time frames for
conducting such evaluations systematically, as well as on a periodic
basis, TSA could help ensure that the SPOT training program is
evaluated in accordance with its directives to help ensure that the
program continues to provide BDOs with the necessary tools required to
carry out their responsibilities.
Conclusions:
TSA developed the SPOT program in the wake of September 11, 2001, in
an effort to respond quickly to potential threats to aviation security
by identifying individuals who may pose a threat to aviation security,
including terrorists planning or executing an attack who were not
likely to be identified by TSA's other screening security measures.
Because TSA did not ensure that SPOT's underlying methodology and work
methods were scientifically validated prior to its nationwide
deployment, an independent panel of experts could help determine
whether a scientific foundation exists for the way in which the SPOT
program uses behavior detection analysis for counterterrorism purposes
in the aviation environment.
With approximately $5.2 billion devoted to screening passengers and
their property in fiscal year 2009, it is important that TSA provides
effective stewardship of taxpayer funds ensuring a return on
investment for each layer of its security system. As one layer of
aviation security, the SPOT program has an estimated projected cost of
about $1.2 billion over the next 5 years if the administration's
requested funding of $232 million for fiscal year 2011 remains at this
level.[Footnote 112] The nation's constrained fiscal environment makes
it imperative that careful choices be made regarding which investments
to pursue and which to discontinue. If an independent expert panel
determines that DHS's study is sufficiently comprehensive to determine
whether the SPOT program is based on valid scientific principles that
can be effectively applied in an airport environment for
counterterrorism purposes, then conducting a comprehensive risk
assessment including threat, vulnerability, and consequence could
strengthen TSA's ability in making resource allocation decisions and
prioritizing its risk mitigation efforts. Moreover, conducting a cost-
benefit analysis could help TSA determine whether SPOT provides
benefits greater than or equal to other security alternatives and
whether its level of investment in the SPOT program is appropriate.
Revising its strategic plan for SPOT to incorporate risk assessment
information, cost and resource analysis, and other essential
components could enhance the plan's usefulness to TSA in making
program management and resource allocation decisions to effectively
manage the deployment of SPOT.
Providing guidance on how to use TSA's resources for running passenger
names against intelligence and criminal databases available to the
Transportation Security Operations Center and helping DHS to connect
disparate pieces of information using the Transportation Information
Sharing System and other related intelligence and crime database and
data sources could better inform DHS and TSA regarding the identity
and background of certain individuals and thereby enhance aviation
security. In addition, implementing the steps called for in the TSA
Office of Strategic Operations plan to provide BDOs with a real-time
mechanism to verify passenger identities and backgrounds via TSA's
Transportation Security Operations Center could strengthen their
ability to rapidly verify the identity and background of passengers
who have caused concern, and increase the likelihood of detecting and
disrupting potential terrorists intending to cause harm to the
aviation system. Additionally, developing outcome-oriented performance
measures, making improvements to the SPOT database, and studying the
feasibility of utilizing video recordings of individuals as they
transited checkpoints and who were later charged with or pleaded
guilty to terrorism-related offenses, could help TSA evaluate the SPOT
program, identify potential vulnerabilities, and assess the
effectiveness of its BDOs. Further, developing a plan for systematic
and periodic evaluation of the training provided to BDOs along with
time frames and milestones for its completion could help ensure that
the SPOT training program is evaluated in accordance with its
directives to help ensure that the program continues to provide BDOs
with the necessary tools required to carry out their responsibilities.
Recommendations for Executive Action:
To help ensure that SPOT is based on valid scientific principles that
can be effectively applied in an airport environment, we recommend
that the Secretary of Homeland Security convene an independent panel
of experts to review the methodology of the DHS S&T Directorate study
on the SPOT program to determine whether the study's methodology is
sufficiently comprehensive to validate the SPOT program. This
assessment should include appropriate input from other federal
agencies with expertise in behavior detection and relevant subject
matter experts.
If this research determines that the SPOT program has a scientifically
validated basis for using behavior detection for counterterrorism
purposes in the airport environment, then we recommend that the TSA
Administrator take the following four actions:
* Conduct a comprehensive risk assessment to include threat,
vulnerability, and consequence of airports nationwide to determine the
effective deployment of SPOT if TSA's ongoing Aviation Modal Risk
Assessment lacks this information.
* Perform a cost-benefit analysis of the SPOT program, including a
comparison of the SPOT program with other security screening programs,
such as random screening, or already existing security measures.
* Revise and implement the SPOT strategic plan by incorporating risk
assessment information, identifying cost and resources, linking it to
other related TSA strategic documents, describing how SPOT is
integrated and implemented with TSA's other layers of aviation
security, and providing guidance on how to effectively link the roles,
responsibilities, and capabilities of federal, state, and local
officials providing program support.
* Study the feasibility of using airport checkpoint-surveillance video
recordings of individuals transiting checkpoints who were later
charged with or pleaded guilty to terrorism-related offenses to
enhance understanding of terrorist behaviors in the airport checkpoint
environment.
Concurrent with the DHS S&T Directorate study of SPOT, and an
independent panel assessment of the soundness of the methodology of
the S&T study, we recommend that the TSA Administrator take the
following six actions to ensure the program's effective implementation:
* To provide additional assurance that TSA utilizes available
resources to support the goals of deterring, detecting, and preventing
security threats to the aviation system, TSA should:
- Provide guidance in the SPOT Standard Operating Procedures or other
TSA directive to BDOs, or other TSA personnel, on inputting data into
the Transportation Information Sharing System and set milestones and a
time frame for deploying Transportation Information Sharing System
access to SPOT airports so that TSA and intelligence community
entities have information from all SPOT LEO referrals readily
available to assist in "connecting the dots" and identifying potential
terror plots.
- Implement the steps called for in the TSA Office of Security
Operations Business plan to develop a standardized process for
allowing BDOs or other designated airport officials to send
information to TSA's Transportation Security Operations Center about
passengers whose behavior indicates that they may pose a threat to
security, and provide guidance on how designated TSA officials are to
receive information back from the Transportation Security Operations
Center.
- Direct the TSA Transportation Security Operations Center to utilize
all of the law enforcement and intelligence databases available to it
when running passenger names, for passengers who have risen to the
level of a LEO referral.
* To better measure the effectiveness of the program and evaluate the
performance of BDOs, TSA should:
- Establish a plan that includes objectives, milestones, and time
frames to develop outcome-oriented performance measures to help refine
the current methods used by Behavior Detection Officers for
identifying individuals who may pose a risk to the aviation system.
- Establish controls to help ensure completeness, accuracy,
authorization, and validity of data collected during SPOT screening.
* To help ensure that TSA provides BDOs with the knowledge and skills
needed to perform their duties, TSA should:
- Establish time frames and milestones for its plan to systematically
conduct evaluations of the SPOT training program on a periodic basis.
Agency Comments and Our Evaluation:
We provided a draft of our report to DHS and TSA on March 19, 2010,
for review and comment. On May 3, 2010, DHS provided written comments,
which are reprinted in appendix II. In commenting on our report, DHS
stated that it concurred with 10 of our recommendations and identified
actions taken, planned, or under way to implement them. However, the
actions DHS reported it plans to take and has underway do not fully
address the intent of our first recommendation. DHS also concurred in
principle with an eleventh recommendation stating that it had convened
a working group to determine the feasibility of implementing it. DHS
commented on the scientific basis underlying SPOT and on two
statements in our report that it believed were inaccurate--
specifically, DHS disagreed with our reliance on a 2008 National
Research Council report published under the auspices of the National
Academy of Sciences on issues related to behavior detection, and
second, on issues related to unpublished research they had cited as a
partial validation of some aspects of the SPOT program.[Footnote 113]
Finally, DHS commented on our conclusion regarding the use of the SPOT
referral data.
Regarding our first recommendation that DHS convene an independent
panel of experts to review the methodology of DHS's Science and
Technology Directorate (S&T) study on SPOT, and to include appropriate
input from other federal agencies with relevant expertise, DHS
concurred and stated the current process includes an independent
review of the program that will include input from other federal
agencies and relevant experts. Although DHS has contracted with the
American Institutes for Research to conduct its study, it remains
unclear who will oversee this review and whether they are sufficiently
independent from the current research process. DHS's response also
does not describe how the review currently planned is designed to
determine whether the study's methodology is sufficiently
comprehensive to validate the SPOT program. As we noted in our report,
research on other issues, such as determining the number of
individuals needed to observe a given number of passengers moving at a
given rate per day in an airport environment or the duration that such
observation can be conducted by BDOs before observation fatigue
affects effectiveness, could provide additional information on the
extent to which SPOT can be effectively implemented in airports. Dr.
Paul Ekman, a leading research scientist in the field of behavior
detection, told us that additional research could help determine the
need for periodic refresher training since no research has yet
determined whether behavior detection is easily forgotten or can be
potentially degraded with time or lack of use. Thus, questions exist
as to whether behavior detection principles can be reliably and
effectively used for counterterrorism purposes in airport settings to
identify individuals who may pose a risk to the aviation system. To
help ensure an objective assessment of the study's methodology and
findings, DHS could benefit from convening an independent panel of
experts from outside DHS to determine whether the study's methodology
is sufficiently comprehensive to validate the SPOT program.
DHS also concurred with our second recommendation to conduct a
comprehensive risk assessment to determine the effective deployment of
SPOT. DHS stated that TSA's Aviation Modal Risk Assessment is designed
to evaluate overall transportation security risk, not deployment
strategies. However, DHS noted that TSA is in the process of
conducting an initial risk analysis using its risk management analysis
tool and plans to update this analysis in the future. However, it is
not clear from DHS's comments how this analysis will incorporate an
assessment of TSA's deployment strategy for SPOT.
DHS also concurred with our third recommendation to perform a cost-
benefit analysis of SPOT. DHS noted that TSA is developing an initial
cost-benefit analysis and that the flexibility of behavior detection
officers already suggests that behavior detection is cost-effective.
However, it is not clear from DHS's comments whether its cost-benefit
analysis will include a comparison of the SPOT program with other
security screening programs, such as random screening, or already
existing security measures as we recommended. Completing its cost-
benefit analysis and comparing it to other screening programs should
help establish whether the SPOT program is cost-effective compared to
other layers of security.
With regard to our fourth recommendation to revise and implement the
SPOT strategic plan using risk assessment information, DHS concurred
and noted that analysis facilitated by the risk management analysis
tool will allow the program to revise the SPOT strategic plan to
incorporate the elements identified in our recommendation.
DHS also concurred with our fifth recommendation to study the
feasibility of using airport checkpoint-surveillance video recordings
to enhance its understanding of terrorist behaviors. DHS noted that
TSA agrees this could be a useful tool and is working with DHS's S&T
Directorate to utilize video case studies of terrorists, if possible.
These cases studies could help TSA determine what behaviors had been
demonstrated by these persons convicted of terrorist-related offenses
who went through SPOT airports, and what could be learned from the
observed behaviors.
DHS concurred with our sixth recommendation that TSA provide guidance
in the SPOT SOP or other directives to BDOs, or other TSA personnel,
on how to input data into the Transportation Information Sharing
System database. DHS stated that the SPOT SOP is undergoing revision,
and that the revised version will provide guidance directing the input
of BDO data into the Transportation Information Sharing System. DHS
anticipates release of the updated SPOT SOP in fiscal year 2010. DHS
also agreed that TSA should set milestones and a time frame for
deploying Transportation Information Sharing System access to SPOT
airports so that TSA and intelligence community entities have
information from all SPOT LEO referrals readily available to assist in
"connecting the dots" and identifying potential terror plots. DHS
stated that TSA is currently drafting a plan to include milestones and
a time frame for deploying System access to all SPOT airports.
DHS concurred with our seventh recommendation to develop a
standardized process to allow BDOs or other designated airport
officials to send information to TSA's Transportation Security
Operations Center about passengers whose behavior indicates they may
pose a threat to security, and to provide guidance on how designated
TSA officials are to receive information back from the Center. DHS
stated that TSA has convened a working group to address this
recommendation. Moreover, TSA is developing a system and procedure for
sending and receiving information from the Center and stated that it
anticipates having a system in place later in fiscal year 2010.
DHS concurred in principle with regard to our eighth recommendation
that the Transportation Security Operations Center utilize all of the
databases available to it when conducting checks on passengers who
rise to the level of a LEO referral against intelligence and criminal
databases. DHS stated that TSA has convened a working group to address
this recommendation. According to DHS, this group will conduct a study
during fiscal year 2010 to determine the feasibility of fully
implementing this recommendation. As such, the study is to review the
various authorities, permissions, and limitations of each of the
databases or systems cited in our report. DHS stated that access to
some of the systems, requires more justification than a BDO referral.
Further, according to DHS, because some of the databases or systems
contain classified information, TSA will also need to adopt a
communication strategy to transmit the passenger information between
the BDO and Transportation Security Operations Center. DHS stated that
TSA will work on a process to collect the passenger information,
verify the passenger's identity, through checks of databases, and
analyze that information to determine if the passenger is the subject
of an investigation and may pose a risk to aviation security.
With regard to our ninth recommendation to establish a plan with
objectives, milestones, and time frames to develop outcome-oriented
performance measures for BDOs, DHS concurred and stated that TSA
intends to consult with experts to develop outcome-oriented
performance measures.
DHS also concurred with our tenth recommendation to establish controls
for SPOT data. DHS noted that TSA established additional controls as
part of the SPOT database migration to TSA's Performance Management
Information System and is exploring an additional technology solution
to reduce possible errors. As noted in our report, since these changes
to the database were not complete at the time of our audit, we could
not assess whether the problems we identified with the database had
been corrected.
Regarding our eleventh recommendation to establish time frames and
milestones to systematically evaluate the SPOT training program on a
periodic basis, DHS concurred and stated that TSA intends to develop
such a plan following completion of DHS's S&T Directorate's BDO Job
Task Analysis, and TSA's training gap analysis, which identifies gaps
in the training curriculum.
DHS also commented on the scientific basis underlying SPOT.
Specifically, DHS stated that decades of scientific research has shown
the SPOT behaviors to be "universal in their manifestation." However,
according to DHS, its S&T Directorate is examining the extent to which
behavior indicators are appropriate for screening purposes and lead to
appropriate and correct security decisions. DHS also commented that
the results of this work, which is currently underway, will establish
a scientific basis of the extent to which the SPOT program instruments
and methods are valid. Thus, DHS's comments suggest that additional
research is needed to determine whether these behaviors can be used in
an airport environment for screening passengers to identify threats to
the aviation system.
Moreover, DHS took issue with our use of a report from the National
Research Council of the National Academy of Sciences stating that we
improperly relied upon this report.[Footnote 114] We disagree. DHS
questioned the findings of the National Research Council report and
stated that it lacked sufficient information for its conclusions
because it principally focused on privacy as it relates to data mining
and behavioral surveillance and was not intended to represent an
exhaustive or definitive review of the research or operational
literature on behavioral screening, including recent unpublished DHS,
defense, and intelligence community studies. DHS also stated that the
National Research Council report did not study the SPOT program and
that the researchers did not conduct interviews with SPOT personnel.
As we noted in our report, although the National Research Council
report addresses broader issues related to privacy and data mining, a
senior Council official--and one of the authors of the study--stated
that the committee included behavior detection as a focus because any
behavior detection program could have privacy implications. This
official added that the primary objective of the report was to develop
a framework for sound decision making for programs, such as SPOT, and
help ensure a sound scientific and legal basis. According to this
official, the National Academy of Sciences' Committee on Technical and
Privacy Dimensions of Information for Terrorism Prevention and Other
National Goals--which had oversight of the report--was briefed on the
SPOT program as part of the study. The Committee also conducted
meetings with three experts in behavior detection as part of their
research. During the course of our review, we interviewed three
Committee members responsible for developing the report's findings, as
well as four other behavior detection experts, including the three who
participated in the National Research Council study. Our discussions
with these experts corroborated the report's findings. Thus, we
believe that our use of the Council report was an appropriate and a
necessary part of our review.
However, the National Research Council report was only one of many
sources that we analyzed with regard to the science of behavioral and
physiological screening, and its applicability to an airport
environment. As we noted in the description of our methodology, our
study included interviews with officials from DHS as well as several
of its components and other U.S. government agencies--each of which
use elements of behavior detection in their daily work. We also
interviewed El Al airline officials, a former director of security at
Israel's Ben-Gurion airport, and seven nationally recognized experts
in behavior detection as part of our review. Moreover, as we explained
in the discussion of our scope and methodology, we conducted a survey
about the SPOT program of all 118 Federal Security Directors for all
SPOT airports, and conducted site visits to 15 SPOT airports. In
addition, we analyzed the SPOT referral database, to the extent the
data permitted, covering a 4-year period and the results from 2
billion passengers passing through SPOT airports. Moreover, we
attended both the basic and advanced training courses in behavior
detection provided by TSA to BDOs, in order to better understand how
the program is carried out. Therefore, our analysis of the program was
not derived from or based on a single study by the National Research
Council as DHS suggested, but rather is based on all of the
information we gathered and synthesized from multiple, diverse, expert
sources, each of which provided different perspectives about the
program, as well as about behavior detection in general.
DHS also disagreed with the accuracy of a statement included in our
report that noted DHS S&T could not provide us with specific contacts
related to sources of information for certain research it cited as
support for the SPOT program. In its comments, DHS stated that it had
provided us with all requested documents that represent DHS's S&T
Directorate-sponsored research. We agree. However, DHS did not provide
us with contact information for the sources of unpublished studies by
the Department of Defense and other intelligence community studies
that DHS S&T had cited as support for the SPOT program. Without such
information, we are unable to verify the contents of these unpublished
studies.
Finally, DHS stated that while we were unable to use the SPOT referral
data to assess whether any behavior or combination of SPOT behaviors
could be used to reliably predict the final outcome of an incident
involving the use of SPOT, it was able to analyze the SPOT referral
database successfully after working with TSA to verify scores assigned
to different indicators. Our concern with the data did not involve the
question of whether some behaviors were entered erroneously, nor
whether errors in coding were excessive or non-random. Rather, we were
concerned with whether the data on behaviors were complete.
Specifically, it cannot be determined from the SPOT referral database
whether all behaviors observed were included for each referred
passenger by each BDO or whether only the behaviors that were
sufficient for a LEO referral were recorded into the database. It is
not possible to determine from the database if the number of observed
behaviors entered for a given passenger was the total number of
observed behaviors, or whether additional behaviors were observed. A
rigorous analysis of the relative effects of the different behaviors
on the outcomes of the use of SPOT would require each BDO to record,
for each of the observable behaviors, whether it was or was not
observed.
TSA also provided technical comments that we incorporated as
appropriate.
We will send copies of this report to the Secretary of Homeland
Security; the TSA Administrator (Acting); and interested congressional
committees as appropriate. The report will also be available at no
charge on the GAO Web site at [hyperlink, http://www.gao.gov].
If you or your staff have any questions about this report, please
contact me at (202) 512-4379 or lords@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. Key contributors to this report are
acknowledged in appendix III.
Sincerely yours,
Signed by:
Stephen M. Lord:
Director, Homeland Security and Justice Issues:
[End of section]
Appendix I: Scope and Methodology:
To determine the extent to which the Transportation Security
Administration (TSA) determined whether the Screening of Passengers By
Observation Techniques (SPOT) program had a scientifically-validated
basis for identifying passengers before deploying it, we reviewed
literature on behavior analysis by subject matter experts, interviewed
seven experts in behavior analysis, interviewed other federal agencies
and entities about how they use behavior detection techniques, and
analyzed relevant reports and books on the topic. These included a
2008 study by the National Research Council of the National Academy of
Sciences that has a discussion regarding deception and behavioral
surveillance, as well as other issues related to behavioral
analysis.[Footnote 115] We interviewed Dr. Herbert S. Lin, who was a
primary author of the report, as well as Dr. Robert W. Levenson, and
Dr. Stephen E. Fienberg, both members of the Academy committee that
oversaw the report, about the report's findings with regard to
behavior detection, and the extent to which behavior detection in a
complex environment, such as an airport terminal, has been validated
with regard to its effectiveness in identifying persons who may be a
risk to aviation security. Other behavior detection experts we
consulted were Dr. Paul Ekman;[Footnote 116] Dr. Mark Frank;[Footnote
117] Dr. David Givens;[Footnote 118] Dr. David Matsumoto;[Footnote
119] and Mr. Rafi Ron, former director of security at Israel's Ben-
Gurion Airport. Dr. Ekman, Dr. Frank, and Mr. Ron provided expert
advice for the National Research Council study. Dr. Givens was
identified by TSA as having been their principal source for the
nonverbal behavior indicators used by the SPOT program. We also
interviewed Dr. Lawrence M. Wein, an expert in emergency responses to
terror attacks and mathematical models in operations management.
[Footnote 120] In addition, we interviewed officials from the
Department of Homeland Security's (DHS) Science and Technology (S&T)
Directorate regarding their ongoing research into behavior detection.
Although the views of these experts cannot be generalized across all
experts in behavior analysis because we selected individuals based on
their publications on behavioral analysis or related topics, their
recognized accomplishments and expertise, and, in some cases, TSA's
use of their work or expertise to design and review the SPOT program's
behaviors, they provided us with an overall understanding of the
fundamentals of behavior analysis, and how it could be applied.
To determine the basis for TSA's strategy to develop and deploy SPOT
and evaluate to what extent SPOT was informed by a cost-benefit
analysis and a strategic plan, we reviewed program documentation,
including briefings prepared by the SPOT program office during the
course of developing and fielding SPOT, two versions of a strategic
plan for SPOT, and the 2009 SPOT standard operating procedures
guidance. We compared the plans and analyses used by TSA to develop
and implement SPOT to criteria on how to develop and implement
programs in DHS's 2006 Cost Benefit Analysis Guidebook,[Footnote 121]
as well as to Office of Management and Budget guidance on the utility
of cost-benefit analyses in program implementation.[Footnote 122] We
also analyzed the development of SPOT in light of the standards and
criteria cited in DHS's 2006 National Infrastructure Protection Plan.
We met with relevant TSA officials to discuss these issues. To assess
whether DHS developed an effective strategic plan for SPOT prior to
implementing the program, we interviewed TSA officials involved in
development of the SPOT strategic plan. We analyzed whether the SPOT
plan incorporated the desirable characteristics of an effective
strategic plan as identified by previous GAO work on what strategic
plans should include to be considered effective, such as a risk
assessment, cost and resources analysis, and a means for collaboration
with other key entities.[Footnote 123] We also examined it in light of
the requirements of the Government Performance and Results Act of
1993, which specifies the elements of strategic plans for government
programs.[Footnote 124] We assessed whether the SPOT strategic plan
was followed by TSA. As part of our analysis of the planning for SPOT
before it was implemented on a nationwide basis, we reviewed TSA
documentation related to the development and pilot testing of SPOT,
such as a TSA white paper on SPOT, and interviewed key program
officials from both headquarters and field offices.[Footnote 125]
We also interviewed cognizant officials from other U.S. government
agencies and agency entities that utilize behavior detection in their
work, including U.S. Customs and Border Protection (CBP), the U.S.
Secret Service, the TSA's Federal Air Marshal Service (FAMS)
component, and the Federal Bureau of Investigation (FBI). We sought
their views on the utility of various behavior detection methods,
their experience with practicing behavior detection, and asked them
about the extent to which TSA had consulted with them in developing
and implementing the SPOT program.
To better understand how SPOT incorporated expertise about the use of
behavior detection in an airport setting, we interviewed officials
from Israel's El Al Airlines, which is cited by TSA as having provided
part of the basis of the SPOT program. We asked about El Al's methods
to ensure the security of its passenger aircraft, and also interviewed
a former head of security at Israel's Ben-Gurion airport, who has
advised TSA on security issues. We asked TSA and SPOT program
officials about their consultations with El Al, and about the ways in
which they had utilized El Al's expertise, as well as about any other
entities whose expertise they may have adopted into SPOT.
To determine the challenges, if any, that emerged during
implementation of the SPOT program, we interviewed headquarters and
field personnel about how the program has utilized the resources
available to it to ensure that it is effective. These resources
included the support of law enforcement officers (LEOs), to whom
passengers are referred by Behavior Detection Officers (BDOs) for
additional questioning. In addition, we interviewed SPOT program and
TSA officials about the databases available to them at TSA's
Transportation Security Operations Center to determine if a suspect
passenger is being sought by other U.S. law enforcement or
intelligence entities, and whether there is guidance for BDOs on when
and how to contact the Transportation Security Operations Center. We
also asked about whether there is guidance and training for BDOs on
how to access TSA's Transportation Information Sharing System
database, which is owned by FAMS, and is available through the
Transportation Security Operations Center.[Footnote 126] To determine
if any management challenges had emerged related to management
controls in developing and implementing SPOT, we compared TSA's
approach for implementing and managing the SPOT program with GAO's
Standards for Internal Control in the Federal Government[Footnote 127]
and with risk management principles we had previously identified.
[Footnote 128] Our legal counsel office reviewed court decisions
relevant to the SPOT program. In addition, we interviewed attorneys
from the American Civil Liberties Union, and obtained and reviewed
TSA's Privacy Impact Assessments for SPOT, the Transportation Security
Operations Center, and the Transportation Information Sharing System.
We also met with and discussed relevant privacy and legal issues with
TSA's Offices of Privacy and Civil Rights/Civil Liberties. To obtain
data about certain aspects of the SPOT program that the SPOT program
office did not have, we conducted a survey of Federal Security
Directors[Footnote 129]whose responsibilities included security at all
161 SPOT airports at the time of our survey. (Some Federal Security
Directors have responsibility for more than one airport.) We obtained
a 100 percent response rate. This survey asked, among other things,
about whether there were cameras at security checkpoints that record
the interactions of Transportation Security Officers (TSO), BDOs, and
passengers; if the airport authority had an agreement with TSA that
specifies certain law enforcement actions during a SPOT referral; and
if there was an agreement, or any other comparable guidance that
specified a time limit for LEOs to come to checkpoints after being
called for help by BDOs.
To determine the extent to which TSA has measured SPOT's effect on
aviation security, we obtained and analyzed the TSA SPOT referral
database, which records all incidents in which BDOs refer passengers
to secondary, more intensive questioning, and which also records all
incidents in which BDOs chose to refer passengers to LEOs. We found
that the SPOT database was sufficiently reliable to count the number
of arrests resulting from referrals from BDOs to LEOs, for examining
the reasons for each arrest, and for counting the percentage of times
that LEOs responded to BDO calls for service, and the length of time
required. Use of these data required us to resolve apparent
contradictions and anomalies in the database to make the data usable.
Because of data problems, we were unable to conduct analyses to assess
whether any behavior or combination of behaviors could be used to
predict the final outcome of an incident involving the use of SPOT. In
addition, we reviewed relevant standardization team reports and
observed a standardization team visit in operation.
In addition, we spoke with BDO managers, Federal Security Directors,
and Assistant Federal Security Directors to determine how BDOs are
evaluated. To do so, we conducted site visits to 15 commercial
airports at which BDOs and SPOT have been deployed, or almost 10
percent of the 161 airports with SPOT. We chose these airports taking
into account the following criteria, among others: (1) each airport
had BDOs deployed, and at each, the SPOT program had been in effect
for no less than 3 months; (2) airports were chosen to provide a
variety of sizes, as measured in annual passenger volume; physical
location within the country (northeast, southwest, central, Pacific
Coast, rural, urban); and estimated risk of terrorist incident, using
DHS's Current Airports Threat Assessment[Footnote 130] list (visiting
6 that were in the top 10, and others much lower); (3) BDOs who are
employed by contractors, rather than employed directly by TSA; and (4)
airports with LEOs who were identified to us by TSA as having received
some form of behavior detection training and airports where they were
not known to have received such training. In addition, we took into
account the location of the airports with regard their proximity to
subject matter experts on behavior detection whom we wished to
interview, as well as the time and cost required to reach certain
airports.
At each of the airports we visited, we interviewed cognizant
officials, including the Federal Security Director or Assistant
assigned to the airport, the BDO program manager, one or two BDOs, and
one or two LEOs who have interacted with BDOs. Since each of these
airports differs in terms of passenger volume, physical size and
layout, geographic location, and potential value as a target for
terrorism, among other things, the results from these visits are not
generalizable to other airports. However, these visits provided
helpful insight into the operation of SPOT at airports.
In addition, to determine if individuals had transited SPOT airports
who were later charged with or pleaded guilty to terrorism-related
offenses, we reviewed information contained in (1) the Treasury
Enforcement Communication System II database maintained by CBP;
[Footnote 131] (2) Department of Justice information and court
documents, including indictments and related documents; and (3) media
accounts of individuals accused of terrorism-related activities. We
compared information pertaining to these individuals' dates of transit
to the dates when SPOT was deployed to the various airports identified
in the Treasury Enforcement Communication System and Justice
Department data to determine if SPOT had been deployed at a given
airport when the transits occurred. Further, we used our survey of
Federal Security Directors at SPOT airports to determine the extent to
which video surveillance cameras are present at checkpoints.
To assess the extent that SPOT training incorporates the attributes of
an effective training program, we had training experts at TSA
headquarters complete a training assessment tool that we developed
using our prior work for assessing training courses and curricula.
[Footnote 132] To address training-related issues, including to
understand better how other entities train their employees in behavior
detection, and what their curricula include, we conducted site visits
to the Secret Service, FAMS, CBP, and the FBI, and also interviewed
nongovernmental experts on behavior detection (our selection of these
experts is discussed above). As part of our assessment of SPOT
training, we attended the basic SPOT training course given to BDOs, as
well as the advanced SPOT course on behavior detection. We interviewed
BDOs and BDO managers about the SPOT training, as well as officials of
El Al airlines, with regard to how El Al trains and tests its
personnel who utilize behavior recognition and analysis as part of
their assessment of El Al passengers.
We conducted this performance audit from May 2008 through May 2010, in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.
[End of section]
Appendix II: DHS Comments:
U.S. Department of Homeland Security:
Washington, DC 20528:
May 3, 2010:
Mr. Steve Lord:
Director, Homeland Security & Justice Issues:
U.S. Government Accountability Office (GAO):
441 G Street, NW:
Washington, DC 20548:
Dear Mr. Lord:
Thank you for the opportunity to review and comment on GAO-10-157SU,
the draft report titled: Aviation Security: Efforts to Validate
Aspects of TSA 's Screening of Passengers by Observation Techniques
(SPOT) Program Underway, But Opportunities Exist to Strengthen
Validation and Address Operational Changes. The Transportation
Security Administration (TSA) appreciates the U.S. Government
Accountability Office's work in planning and conducting its review and
issuing this report.
TSA deployed the SPOT program in an effort to mitigate the threat of
individuals with potentially hostile intent from boarding a commercial
airplane and causing harm. Congress has encouraged the use of behavior
recognition to enhance aviation security and has provided resources to
support its implementation and expansion. The SPOT program fulfills
the mandate of Section 1611 of the Implementing Recommendations of the
9/11 Commission Act, P.L. 11053, that "TSA shall provide advanced
training to the transportation security officers for the development
of specialized security skills, including behavior observation and
analysis...in order to enhance the effectiveness of layered
transportation security measures."
Intelligence continues to show there is no specific terrorist profile.
In a March 10, 2010, hearing before the Senate Homeland Security and
Governmental Affairs Committee, TSA Acting Administrator Gale Rossides
highlighted the challenge faced by TSA leaders in "balancing the
requirement to screen all passengers and to actually focus our
officers' attention on the right passengers." TSA designed SPOT to
increase its ability to focus on the "right passengers" by identifying
persons exhibiting behaviors and appearances that may indicate stress,
fear, and deception, and distinguish them from other travelers.
SPOT is Based on Scientific Research and Law Enforcement Practices:
TSAs development and deployment of SPOT was a planned and deliberate
process based on more than 3 years of operational test-bed assessment
of SPOT at Boston's Logan International Airport from June 2003 until
nationwide rollout began in fiscal year (FY) 2007. TSA carefully
developed SPOT by using selective behaviors recognized within both the
scientific and law enforcement communities as displaying stress, fear,
and deception. A SPOT working group, made up of various TSA and U.S.
Department of Homeland Security (DHS) components,[Footnote 133] was
created in February 2004. Other organizations, such as the
Massachusetts State Police, the Federal Bureau of Investigation (FBI)
Behavioral Sciences Unit, and the Federal Law Enforcement Training
Center, were also involved in SPOT development. Through these working
groups, TSA has developed and finalized SPOT standard operating
procedures (SOPs) for a common ability to assess behaviors indicating
hostile intent for both aviation and mass transit modes of
transportation. TSA continues to consult with its SPOT working group
partners as it updates the procedures and science behind the program.
Decades of scientific research have shown the behaviors to be
universal in their manifestation. In fact, the DHS Science and
Technology Directorate (S&T) completed a study on suicide bomber
indicators in July 2009 that illustrates a very high degree of overlap
between operationally reported suicide bomber indicators and TSA SPOT
behaviors. This result further bolsters TSA's contention that the SPOT
program draws from the best practices of many defense, intelligence,
and law enforcement organizations.
SPOT Scientific Validation is Ongoing:
S&T began research in 2007 to examine the validity of the SPOT
program. The series of studies involved in this research is designed
to assess the validity of the SPOT scoring system, including the use
of individual behavioral indicators to identify high-risk travelers.
More specifically, S&T's research plan aims to examine the extent to
which these behavioral indicators are appropriate for screening
purposes and lead to appropriate and correct security decisions. When
this study is complete, SPOT will be one of the most, if not the most,
rigorously tested behavior-based security screening programs in
existence.
Results of this work will establish a scientific basis of the extent
to which the SPOT program, including its instrument and methods, such
as the SPOT Referral Report and SOPs, are valid. Although it is
challenging to establish the validity of a deterrent program in which
the outcomes of interest are extremely rare, critical elements of
reliability and validity will be rigorously assessed. Of particular
importance is the evaluation of criterion-related validity, or the
extent to which travelers are correctly selected for screening based
on the SPOT scoring system. Establishing this degree of classification
accuracy justifies the use of the SPOT program to discriminate high-
risk travelers from low-risk travelers. Regardless of any other
metrics, the extent to which the SPOT scores accurately identify high-
risk travelers is critically important to program validity.
Following criterion-related validity, the next central element of
validity is the consistency of implementation of the instrument and
program. This will be examined in a variety of ways, including an
investigation of the consistency in the operational use of SPOT
behavioral indicators Behavior Detection Officers (BDOs) and across
locations and time periods, all of which represents reliability
assessment. Finally, construct-related validity, or the extent to
which the SPOT program behaviors truly represent the expressions of
high-risk travelers, will be examined by comparing the SPOT behaviors
to similar instruments in use for the same purpose. S&T's July 2009
study of suicide bomber indicators was the first step in evaluating
construct-related validity.
This research is expected to be completed in FY 2011. TSA understands
that after this validation is complete, there will be other areas
where further research should be conducted, and it is TSA's intention
to complete this research.
National Academy of Sciences (NAS) Report Does Not Represent an
Exhaustive or Definitive Review of the Research or Operational
Literature on Behavioral Screening:
TSA would like to specifically address a few comments in the GAO-10-
157SU report that we believe are inaccurate. The report draws heavily
from a National Academy of Sciences (NAS) report which is being
improperly relied upon. As the sponsor of the NAS study, DHS S&T
questioned its findings, stating that the study lacked sufficient
information for its conclusions because the NAS study principally
focused on privacy as it relates to behavioral surveillance”not on
behavioral surveillance technology itself. The study was not intended
to, and the results do not represent an exhaustive or definitive
review of the research or operational literature on behavioral and
physiological screening, including recent findings from unpublished
DHS, defense, and intelligence community studies. Furthermore, it
should be noted that the report did not study the SPOT program, nor
did any of the researchers conduct interviews with SPOT program
personnel.
Additionally, GAO states that "DHS S&T could not provide us with
specific contacts related to the sources of this research." This
statement is not accurate. The record should reflect that DHS S&T
provided all requested documents that represented S&T-sponsored
research and for which S&T possessed the requisite release authority.
DHS was not able to release specific documents related to research for
which it was not the originator.
The report further states that the audit team was unable to use the
SPOT referral data to assess whether any behavior or combination of
SPOT behaviors could be used to reliably predict the final outcome of
an incident involving the use of SPOT. However, DHS S&T was able to
successfully conduct some preliminary analysis of the SPOT referral
database. Prior to analysis of the SPOT reports, S&T worked with TSA
to verify the scores assigned to each indicator with the SPOT score
sheets and to rescore the pertinent sections and total accordingly for
nearly 100,000 operational reports from 2008. While random errors were
noted, errors in large databases that require manual entry are not
uncommon. Convention suggests that large databases like this typically
include an error rate of 3 to 5 percent. As long as such errors are
random, the analytical method is robust enough to account for random
errors in this range.
In conclusion, TSA strongly believes that behavior detection is a
vital layer in its aviation security strategy, and will continue to
strengthen as the program matures. Leaders within the community of
behavior detection researchers agree. TSA appreciates GAO's work to
identify opportunities to enhance the SPOT program, and we will
continue to work diligently to address the issues identified by GAO.
Our ongoing progress demonstrates our commitment to TSA's mission of
securing our Nation's transportation systems.
We also appreciate the opportunity to provide you with, in
collaboration with OHS S&T, comments to GAO's audit recommendations.
Recommendation 1; To help ensure that SPOT is based on valid
scientific principles that can be effectively applied in an airport
environment, we (GAO) recommend that the Secretary of Homeland
Security convene an independent panel of experts to review the
methodology of the S&T Directorate study on the SPOT program before
the study is implemented to determine whether the study's methodology
is sufficiently comprehensive to validate the SPOT program. This
assessment should include appropriate input from other federal
agencies with expertise in behavior detection and relevant subject
matter experts.
Concurs: The U.S. Department of Homeland Security (DHS) Science &
Technology Directorate's (S&T) current validation process includes an
independent and comprehensive review of the ongoing SPOT study to be
conducted in support of and in collaboration with the TSA SPOT
program. The assessment will include input from other Federal agencies
with expertise in behavior detection and relevant subject matter
experts. S&T will work with TSA to present the SPOT validation project
to the panel, produce a report summarizing the panel's
recommendations, and implement pertinent suggestions in FY 2010.
GAO further recommends that if this research determines that the SPOT
program has a scientifically validated basis for using behavior
detection for counterterrorism purposes in the airport environment,
then the TSA Administrator take the following actions:
Recommendation 2: Conduct a comprehensive risk assessment to include
threat, vulnerability, and consequence of airports nationwide to
determine the effective deployment of SPOT If TSA's ongoing Aviation
Modal Risk Assessment lacks this information.
Concur: TSA's Aviation Modal Risk Assessment (AMRA) is designed to
evaluate the transportation security risk landscape and compare it to
other modes. However the AMRA does not evaluate risk effectiveness of
countermeasures or optimal deployment strategies. For the Aviation
mode, TSA uses the Risk Management Analysis Tool (RMAT), a risk
simulation model based on laboratory and operational data that
evaluates risk using threat inputs, vulnerability information, and
consequence estimates. TSA is in the process of conducting an initial
risk analysis on the SPOT program using RMAT. The risk analysis is
based on the initial SPOT validation results and will be updated as
the validation study results are finalized.
Recommendation 3: Perform a cost-benefit analysis of the SPOT program
including a comparison of the SPOT program with other security
screening programs, such as random screening, or already existing
security measures.
Concur: The SPOT program will use RMAT to perform a cost-benefit
analysis of Behavior Detection Officers (BD0s) as a countermeasure.
The first step in the process is the initial risk assessment that is
being conducted on the SPOT program using RMAT. For the cost-benefit
analysis, costs will be defined as the 5-year total cost of the
countermeasure across the aviation system. Benefit will be defined as
risk-reduction across the aviation security system against a portfolio
of scenarios. TSA is currently developing an initial cost-benefit
analysis for a variety of passenger-screening countermeasures
including BDOs using the RMAT tool as a basis for analysis. BDOs'
flexibility across a variety of risk scenarios suggests that behavior
detection is a cost-effective countermeasure.
Recommendation 4. Revise and implement the SPOT strategic plan by
incorporating risk assessment information, identifying cost and
resources, linking it to other related TSA strategic documents,
describing how SPOT is integrated and implemented with TSA's other
layers of aviation security, and providing guidance on bow to
effectively link the roles, responsibilities and capabilities of
federal, state, and local officials providing program support.
Concur: The RMAT risk analysis of the BDO program is assisting the
SPOT program in identifying other countermeasure capabilities that are
linked to the behavior detection capability. This analysis will allow
the SPOT program to develop a revision to the SPOT strategic plan that
will incorporate the elements identified in the recommendation.
Recommendation 5: Study the feasibility of using airport checkpoint-
surveillance video recordings of individuals transiting checkpoints,
and who were later charged with or pleaded guilty to terrorism-related
offenses, to enhance its understanding of terrorist behaviors in the
airport checkpoint environment.
Concur: TSA will study the feasibility of using checkpoint
surveillance video recordings of individuals transiting checkpoints,
and who were later charged with or pleaded guilty to terrorism-related
offenses. TSA agrees that this could be a useful tool in understanding
terrorist behaviors in the checkpoint environment.
Additionally, TSA is currently working with DHS S&T/Human Factors to
conduct operational video validation of the SPOT program. TSA will use
a variety of video case studies to validate the SPOT program
including, if possible, reviewing video of terrorists transiting the
TSA checkpoint. It is exceedingly rare, however, for video cameras to
capture terrorists transiting TSA checkpoints. Unfortunately, this
factor significantly reduces the feasibility of conducting these case
studies.
GAO also recommends that concurrent with the DIIS S&T Directorate
study of SPOT, and an independent panel assessment of the soundness of
the methodology of the S&T study, the TSA Administrator take the
following actions:
Recommendation 6: Provide guidance in the SPOT SOP or other TSA
directive to BDOs, or other TSA personnel, on inputting data into the
Transportation Information Sharing System (TISS) and set milestones
and a timeframe for deploying Transportation Information Sharing
System access to SPOT airports so that TSA and intelligence community
entities have information from all SPOT Law Enforcement officer (LEO)
referrals readily available to assist in "connecting the dots" and
identifying potential terror plots.
Concur: TSA is currently undergoing a revision of the SPOT Standard
Operating Procedure (SOP). The SOP will provide guidance directing the
input of BDO data into TISS. TSA anticipates release of the updated
SPOT SOP in FY 2010. Additionally, TSA is currently drafting a formal
plan to include milestones and a timeframe for deploying T1SS access
to all SPOT airports.
Recommendation 7: Implement the steps called for in the TSA Office of
Security Operations Business plan to develop a standardized process
for allowing BDOs or other designated airport officials to send
information to TSA's Transportation Security Operations Center (TSOC)
about passengers whose behavior indicates that they may pose a threat
to security, and provide guidance on how designated TSA officials are
to receive information back from the Transportation Security
Operations Center.
Concur: TSA has convened a working group made up of members of the
Office of Security Operations, Office of Chief Counsel, Office of
Intelligence, and the Office of Law Enforcement/Federal Air Marshal
Service (FAMS) to address this recommendation. TSA is developing a
system and procedure for sending and receiving information from the
TSOC and anticipates having a system in place in FY 2010. It should be
noted that information from BDO referrals has been transmitted to the
TSOC previously; however, TSA agrees to institute a standardized
process.
Recommendation 8: Utilize all of the databases available to the
Transportation Security Operations Center when running passengers who
rise to the level of a LEO referral against intelligence and criminal
databases.
Concur in principle: TSA has convened a working group composed of
members of the Office of Security Operations, Office of Chief Counsel,
Office of Intelligence, and the Office of Law Enforcement/FAMS to
address this recommendation. This group will conduct a feasibility
study during FY 2010 to examine if this recommendation can be fully
implemented. This study will look at the various authorities,
permissions, and limitations of each of the databases or systems cited
in this audit. Access to some of the systems, such as Criminal History
Record Check (CHRC), requires more justification than a BDO referral.
Because some of the databases or systems contain classified
information, TSA will also need to adopt a communication strategy to
transmit the passenger information back and forth between the BDO and
TSOC. TSA will work on a process to collect the passenger information,
verify the passenger's identity, through checks of databases, and
analyze that information to determine if the passenger is the subject
of an investigation and may pose a risk to aviation.
Recommendation 9: Establish a plan that includes objectives,
milestones, and timeframes to develop outcome-oriented performance
measures to help refine the current methods used by Behavior Detection
Officers for identifying individuals who may pose a risk to the
aviation system.
Concur: TSA understands the value of outcome-oriented performance
measures. However, as noted by GAO, there is difficulty in
establishing these measures for a deterrence-based program.
Nonetheless, TSA will consult with industry experts to develop outcome-
oriented performance measures. TSA will establish a plan that includes
objectives, milestones. and timeframes, with an end result of
producing outcome-oriented performance measures to help refine the
current methods used by BDOs for identifying individuals who may pose
a risk to the aviation system.
Recommendation 10: Establish controls to help ensure completeness,
accuracy, authorization, and validity of data collected during SPOT
screening.
Concur: In March 2010, TSA migrated the SPOT database to TSA's
Performance Management Information System. This migration greatly
enhances the SPOT program's capabilities, as they relate to
completeness, accuracy, authorization, and validity of data collected
during SPOT screening. Additional controls have been put in place to
address the shortcomings of the previous database which were
highlighted by GAO. TSA is also examining a technology solution to
allow one-time transcription of all SPOT referral data. This will
reduce the possibility of errors due to incorrect transcription from
one medium to another.
Recommendation 11: Establish timeframes and milestones for its plan to
systematically conduct evaluations of the SPOT training program on a
periodic basis.
Concur: DHS S&T, in conjunction with TSA has sponsored a BDO Job Task
Analysis (JTA). Outputs of the JTA will include Knowledge, Skills,
Abilities, and Other characteristics of BDOs and Training Learning
Objectives. These two items will enable TSA to conduct an in-depth
training gap analysis. This analysis will begin immediately following
completion of the JTA and will take approximately three months to
complete. Upon completion of the training gap analysis, TSA will
develop detailed project plans with milestones and schedules based on
the scope of the overall curriculum development/revision effort.
Sincerely yours,
Signed by:
Jerald E. Levine:
Director:
DHS GAO/OIG Liaison Office:
[End of section]
Appendix III: GAO Contacts and Staff Acknowledgments:
GAO Contact:
Stephen M. Lord, (202) 512-4379, or lords@gao.gov:
Staff Acknowledgments:
In addition to the contact named above, David M. Bruno, Assistant
Director, and Jonathan R. Tumin, managed this assignment. Ryan
Consaul, Jeff C. Jensen, Kevin Remondini, and Julie E. Silvers made
significant contributions to the work. Arthur James, Jr., Amanda
Miller, and Douglas Sloane assisted with design, methodology, and data
analysis. Chris Dionis assisted with issues related to training.
Katherine Davis and Debra Sebastian provided assistance in report
preparation; Tracey King and Tom Lombardi provided legal support; and
Pille Anvelt and Barbara Hills developed the report graphics.
[End of section]
Footnotes:
[1] BDOs must have at least 12 months experience as a TSO, or related
security work experience, and must pass a BDO training course.
[2] TSA cautions that the applicability of El Al's security processes
to those used by TSA is constrained by differences in the scale of El
Al's worldwide operations and the flexibilities that El Al has in
implementing security processes compared to constraints on TSA. For
example, El Al security screeners are encouraged to spend as much time
with passengers as needed, and are not concerned whether passengers
experience delays in boarding an aircraft.
[3] For the purposes of this report, the term "TSA-regulated airport"
refers to a U.S. airport operating under a TSA-approved security
program.
[4] Visible Intermodal Prevention and Response teams are comprised of
federal air marshals, surface transportation security inspectors,
TSOs, BDOs, and canines.
[5] TSA classifies its regulated airports in the United States into
one of five categories--X, I, II, III, and IV. Generally, category X
airports have the largest number of passenger boardings and category
IV airports have the least.
[6] See H.R. Rep. No. 111-298 at 77 (2009) (Conf. Rep.). The
conference report directed TSA to report, no later than 60 days after
enactment, on the scientific basis for using behavior pattern
recognition for observing airline passengers for signs of hostile
intent, the effectiveness of the SPOT program in meeting its goals and
objectives, and the justification for expanding the program. The
conference report also directed us to review this report and to
provide our findings to the Committees no later than 120 days after
the TSA report is submitted. TSA completed its report to Congress on
March 15, 2010.
[7] National Research Council, Protecting Individual Privacy in the
Struggle Against Terrorists: A Framework for Assessment (Washington,
D.C.: National Academies Press, 2008). We reviewed the approach used
and the information provided in this study and found the study and its
results to be reliable for the purposes for which we used it in this
report.
[8] See appendix I for additional information on the experts we
interviewed.
[9] For reasons of scope, we did not assess the scientific basis of
the methods and processes used by these agencies in their application
of behavioral detection.
[10] Although SPOT is based in some respects on El Al's aviation
security program, El Al's processes differ in substantive ways from
those used by the SPOT program. In particular, El Al does not use a
list of specific behaviors with numerical values for each, or a
numerical threshold to determine whether or not to question a
passenger; rather, El Al security officers utilize behavioral
indicators as a basis for interviewing all passengers boarding El Al
passenger aircraft, and accessing relevant intelligence databases,
when deemed appropriate. In addition, El Al officials told us that
they train all their personnel--not just security officers--in
elements of behavior analysis, and conduct covert tests of their
employees' attentiveness at frequent intervals. According to these
officials, El Al also permits what is termed "profiling," in which
passengers may be singled out for further questioning based on their
nationality, ethnicity, religion, appearance, or other ascriptive
characteristics, but these are not the only basis on which a passenger
may be questioned. In addition, El Al security officers are empowered
to bar any passenger from boarding an aircraft. The scale of El Al
operations is considerably smaller than that of major airlines
operating within the United States. As of 2008, El Al had a fleet of
34 aircraft. In Israel, El Al operates out of one hub airport, Ben-
Gurion International, and also flies to Eilat, a city in southern
Israel; in contrast, there are 457 TSA-regulated airports in the
United States. In 2008, El Al had passenger boardings of about 3.6
million; in contrast, Southwest Airlines alone flew about 102 million
passengers in the same year.
[11] Unless otherwise noted in the report, we refer to the SPOT
strategic plan issued in March 2007.
[12] DHS, Cost Benefit Analysis Guidebook (Washington, D.C.: Feb. 1,
2006).
[13] OMB, Circular No. A-94, Guidelines and Discount Rates for Benefit-
Cost Analysis of Federal Programs (Washington, D.C.: October 1992);
Circular No. A-4, Regulatory Analysis (Washington, D.C.: Sept. 2003).
[14] See appendix I for additional details on the airports we visited.
[15] At airports participating in TSA's Screening Partnership Program,
private-sector contractors perform screening activities, including
SPOT, in accordance with TSA requirements and oversight. See 49 U.S.C.
§ 44920. Unless otherwise specified, references to TSOs include
private-sector contract screeners. For more information, see GAO,
Aviation Security: Progress Made to Set Up Program Using Private-
Sector Airport Screeners, but More Work Remains, GAO-06-166
(Washington, D.C.: Mar. 31, 2006).
[16] The TSA Current Airport Threat Assessment is a threat estimate
designed to provide a snapshot of the current terrorist threat to
airports in the United States as well as for major international
airports serving as last points of departure for U.S. airlines.
[17] GAO, Standards for Internal Controls in the Federal Government,
[hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]
(Washington, D.C.: November 1999).
[18] See GAO, Risk Management: Further Refinements Needed to Assess
Risks and Prioritize Protective Measures at Ports and Other Critical
Infrastructure, [hyperlink, http://www.gao.gov/products/GAO-06-91]
(Washington, D.C.: Dec. 15, 2005) and Transportation Security:
Comprehensive Risk Assessments and Stronger Internal Controls Needed
to Help Inform TSA Resource Allocation, [hyperlink,
http://www.gao.gov/products/GAO-09-492] (Washington, D.C.: Mar. 27,
2009).
[19] Federal Security Directors are the highest ranking TSA officials
responsible for security operations at TSA-regulated airports. See 49
U.S.C. § 44933. They and their assistants coordinate with both federal
and nonfederal entities present at their airports, including the FAMS,
the Drug Enforcement Administration, and CBP. When appropriate,
Federal Security Directors may bar an individual from boarding an
aircraft.
[20] The SPOT referral data we analyzed covered the period May 29,
2004, through August 31, 2008. These were the data available at the
time of our analysis.
[21] GAO, Human Capital: A Guide for Assessing Strategic Training and
Development Efforts in the Federal Government, [hyperlink,
http://www.gao.gov/products/GAO-04-546G] (Washington, D.C.: Mar. 1,
2004).
[22] See Pub. L. No. 107-71, 115 Stat. 597 (2001). For purposes of
this report, "commercial aircraft" refers to a U.S. or foreign-based
air carrier operating under TSA-approved security programs with
regularly scheduled passenger operations to or from a U.S. airport.
[23] See 49 C.F.R. pt. 1542. Some commercial airports with fewer than
2,500 annual enplanements (passengers boarding an aircraft) do not
have TSA-approved screening processes. Enplanements are the number of
paying passengers on a scheduled or nonscheduled (charter) flight.
Infants and airline personnel are not included. A stop at an airport
is not considered an enplanement if the passenger does not transfer
aircraft.
[24] Private-sector screeners under contract to and overseen by TSA,
and not TSOs, perform screening activities at airports participating
in TSA's Screening Partnership Program. See 49 U.S.C. § 44920.
[25] These requests would typically be made to the law enforcement
entity employing the LEO, such as the airport authority police
department. The department would have a computer that can access NCIC.
According to the FBI's website, the NCIC database consists of 19 files
or databases. Seven are property files which contain records for
articles, boats, guns, license plates, securities, vehicles, and
vehicle and boat parts. Twelve are person files are the Convicted
Sexual Offender Registry, Foreign Fugitive, Identity Theft,
Immigration Violator, Missing Person, Protection Order, Supervised
Release, Unidentified Person, U.S. Secret Service Protective, Violent
Gang and Terrorist Organization, and Wanted Person Files. The
Interstate Identification Index, which contains automated criminal
history record information, is also accessible through the same
network as NCIC. The Violent Gang and Terrorist Organization file
includes the names of known or suspected terrorists.
[26] See 49 CFR §§ 1542.215, .217.
[27] TSA-regulated airports have regular commercial passenger service
and comply with TSA regulations for passengers and their property in
order to operate.
[28] National Research Council, Protecting Individual Privacy in the
Struggle Against Terrorists: A Framework for Assessment (Washington,
D.C.: National Academies Press, 2008). The report's preparation was
overseen by the National Academy of Sciences Committee on Technical
and Privacy Dimensions of Information for Terrorism Prevention and
Other National Goals. Although the report addresses broader issues
related to privacy and data mining, a senior National Research Council
official stated that the committee included behavior detection as a
focus because any behavior detection program could have privacy
implications.
[29] Specifically, the report states that the scientific support for
linkages between behavioral and physiological markers and mental state
is strongest for elementary states, such as simple emotions; weak for
more complex states, such as deception; and nonexistent for highly
complex states, such as when individuals hold terrorist intent and
beliefs.
[30] DHS's S&T Directorate could not provide us with specific contacts
related to the sources of this research.
[31] Peer review is the process of subjecting an author's scholarly
work, research, or ideas to the scrutiny of others who are experts in
the same field. Such review is considered a form of scientific
validation.
[32] For example, we reported that the National Institutes of Health
did not post its researchers' final reports because the risks
associated with posting results that have not been scrutinized and
validated by peer review are too great. See GAO, University Research:
Most Federal Agencies Need to Better Protect against Financial
Conflicts of Interest, [hyperlink,
http://www.gao.gov/products/GAO-04-31] (Washington, D.C.: November
2003).
[33] Dr. Ekman is professor emeritus of psychology at the University
of California Medical School, San Francisco, and is considered one of
the world's foremost experts on facial expressions. His books include:
Emotions Revealed: Recognizing Faces and Feelings to Improve
Communications and Emotional Life (New York: Holt and Company, 2003);
Emotion in the Human Face (New York: Pergamon Press, 1972); Unmasking
the Face: A Guide to Recognizing Emotions from Facial Clues (Englewood
Cliffs, N.J.: Prentice-Hall, 1975). Dr. Ekman has published more than
100 articles.
[34] Mr. Walters is the author of the Principles of Kinesic Interview
and Interrogation: 2ND Edition as well as numerous training materials
related to interviewing and interrogation techniques.
[35] Dr. Givens is the director of the nonprofit Center for Nonverbal
Studies, in Spokane, Washington. Dr. Givens is the author of Love
Signals: A Practical Field Guide to the Body Language of Courtship
(St. Martin's, New York, 2005) and Crime Signals: How to Spot a
Criminal Before You Become a Victim (St. Martin's, 2008). The Center's
Web site links to Dr. Givens' online reference tool, The Nonverbal
Dictionary of Gestures, Signs and Body Language Cues. Dr. Givens said
that he had did not know which nonverbal indicators had been selected
by TSA for use in SPOT, that he had not been asked by TSA to review
their choices from his list, or to review other aspects of the SPOT
program. According to Dr. Givens, attempting to detect more than nine
nonverbal indicators would be difficult for most individuals, even
those trained in behavior detection.
[36] Dr. Frank is Associate Professor, Department of Communication,
College of Arts and Sciences, at the University at Buffalo, State
University of New York. He is on the Advisory Board of the
University's Center for Unified Biometrics and Sensors, and has
conducted research supported by DHS, the Defense Advanced Research
Projects Agency, and the National Science Foundation. Dr. Frank told
us that he had observed SPOT at an airport and had some coordination
with TSA. However, he said that he had not reviewed the SPOT training
curriculum or the SPOT scoring system. Dr. Frank stated that no study
has been performed to validate use of behavior detection in an airport
setting.
[37] According to DHS's S&T Directorate, it completed a study on
suicide bomber indicators in July 2009. The program manager stated
that they reviewed 157 documents and identified approximately 1,200
suicide indicators, which were similar to SPOT suicide bomber
indicators. S&T stated that the study provides preliminary support for
the detection of suicide bomber indicators and that SPOT represents
best practices from defense and intelligence organizations.
[38] According to TSA, the FBI participated in discussions related to
SPOT's development in 2006.
[39] American Institutes for Research, Behavioral Indicators Related
to Deception in Individuals with Hostile Intentions: Interim Results
(Washington, D.C.: February 2008). According to S&T officials, this
review included research conducted prior to 2005.
[40] DHS could not provide us with specific contacts related to the
sources of this research; we were therefore unable to determine the
extent to which it has demonstrated linkages between behavioral and
physiological indicators and deception.
[41] Statement of the Under Secretary (Acting), DHS S&T Directorate,
before the Subcommittee on Homeland Security, Committee on
Appropriations, U.S. House of Representatives, March 26, 2009.
[42] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1].
[43] See GAO, Oil and Gas Royalties: The Federal System for Collecting
Oil and Gas Revenues Needs Comprehensive Reassessment, [hyperlink,
http://www.gao.gov/products/GAO-08-691] (Washington, D.C.: Sept. 3,
2008). GAO, Combating Nuclear Smuggling: Additional Actions Needed to
Ensure Adequate Testing of Next Generation Radiation Detection
Equipment, [hyperlink, http://www.gao.gov/products/GAO-07-124T]
(Washington, D.C.: Sept. 18, 2007). GAO, Space Operations: NASA
Efforts to Develop and Deploy Advanced Spacecraft Computers,
[hyperlink, http://www.gao.gov/products/GAO/IMTEC-89-17] (Washington,
D.C.: Mar. 31, 1989). GAO, Quadrennial Defense Review: Future Reviews
Could Benefit from Improved Department of Defense Analyses and Changes
to Legislative Requirements, [hyperlink,
http://www.gao.gov/products/GAO-07-709] (Washington, D.C.: Sept. 14,
2007). GAO, Coast Guard: Challenges for Addressing Budget Constraints,
[hyperlink, http://www.gao.gov/products/GAO/RCED-97-110] (Washington,
D.C.: May 1997).
[44] DHS's NIPP defines risk as a function of threat, vulnerability,
and consequence. Threat is an indication of the likelihood that a
specific type of attack will be initiated against a specific target or
class of targets. Vulnerability is the probability that a particular
attempted attack will succeed against a particular target or class of
targets. Consequence is the effect of a successful attack.
[45] As updated in 2009, the NIPP states that to be complete a risk
assessment is to assess threat, vulnerability, and consequence for
every defined risk scenario. However, because the original 2006
version of the NIPP described risk assessments that included all three
components as "credible," our previous reports use this term rather
than "complete" (see GAO, Transportation Security: Comprehensive Risk
Assessments and Stronger Internal Controls Needed to Help Inform TSA
Resource Allocation, [hyperlink,
http://www.gao.gov/products/GAO-09-492] (Washington, D.C.: Mar. 27,
2009)).
[46] We reported in March 2009 that TSA does not assign uncertainty or
varying levels of confidence associated with the intelligence
information used to identify threats to the transportation sector and
guide TSA's planning and investment decisions. Since TSA's
intelligence products have not assigned confidence levels to its
analytic judgments, it is difficult for TSA to correctly prioritize
its tactics and investments based on uncertain intelligence. In March
2009, we recommended that TSA work with the Director of National
Intelligence to determine the best approach for assigning uncertainty
or confidence levels to analytic intelligence products and to apply
this approach. TSA agreed with this recommendation and said it has
begun taking action to address it. See [hyperlink,
http://www.gao.gov/products/GAO-09-492].
[47] The AMRA is being developed by TSA pursuant to the National
Strategy for Aviation Security, which was issued according to Homeland
Security Presidential Directive-16. HSPD-16 directs the production of
an overarching national strategy to optimize and integrate government-
wide aviation security efforts. AMRA was previously known as the Air
Domain Risk Assessment or ADRA.
[48] Commercial aviation includes that sector of the nation's civil
aviation system that provides for the transportation of individuals by
scheduled or chartered operations for a fee, including air carriers
and airports. General aviation includes all civil aviation other than
commercial and military operations, including flight operations such
as personal/family transportation, emergency services, wildlife and
land surveys, traffic reporting, agricultural aviation, firefighting,
and law enforcement. Air cargo is defined as cargo carried on
passenger and all-cargo aircraft.
[49] In addition, TSA states that its risk management analysis toolset
may also be useful in prioritizing BDO deployments since the toolset
analyzes various scenarios for which the use of BDOs may be applicable.
[50] See GAO, Reexamining Regulations: Opportunities Exist to Improve
Effectiveness and Transparency of Retrospective Reviews, [hyperlink,
http://www.gao.gov/products/GAO-07-791] (Washington, D.C.: July 16,
2007).
[51] DHS, Cost Benefit Analysis Guidebook (Washington, D.C.: February
2006); OMB, Circular No. A-94, Guidelines and Discount Rates for
Benefit-Cost Analysis of Federal Programs (Washington, D.C.: October
1992); OMB, Circular No. A-4, Regulatory Analysis (Washington, D.C.:
September 2003).
[52] A pilot test is a preliminary test or study to try out procedures
and discover problems before the main study begins. This enables
researchers to make last minute corrections and adjustments. In a
pilot, the entire study with all of its instruments and procedures is
conducted in miniature. See W.P. Vogt, Dictionary of Statistics and
Methodology: A Nontechnical Guide for the Social Sciences (Newbury
Park: Sage Publications, 1993).
[53] The increase rate for TSA's other screening operations combined
was about 0.27 percent from fiscal year 2007 to fiscal year 2009 (from
$3.727 billion to $3.737 billion, a $10 million increase). The
screening operations account includes privatized screening; passenger
and baggage screener performance, compensation, and benefits; screener
training and other; human resource services; and checkpoint support.
[54] See H.R. Rep. No. 111-298, at 77 (2009) (Conf. Rep.).
[55] [hyperlink, http://www.gao.gov/products/GAO-04-408T].
[56] Pub. L. No. 103-62, 107 Stat. 285 (1993).
[57] See GAO, National Preparedness: FEMA Has Made Progress, but Needs
to Complete and Integrate Planning, Exercise, and Assessment Efforts,
[hyperlink, http://www.gao.gov/products/GAO-09-369] (Washington, D.C.:
Apr. 30, 2009).
[58] TSA, Strategic Plan for Behavior Detection Program (Washington,
D.C.: 2007).
[59] TSA, Office of Security Operations, Strategy Deployment
Organizational Business Plan for Fiscal Year 2010 (Washington, D.C.:
Apr. 8, 2008).
[60] Within the Transportation System Sector-Specific Plan, the
aviation implementation plan outlines transportation security goals
and key objectives with associated programs within the aviation
community. The plan notes that SPOT is intended to identify suspicious
activities within the aviation domain.
[61] TSA issued its Passenger Checkpoint Screening Program Strategic
Plan in August 2008 to outline its strategy and approach for
implementing advanced security capabilities in airport checkpoints
using a combination of people, processes, and technology at all
airport checkpoints. The plan cites TSA's behavior detection
capability as one of three strategic initiatives.
[62] The Transportation Security Operations Center is the central
operations and information-gathering point for TSA across the nation;
it serves as a 24/7-point of contact for all transportation security
concerns, providing access to multiple criminal justice and
intelligence-related databases.
[63] DHS, Privacy Impact Assessment for the TSA Operations Center
Incident Management System (Washington, D.C.: July 8, 2008), and
Privacy Impact Assessment for the Screening of Passengers by
Observation Techniques (SPOT) Program (Washington, D.C.: Aug. 5, 2008).
[64] DHS, Privacy Impact Assessment for the Transportation Information
Sharing System (Washington, D.C.: June 2008). The Transportation
Information Sharing System is a database owned by the TSA's FAMS
component; the data entered into it may be shared with other federal,
state, or local law enforcement and law enforcement support entities.
Federal air marshals file reports related to the observation of
suspicious activities and input this information, as well as incident
reports submitted by airline employees and other individuals within
the aviation domain, into the Transportation Information Sharing
System.
[65] Because the SPOT program has not been scientifically validated,
it cannot be determined if these anecdotal results were better than if
passengers had been pulled aside at random, rather than as a
consequence of being identified for further screening by BDOs.
[66] Some details about the process were deleted because TSA
considered them to be Sensitive Security Information.
[67] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1].
[68] Some details from our analysis were deleted because TSA
considered them to be Sensitive Security Information.
[69] This information can be submitted about individuals whose
suspicious activity resulted in BDO or LEO referral. See TSA's July
2008 Transportation Security Operations Center Privacy Impact
Assessment.
[70] Some details of our survey results were deleted because TSA
considered them to be Sensitive Security Information.
[71] See [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1].
For example, information should be recorded and communicated to
management and others within the entity who need it and in a form and
within a time frame that enables them to carry out their internal
control and other responsibilities. Further, effective information
technology management is critical to achieving useful, reliable, and
continuous recording and communication of information.
[72] TSA, Strategy Deployment, Organizational Business Plan, Office of
Security Operations, Fiscal Year 2010 (Washington, D.C.: April 2008).
According to TSA, the Office of Security Operations is the operational
arm of TSA and employs the largest TSA workforce. It is responsible
for airport checkpoint and baggage screening operations as well as
other special programs designed to secure all assigned transportation
modes.
[73] In March 2010, TSA told us that over the next 18 months, it will
expand access to information classified up to the "Secret" level to an
additional 10,000 TSA personnel, including all BDOs, all SPOT
Transportation Security Managers (who are responsible for the local
operations of the SPOT program and supervision of the BDOs), and all
Supervisory TSOs (who directly supervise TSOs and the screening
process).
[74] The Terrorist Screening Database is the central terrorist
watchlist consolidated by the FBI's Terrorist Screening Center and
used by multiple agencies to compile their specific watchlists and for
screening.
[75] The other databases available to TSA are omitted because TSA
considered them to be Sensitive Security Information.
[76] The No-Fly list is used to identify individuals who should be
prevented from boarding an aircraft. The No-Fly and Selectee lists
contain applicable records from the FBI's Terrorist Screening Center
consolidated database of known or suspected terrorists. Pursuant to
Homeland Security Presidential Directive 6, dated September 16, 2003,
the Terrorist Screening Center--operational since December 2003 under
the administration of the FBI--was established to develop and maintain
the U.S. government's consolidated terrorist screening database (the
watch list) and to provide for the use of watch-list records during
security-related screening processes. See GAO-08-136T, Aviation
Security: TSA Is Enhancing Its Oversight of Air Carrier Efforts to
Screen Passengers against Terrorist Watch-List Records, but Expects
Ultimate Solution to Be Implementation of Secure Flight (Washington,
D.C.: Sept. 9, 2008).
[77] According to DHS, the Terrorist Identity Datamark Environment is
the database maintained by the National Counterterrorism Center--the
primary organization in the U.S. government for integrating and
analyzing intelligence pertaining to terrorism and counterterrorism--
to serve as a central repository for all information on known or
suspected international terrorists with the exception of purely
domestic terrorism information. See, DHS, Office of Inspector General,
The DHS Process for Nominating Individuals to the Consolidated
Terrorist Watchlist (Washington, D.C.: February 2008).
[78] DHS, National Infrastructure Protection Plan: Partnering to
Enhance Protection and Resiliency (Washington, D.C.: 2009);
[hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]; and GAO,
Performance Measurement and Evaluation: Definitions and Relationships,
[hyperlink, http://www.gao.gov/products/GAO-05-739SP] (Washington,
D.C.: May 2005).
[79] The Project Management Institute, The Standard for Program
ManagementŠ (2006).
[80] GAO, Executive Guide: Effectively Implementing the Government
Performance and Results Act, [hyperlink,
http://www.gao.gov/products/GAO/GGD-96-118] (Washington, D.C.: June
1996).
[81] See S. Rep. No. 111-31, at 56-57 (2009); see also S. Rep. No. 110-
396, at 59 (2008).
[82] See H.R. Rep. No. 111-298, at 77 (2009) (Conf. Rep.). The report
further directs that GAO review the report submitted by TSA and
provide its findings to the committees no later than 120 days after
the SPOT report is submitted to the committees.
[83] Output measures help determine the extent to which an activity
was performed as planned. Outcome-related measures are more robust
measures because they provide a more comprehensive assessment of the
success of the agency's efforts, as stated in DHS's 2009 NIPP.
[84] OMB and the Council for Excellence in Government, Performance
Measurement Challenges and Strategies (Washington, D.C.: June 18,
2003).
[85] The Office's primary work involves metrics infrastructure; it
assists TSA programs, if requested, in developing applications to
track quantitative measures, such as surrendered items. It also tracks
data for its Management Objectives Report related to three areas:
employees, security effectiveness, and efficiency.
[86] The types of prohibited items found have included knives, guns,
gun ammunition, certain chemicals, strike-anywhere matches, and
certain liquids/gels/aerosols; other illegal items discovered include
narcotics and fraudulent identity documents.
[87] According to TSA, TSOs focus on detecting high-risk threats which
have the ability to cause catastrophic damage to an airplane in flight
(e.g., explosives).
[88] OMB and the Council for Excellence in Government, Performance
Measurement Challenges and Strategies (Washington, D.C.: June 18,
2003).
[89] We also found that the SPOT referral database had a number of
weaknesses. TSA designated our discussion of these weakness as
sensitive security information.
[90] Our estimate of the total number of passengers who went through
checkpoints is based on Bureau of Transportation Statistics data that
we obtained for the airports at which SPOT was deployed during this
period. Some figures were rounded.
[91] The analysis included only flights leaving the United States.
Department of Justice data show that more than 400 individuals have
been convicted in the United States for terrorism-related offenses
since September 11, 2001. We did not examine the travel itineraries of
all these individuals.
[92] The events included the Mumbai, India attack of 2008; a plot to
attack the Quantico, Virginia, Marine base in 2008; an effort by five
Americans to receive training and fight in Pakistan in December 2009;
a plot to attack infrastructure in New York City in 2009; an effort to
provide men and support for terrorists in Somalia in 2008; and an
attack on a U.S. base in Afghanistan by an American who received
training in Pakistan. We were unable to confirm whether BDOs were
stationed at the checkpoints used by these individuals at the time
they traveled.
[93] See, GAO, Supply Chain Security: Feasibility and Cost-Benefit
Analysis Would Assist DHS and Congress in Assessing and Implementing
the Requirement to Scan 100 Percent of U.S.-Bound Containers,
[hyperlink, http://www.gao.gov/products/GAO-10-12] (Washington, D.C.:
Oct. 30, 2009). The Project Management Institute, A Guide to the
Project Management Body of Knowledge.
[94] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1].
[95] G-Band, or Expert BDOs, have advanced to a lead role, are able to
provide technical expertise on the SPOT program, and are one band away
from a supervisory role.
[96] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1].
[97] TSA's Operational and Technical Training Division, within the
Office of Security Operations, provides assistance with development
and implementation of technical training for screening, Behavior
Detection Officers, Bomb Appraisal Officers, the Aviation Direct
Access Screening Program, and technical management training.
[98] In May 2009, the title of the course was changed to "Additional
Behavior Detection Techniques" because ABDT is actually a supplemental
tool for BDOs to use during the Casual Conversation phase of SPOT
Referral Screening. The course was formerly titled "Advanced Behavior
Detection Techniques." Microfacial expressions are very brief facial
expressions that can last as little as 1/25 of a second.
[99] GAO, Human Capital: A Guide for Assessing Strategic Training and
Development Efforts in the Federal Government, [hyperlink,
http://www.gao.gov/products/GAO-04-546G] (Washington, D.C.: Mar. 1,
2004).
[100] The TSA staff member attended the following external training
courses: John Reid and Associates' Reid Techniques of Interrogation
and Advanced Reid Techniques of Interrogation; Massachusetts State
Police Academy's Basic Investigations and Professional Development
Program Interview Techniques; International Security Defense Systems'
Verification Agent for Virgin Atlantic Security Systems; New Mexico
Technology, Materials and Research Center's Prevention and Response to
Suicide Bomber Indicators; Abraxis Corporation's Detecting Deception
and Eliciting Response; Langevin Learning Services' Instructional
Techniques for New Instructors; Ekman Group's Understanding Emotions
and Detecting Truth; Chameleon Associates' Suspicious Behavior
Detection; and Federal Transit Administration's Terrorist Awareness,
Recognition, and Response.
[101] The purpose of the SPOT working group was to help refine the
list of SPOT behaviors and to develop standard operating procedures
and a concept of operations for the program.
[102] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] and
[hyperlink, http://www.gao.gov/products/GAO-04-546G].
[103] [hyperlink, http://www.gao.gov/products/GAO-04-546G].
[104] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1].
[105] The SPOT program retains the discretion to train BDOs at a site
other than one of the five training facilities if it is more fiscally
responsible to do so. For example, if there are 15 BDO candidates at a
single airport, the SPOT program will train them at that airport
rather than sending them to a training facility.
[106] Some SPOT instructors have been evaluated multiple times. While
the SPOT program office provided us with print or electronic copies of
all SPOT instructor evaluations, some forms contained only numeric
ratings and no written comments; others had no numeric scores. Because
instructor names were redacted from the evaluations, the numbers may
include duplicates. Additionally, the evaluations containing written
comments were not always filled out using complete sentences, making
it difficult to ascertain the rater's assessment of the instructor.
[107] SPOT Instructors are evaluated using a Quality Assurance
Instructor Evaluation, TSA Form 1909. Using this form, the evaluator
assigns either 0 (zero) points, 0.5 points, or 1 point for each of 57
ratable items depending on whether the instructor meets the standard
as written, needs improvement to meet the standard, or does not meet
the standard. The total points are then entered into a formula that
generates a percentage. This percentage is used to determine the
overall rating. Instructors receiving a score of 95 percent to 100
percent are rated as exceeds expectations; 85 percent to 94 percent
are rated as meets expectations; 75 percent to 84 percent are rated as
needs improvement; and 0 percent to 74 percent are rated as
unsatisfactory.
[108] [hyperlink, http://www.gao.gov/products/GAO-04-546G].
[109] TSA, Operational and Technical Training Division, Training
Standards (Sept. 28, 2006).
[110] The Project Management Institute, The Standard for Program
ManagementŠ (2006).
[111] The training gap analysis identifies gaps in the training
curriculum.
[112] This estimate assumes that there would be no further increases
for SPOT over the next 5 years above the requested $232 million level
for fiscal year 2011. However, to stay even with inflation, the
allocation would likely increase somewhat each year.
[113] The National Research Council is a component of the National
Academy of Sciences, a part of a private, nonprofit institution, the
National Academies, which provide science, technology, and health
policy advice under a congressional charter.
[114] National Research Council, Protecting Individual Privacy in the
Struggle Against Terrorists: A Framework for Assessment (Washington,
D.C.: National Academies Press, 2008). The report's preparation was
overseen by the National Academy of Science's 21-member Committee on
Technical and Privacy Dimensions of Information for Terrorism
Prevention and Other National Goals. We reviewed the approach used and
the information provided in this study and found the study to be
credible for our purposes. The contributors included recognized
experts across a variety of fields, including William J. Perry, former
Secretary of Defense, and Dr. Tara O'Toole, then-CEO and Director of
the Center for Biosecurity of the University of Pittsburgh Medical
Center, Professor of Medicine and of Public Health at the University
of Pittsburgh. (Dr. O'Toole was subsequently nominated and confirmed
as the Under Secretary of the DHS Science and Technology Directorate.)
[115] National Research Council, Protecting Individual Privacy in the
Struggle Against Terrorists: A Framework for Assessment (Washington,
D.C.: National Academies Press, 2008). The report's preparation was
overseen by the NAS's 21-member Committee on Technical and Privacy
Dimensions of Information for Terrorism Prevention and Other National
Goals. We reviewed the approach used and the information provided in
this study and found the study to be credible for our purposes. The
contributors included recognized experts across a variety of fields,
including William J. Perry, former Secretary of Defense, and Dr. Tara
O'Toole, then-CEO and Director of the Center for Biosecurity of the
University of Pittsburgh Medical Center, Professor of Medicine and of
Public Health at the University of Pittsburgh. (Dr. O'Toole was
subsequently nominated and confirmed as the Under Secretary of DHS's
Science and Technology Directorate. The National Research Council is a
component of the National Academy of Sciences, a part of a private,
nonprofit institution, the National Academies, which provide science,
technology, and health policy advice under a congressional charter
[116] Dr. Ekman is professor emeritus of psychology at the University
of California Medical School, San Francisco, and is considered one of
the world's foremost experts on facial expressions. His books include:
Emotions Revealed: Recognizing Faces and Feelings to Improve
Communications and Emotional Life (New York: Holt and Company, 2003);
Emotion in the Human Face (New York: Pergamon Press, 1972); Unmasking
the Face: A guide to Recognizing Emotions from Facial Clues (Englewood
Cliffs, N.J.: Prentice-Hall, 1975). Dr. Ekman has published more than
100 articles.
[117] Dr. Frank is Associate Professor, Department of Communication,
College of Arts and Sciences, at the University at Buffalo, State
University of New York. He is on the Advisory Board of the
University's Center for Unified Biometrics and Sensors, and has
conducted research supported by DHS, the Defense Advanced Research
Projects Agency, and the National Science Foundation.
[118] Dr. Givens is the director of the nonprofit Center for Nonverbal
Studies, in Spokane, Washington. He is the author of Love Signals: A
Practical Field Guide to the Body Language of Courtship (St. Martin's,
New York, 2005) and Crime Signals: How to Spot a Criminal Before You
Become a Victim (St. Martin's, 2008). The Center's Web site links to
Dr. Givens' reference tool, The Nonverbal Dictionary of Gestures,
Signs and Body Language Cues.
[119] Dr. Matsumoto is a Professor, Department of Psychology at San
Francisco State University, and is an associate of Dr. Ekman.
[120] Dr. Wein is the Paul E. Holden Professor of Management Science
at the Graduate School of Business, Stanford University. His homeland
security-related work includes four papers in Proceedings of the
National Academy of Sciences, on an emergency response to a smallpox
attack, an emergency response to an anthrax attack, a biometric
analysis of the US-VISIT Program, and an analysis of a bioterror
attack on the milk supply.
[121] DHS, Cost Benefit Analysis Guidebook (Washington, D.C.: February
2006).
[122] Office of Management and Budget (OMB), Circular No. A-94,
Guidelines and Discount Rates for Benefit-Cost Analysis of Federal
Programs (Washington, D.C.: October 1992); and Circular A-4,
Regulatory Analysis (Washington, D.C.: September 2003).
[123] [hyperlink, http://www.gao.gov/products/GAO-04-408T].
[124] Pub. L. No. 103-62, 107 Stat. 285 (1993).
[125] TSA, Screening of Passengers by Observation Technique (SPOT)
White Paper for the Department of Homeland Security (Washington, D.C.:
Feb. 22, 2005).
[126] The data from interviews of suspicious passengers by FAMS are
inputted into the Transportation Information Sharing System, as are
reports sent to FAMS from airline employees about suspicious
passengers.
[127] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1].
[128] GAO, Transportation Security: Comprehensive Risk Assessments and
Stronger Internal Controls Needed to Help Inform TSA Resource
Allocation, [hyperlink, http://www.gao.gov/products/GAO-09-492]
(Washington, D.C.: Mar. 27, 2009).
[129] Federal Security Directors are the highest ranking TSA security
officials at U.S. airports; Assistant Federal Security Directors are
their assistants. Both are responsible for all aspects of security at
airports, including coordination with federal and nonfederal law
enforcement entities operating at airports, such as FAMS, the Drug
Enforcement Administration, and CBP.
[130] The Current Airports Threat Assessment is a threat estimate
designed to provide a snapshot of the current terrorist threat to
airports in the United States as well as for major international
airports serving as last points of departure for U.S. airlines.
[131] The Treasury Enforcement Communication System was designed to
provide controlled access to a large database of information about
suspects and to interface with a number of other law enforcement
systems. These capabilities are provided to users through various
applications, including the Inspection/Interagency Border Inspection
System applications that facilitate passenger processing through the
implementation of innovative border control technology.
[132] GAO, Human Capital: A Guide for Assessing Strategic Training and
Development Efforts in the Federal Government, [hyperlink,
http://www.gao.gov/products/GAO-04-546G] (Washington, D.C.: Mar. 1,
2004).
[133] Includes TSA's Office of Civil Rights, Office of Chief Counsel,
and Privacy Office, and DHS's Policy Office and Transportation
Security Laboratory.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
