Aviation Security
Screener Training and Performance Measurement Strengthened, but More Work Remains Gao ID: GAO-05-457 May 2, 2005The screening of airport passengers and their checked baggage is a critical component in securing our nation's commercial aviation system. Since May 2003, GAO has issued six products related to screener training and performance. This report updates the information presented in the prior products and incorporates results from GAO's survey of 155 Federal Security Directors--the ranking Transportation Security Administration (TSA) authority responsible for the leadership and coordination of TSA security activities at the nation's commercial airports. Specifically, this report addresses (1) actions TSA has taken to enhance training for passenger and checked baggage screeners and screening supervisors, (2) how TSA ensures that screeners complete required training, and (3) actions TSA has taken to measure and enhance screener performance in detecting threat objects.
TSA has initiated a number of actions designed to enhance screener training, such as updating the basic screener training course. TSA also established a recurrent training requirement and introduced the Online Learning Center, which makes self-guided training courses available over TSA's intranet and the Internet. Even with these efforts, Federal Security Directors reported that insufficient screener staffing and a lack of high-speed Internet/intranet connectivity at some training facilities have made it difficult to fully utilize training programs and to meet the recurrent training requirement of 3 hours per week, averaged over a quarter year, within regular duty hours. TSA acknowledged that challenges exist in recurrent training delivery and is taking steps to address these challenges, including factoring training into workforce planning efforts and distributing training through written materials and CD-ROMs. However, TSA has not established a plan prioritizing the deployment of high-speed Internet/intranet connectivity to all airport training facilities to facilitate screener access to training materials. TSA lacks adequate internal controls to provide reasonable assurance that screeners receive legislatively mandated basic and remedial training, and to monitor its recurrent training program. Specifically, TSA policy does not clearly specify the responsibility for ensuring that screeners have completed all required training. In addition, TSA officials have no formal policies or methods for monitoring the completion of required training and were unable to provide documentation identifying the completion of remedial training. TSA has implemented and strengthened efforts to measure and enhance screener performance. For example, TSA has increased the number of covert tests it conducts at airports, which test screeners' ability to detect threat objects on passengers, in their carry-on baggage, and in checked baggage. These tests identified that overall, weaknesses and vulnerabilities continue to exist in passenger and checked baggage screening systems at airports of all sizes, at airports with federal screeners, and at airports with private-sector screeners. While these test results are an indicator of performance, they cannot solely be used as a comprehensive measure of any airport's screening performance or any individual screener's performance. We also found that TSA's efforts to measure and enhance screener performance have primarily focused on passenger screening, not checked baggage screening. For example, TSA only uses threat image software on passenger screening X-ray machines, and the recertification testing program does not include an image recognition module for checked baggage screeners. TSA is taking steps to address the overall imbalance in passenger and checked baggage screening performance data. TSA also established performance indexes for the passenger and checked baggage screening systems, to identify an overall desired level of performance. However, TSA has not established performance targets for each of the component indicators that make up the performance indexes, including performance targets for covert testing. TSA plans to finalize these targets by the end of fiscal year 2005.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-05-457, Aviation Security: Screener Training and Performance Measurement Strengthened, but More Work Remains
This is the accessible text file for GAO report number GAO-05-457
entitled 'Aviation Security: Screener Training and Performance
Measurement Strengthened, but More Work Remains' which was released on
May 2, 2005.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Chairman, Subcommittee on Aviation, Committee on
Transportation and Infrastructure, House of Representatives:
United States Government Accountability Office:
GAO:
May 2005:
Aviation Security:
Screener Training and Performance Measurement Strengthened, but More
Work Remains:
GAO-05-457:
GAO Highlights:
Highlights of GAO-05-457, a report to the Chairman, Subcommittee on
Aviation, Committee on Transportation and Infrastructure, House of
Representatives:
Why GAO Did This Study:
The screening of airport passengers and their checked baggage is a
critical component in securing our nation‘s commercial aviation system.
Since May 2003, GAO has issued six products related to screener
training and performance. This report updates the information presented
in the prior products and incorporates results from GAO‘s survey of 155
Federal Security Directors”the ranking Transportation Security
Administration (TSA) authority responsible for the leadership and
coordination of TSA security activities at the nation‘s commercial
airports. Specifically, this report addresses (1) actions TSA has taken
to enhance training for passenger and checked baggage screeners and
screening supervisors, (2) how TSA ensures that screeners complete
required training, and (3) actions TSA has taken to measure and enhance
screener performance in detecting threat objects.
What GAO Found:
TSA has initiated a number of actions designed to enhance screener
training, such as updating the basic screener training course. TSA also
established a recurrent training requirement and introduced the Online
Learning Center, which makes self-guided training courses available
over TSA‘s intranet and the Internet. Even with these efforts, Federal
Security Directors reported that insufficient screener staffing and a
lack of high-speed Internet/intranet connectivity at some training
facilities have made it difficult to fully utilize training programs
and to meet the recurrent training requirement of 3 hours per week,
averaged over a quarter year, within regular duty hours. TSA
acknowledged that challenges exist in recurrent training delivery and
is taking steps to address these challenges, including factoring
training into workforce planning efforts and distributing training
through written materials and CD-ROMs. However, TSA has not established
a plan prioritizing the deployment of high-speed Internet/intranet
connectivity to all airport training facilities to facilitate screener
access to training materials.
TSA lacks adequate internal controls to provide reasonable assurance
that screeners receive legislatively mandated basic and remedial
training, and to monitor its recurrent training program. Specifically,
TSA policy does not clearly specify the responsibility for ensuring
that screeners have completed all required training. In addition, TSA
officials have no formal policies or methods for monitoring the
completion of required training and were unable to provide
documentation identifying the completion of remedial training.
TSA has implemented and strengthened efforts to measure and enhance
screener performance. For example, TSA has increased the number of
covert tests it conducts at airports, which test screeners‘ ability to
detect threat objects on passengers, in their carry-on baggage, and in
checked baggage. These tests identified that overall, weaknesses and
vulnerabilities continue to exist in passenger and checked baggage
screening systems at airports of all sizes, at airports with federal
screeners, and at airports with private-sector screeners. While these
test results are an indicator of performance, they cannot solely be
used as a comprehensive measure of any airport‘s screening performance
or any individual screener‘s performance. We also found that TSA‘s
efforts to measure and enhance screener performance have primarily
focused on passenger screening, not checked baggage screening. For
example, TSA only uses threat image software on passenger screening X-
ray machines, and the recertification testing program does not include
an image recognition module for checked baggage screeners. TSA is
taking steps to address the overall imbalance in passenger and checked
baggage screening performance data. TSA also established performance
indexes for the passenger and checked baggage screening systems, to
identify an overall desired level of performance. However, TSA has not
established performance targets for each of the component indicators
that make up the performance indexes, including performance targets for
covert testing. TSA plans to finalize these targets by the end of
fiscal year 2005.
What GAO Recommends:
GAO is recommending that the Secretary of Homeland Security direct TSA
to develop a plan for completing the deployment of high-speed
connectivity at airport training facilities, and establish and
communicate appropriate internal controls for monitoring the completion
of training.
TSA reviewed a draft of this report and generally agreed with GAO‘s
findings and recommendations.
www.gao.gov/cgi-bin/getrpt?GAO-05-457.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Cathleen Berrick, 202-512-
8777, berrickc@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
TSA Has Enhanced and Expanded Training, but Some Screeners Have
Encountered Difficulty Accessing and Completing Recurrent Training:
TSA Lacks Adequate Controls to Provide Reasonable Assurance That
Screeners Receive Required Training:
Progress Has Been Made in Implementing Tools to Measure and Enhance
Screener Performance, but Key Performance Targets Have Not Been
Finalized:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Summary of Previous Findings Related to Screener Training
and Performance:
Appendix II: Objectives, Scope, and Methodology:
Appendix III: TSA Screener Training Tools Designed to Help Improve
Screener Performance:
Appendix IV: Summary of TSA's Short-Term Action Items for Strengthening
Passenger Screener Performance:
Appendix V: Comments from the Department of Homeland Security:
Appendix VI: GAO Contacts and Acknowledgments:
Tables:
Table 1: Categories of Required Training Provided to TSA Screeners:
Table 2: Training Needed to Enhance Screener Supervision, as Reported
by FSDs:
Table 3: Checkpoint and Checked Baggage Tests Conducted by OIAPR,
September 10, 2002-September 30, 2004:
Table 4: Modules Included in Recertification Knowledge and Skills
Assessment:
Table 5: Screener Recertification Module Testing Percentage Pass Rates,
October 2003--March 2004:
Table 6: Recertification Testing Modules by Screening Function:
Table 7: Components of TSA's Performance Indexes:
Figures:
Figure 1: Commercial Airports by Airport Security Category as of
December 2003:
Figure 2: Passenger Checkpoint and Checked Baggage Screening
Operations:
Figure 3: Description of Screening-Related Positions:
Figure 4: Newly Hired Screeners Trained as Dual-Function Screeners
between April 1, 2004, and September 1, 2004:
Figure 5: Percentage of Airports Reported to Have High-Speed
Connectivity for Training Purposes as of October 2004:
Figure 6: Screener Recertification Results, October 2003-March 2004:
Abbreviations:
ATSA: Aviation and Transportation Security Act:
DHS: Department of Homeland Security:
DOT: Department of Transportation:
EDS: explosive detection system:
ETD: electronic trace detection:
FAA: Federal Aviation Administration:
FSD: Federal Security Director:
IED: improvised explosive device:
MBS: modular bomb set:
MTAT: Mobile Training Assist Team:
OIAPR: Office of Internal Affairs and Program Review:
OIG: Office of the Inspector General:
SIDA: security identification display area:
TIP: Threat Image Projection:
TSA: Transportation Security Administration:
United States Government Accountability Office:
Washington, DC 20548:
May 2, 2005:
The Honorable John Mica:
Chairman, Subcommittee on Aviation:
Committee on Transportation and Infrastructure:
House of Representatives:
Dear Mr. Chairman:
The screening of airport passengers and their checked baggage is a
critical component in securing our nation's commercial aviation system.
In an effort to strengthen the security of commercial aviation, the
President signed the Aviation and Transportation Security Act on
November 19, 2001. The act created the Transportation Security
Administration (TSA) and mandated actions designed to strengthen
aviation security, including requiring that TSA assume responsibility
for conducting passenger and checked baggage screening at over 450
commercial airports in the United States by November 19, 2002. It has
been over 2 years since TSA assumed this responsibility, and the agency
has spent billions of dollars and implemented a wide range of
initiatives to enhance its passenger and checked baggage screening
operations. Despite the attention to passenger and checked baggage
screening operations, however, concerns about the effectiveness of the
screening system remain. For example, covert testing conducted by TSA's
Office of Internal Affairs and Program Review and the Department of
Homeland Security's (DHS) Office of Inspector General identified
weaknesses in the ability of screeners to detect threat objects. (The
results of our analysis of TSA's covert testing data and test program
are included in a separate classified GAO report.)
To determine the progress TSA has made in strengthening its passenger
and checked baggage screening operations, the Subcommittee on Aviation,
House Committee on Transportation and Infrastructure, requested that we
examine TSA efforts to train screeners and to measure and enhance
screener performance. Since we began our work in May 2003, we have
issued six products that address issues related to screener training
and performance, including four to this Subcommittee (see app. I). This
report updates some of the information presented in our prior products.
In addition, it incorporates results from our surveys of 155 Federal
Security Directors (FSD).[Footnote 1] The surveys were designed to
obtain information related to, among other issues, TSA's efforts to
train screeners and supervisors and assess screener performance in
detecting threat objects. Specifically, this report addresses the
following questions: (1) What actions has TSA taken to enhance training
for passenger and checked baggage screeners and screening supervisors?
(2) How does TSA ensure that screeners complete required training? (3)
What actions has TSA taken to measure and enhance screener performance
in detecting threat objects?
In conducting our work, we reviewed TSA documentation related to
screener training requirements and performance testing. We also
analyzed data from our survey responses from all 155 FSDs about
screener training, supervision and performance. We also visited 29
airports of various sizes and geographic locations to obtain a cross-
section of all airports, including the 5 airports with private-sector
screeners.[Footnote 2] To gain a better understanding of training and
performance issues, during these visits, we interviewed FSDs, members
of their management teams, passenger and checked baggage screeners, and
airport officials. However, information obtained during these visits
cannot be generalized to all airports across the nation. Additionally,
we interviewed officials at TSA headquarters and TSA's transportation
security laboratory about their experiences with training and screener
performance. We compared TSA practices and procedures for monitoring
completion of training with the Comptroller General's Standards for
Internal Control in the Federal Government. We assessed the reliability
of the data we acquired from TSA regarding screener testing and
training completion, and found the data to be sufficiently reliable for
our purposes. A more detailed description of our scope and methodology
is contained in appendix II.
We conducted our work from May 2003 through April 2005 in accordance
with generally accepted government auditing standards.
Results in Brief:
TSA has initiated a number of actions designed to enhance passenger
screener, checked baggage screener, and screener supervisor training.
However, at some airports screeners encountered difficulty accessing
and completing recurrent (refresher) training because of technological
and staffing constraints. Among the actions TSA has taken to enhance
training are changes and updates to the basic training program. For
example, TSA added a modular dual-function course during basic training
that covers passenger and checked baggage screening functions and
allows newly hired screeners to perform either function upon completion
of the training. TSA also established a requirement for recurrent
screener training and developed and introduced the Online Learning
Center, which makes self-guided training courses available to employees
over TSA's intranet and the Internet. In addition, TSA has provided
Federal Security Directors with hands-on training tools to use for
local recurrent training and testing. Finally, TSA has taken steps to
provide leadership and technical training to Screening Supervisors.
Despite these improvements, some Federal Security Directors, in
response to open-ended survey questions, identified a desire for more
training in specific areas, including leadership, communication, and
supervision. Further in survey responses, Federal Security Directors
reported that largely because of insufficient screener staffing,
screeners were not always able to meet the recurrent training
requirement within regular duty hours. Federal Security Directors at
some airports also reported ongoing problems with a lack of high-speed
Internet/intranet connectivity, which severely limited screener access
to the Online Learning Center and diminished its value as a learning
tool. As of October 2004, nearly half of the screener workforce did not
have high-speed access to the Online Learning Center at their training
facility. TSA has acknowledged that challenges exist in recurrent
screener training delivery and is taking steps to address these
challenges, including factoring training requirements into workforce
planning efforts and distributing training through written materials
and CD-ROMs until full Internet/intranet connectivity is achieved.
However, TSA does not have a plan for prioritizing and scheduling the
deployment of high-speed connectivity to all airport training
facilities once funding is available. The absence of such a plan limits
TSA's ability to make prudent decisions about how to move forward with
deploying connectivity to all airports to provide screeners access to
online training.
TSA lacks adequate internal controls to provide reasonable assurance
that screeners are receiving legislatively mandated basic and remedial
training, and to monitor the status of its recurrent training program.
The Comptroller General's Standards for Internal Control call for (1)
areas of authority and responsibility to be clearly defined and
appropriate lines of reporting established, (2) transactions and other
significant events to be documented clearly and documentation to be
readily available for examination, and (3) controls generally to be
designed to ensure that ongoing monitoring occurs in the course of
normal operations.[Footnote 3] In addition, the standards also require
that information be communicated within an organization to enable
individuals to carry out their internal control responsibilities.
However, our review of TSA's training program noted weaknesses in each
of these control areas. First, TSA policy does not clearly define the
responsibility for ensuring that screeners have completed all required
training. Second, TSA officials were unable to produce documentation of
remedial training completion for our examination. Third, TSA has no
formal policies for monitoring the completion of required training.
Moreover, because of the lack of high-speed Internet/intranet
connectivity at some airports, staff had to manually input training
data that would otherwise be recorded automatically by the Online
Learning Center, making it challenging for some airports to keep
accurate and up-to-date training records. TSA headquarters officials
acknowledged that it can be difficult for airports to keep the Online
Learning Center up-to-date with the most recent training records
without high-speed connectivity.
TSA has improved its efforts to measure and enhance screener
performance. In September 2003, we first reported on the need for TSA
to strengthen its efforts to measure and enhance screener performance.
At that time, TSA had collected limited data on screener performance.
Specifically, limited covert testing--unannounced undercover tests in
which TSA agents attempt to pass threat objects through screening
checkpoints and in checked baggage--had been performed, the Threat
Image Projection system was not fully operational,[Footnote 4] and TSA
had not fully implemented the annual screener recertification
program.[Footnote 5] Since then, TSA has implemented and strengthened
efforts to collect screener performance data as part of its overall
effort to enhance screener performance. For example, TSA has increased
the number of covert tests it conducts at airports. These tests have
identified that overall, weaknesses and vulnerabilities continue to
exist in the passenger and checked baggage screening systems at
airports of all sizes, at airports with federal screeners, and at
airports with private-sector screeners. While these test results are an
indicator of screener performance, they cannot solely be used as a
comprehensive measure of any airport's screening performance or any
individual screener's performance. TSA has also implemented other
efforts to measure and enhance screener performance. However, these
efforts have primarily focused on passenger screening, not checked
baggage screening. Specifically, the Threat Image Projection system is
only available for passenger screening; the recertification testing
program does not include an image recognition module for checked
baggage screeners; and the screener performance improvement study
focused solely on passenger screeners. TSA is taking steps to address
the overall imbalance in passenger and baggage screening performance
data, including working toward implementing the Threat Image Projection
system for checked baggage screening and developing an image
recognition module for checked baggage screener recertification. TSA
has also implemented a number of other improvements, which it
identified in a passenger screener performance improvement study and
incorporated into a screener performance improvement plan. Furthermore,
TSA has established two performance indexes for the screening systems-
-one for passenger and one for checked baggage screening.[Footnote 6]
These indexes measure overall performance through a composite of
component indicators. However, TSA has not established performance
targets for each of the component indicators--such as covert testing--
which would allow it to draw more meaningful conclusions about its
performance and most effectively direct its improvement efforts.
Although TSA has not yet established performance targets for each of
the component indicators, TSA plans to finalize performance targets for
the indicators by the end of fiscal year 2005.
Certain information we obtained and analyzed regarding screener
training and performance is classified or is considered by TSA to be
sensitive security information. Accordingly, the results of our review
of this information are not included in this report.[Footnote 7]
To help ensure access to and completion of required training, we are
making recommendations to the Secretary of the Department of Homeland
Security to direct the Assistant Secretary, Transportation Security
Administration, to (1) develop a plan for completing the deployment of
high-speed Internet/intranet connectivity to all TSA's airport training
facilities and (2) establish appropriate responsibilities and other
internal controls for monitoring and documenting screener compliance
with training requirements.
We provided a draft copy of this report to DHS for review. DHS, in its
written comments, generally concurred with the findings and
recommendations in the report and agreed that efforts to implement our
recommendations are critical to successful passenger and checked
baggage screening training and performance. DHS described some actions
TSA had taken or planned to take to implement these recommendations.
DHS also stated that TSA had already developed a plan for prioritizing
and scheduling the deployment of high-speed Internet/intranet
connectivity to all TSA airport training facilities. However, although
we requested a copy of the plan several times during our review and
after receiving written comments from DHS, TSA did not provide us with
a copy of this plan. Therefore, we cannot assess the extent to which
the plan DHS referenced in its written comments fulfills our
recommendation. Additionally, DHS stated that it is taking steps to
define responsibilities for monitoring the completion of required
training, and to insert this accountability into performance plans of
all TSA supervisors. The full text of DHS's comments is included in
appendix V.
Background:
The performance of passenger and checked baggage screeners in detecting
threat objects at the nation's airports has been a long-standing
concern. In 1978, screeners failed to detect 13 percent of the
potentially dangerous objects that Federal Aviation Administration
(FAA) agents carried through airport screening checkpoints during
tests. In 1987, screeners did not detect 20 percent of the objects in
similar tests. In tests conducted during the late 1990s, as the testing
objects became more realistic, screeners' abilities to detect dangerous
objects declined further. In April 2004, we, along with the DHS Office
of the Inspector General (OIG), testified that the performance of
screeners continued to be a concern. More recent tests conducted by
TSA's Office of Internal Affairs and Program Review (OIAPR) also
identified weaknesses in the ability of screeners to detect threat
objects, and separate DHS OIG tests identified comparable screener
performance weaknesses. In its July 2004 report, The National
Commission on Terrorist Attacks Upon the United States, known widely as
the 9/11 Commission, also identified the need to improve screener
performance and to better understand the reasons for performance
problems.[Footnote 8]
After the terrorist attacks of September 11, 2001, the President signed
the Aviation and Transportation Security Act (ATSA) into law on
November 19, 2001, with the primary goal of strengthening the security
of the nation's aviation system. ATSA created TSA as an agency with
responsibility for securing all modes of transportation, including
aviation.[Footnote 9] As part of this responsibility, TSA oversees
security operations at the nation's more than 450 commercial airports,
including passenger and checked baggage screening operations. Prior to
the passage of ATSA, air carriers were responsible for screening
passengers and checked baggage, and most used private security firms to
perform this function. FAA was responsible for ensuring compliance with
screening regulations.
Today, TSA security activities at airports are overseen by FSDs. Each
FSD is responsible for overseeing security activities, including
passenger and checked baggage screening, at one or more commercial
airports. TSA classifies the over 450 commercial airports in the United
States into one of five security risk categories (X, I, II, III, and
IV) based on various factors, such as the total number of takeoffs and
landings annually, the extent to which passengers are screened at the
airport, and other special security considerations. In general,
category X airports have the largest number of passenger boardings and
category IV airports have the smallest. TSA periodically reviews
airports in each category and, if appropriate, updates airport
categorizations to reflect current operations. Figure 1 shows the
number of commercial airports by airport security category as of
December 2003.
Figure 1: Commercial Airports by Airport Security Category as of
December 2003:
[See PDF for image]
Note: TSA periodically reviews and updates airport categories to
reflect current operations. We used the categories in place in December
2003 to conduct our analyses during this review.
[End of figure]
In addition to establishing TSA and giving it responsibility for
passenger and checked baggage screening operations, ATSA set forth
specific enhancements to screening operations for TSA to implement,
with deadlines for completing many of them. These requirements included:
* assuming responsibility for screeners and screening operations at
more than 450 commercial airports by November 19, 2002;
* establishing a basic screener training program composed of a minimum
of 40 hours of classroom instruction and 60 hours of on-the-job
training;
* conducting an annual proficiency review of all screeners;
* conducting operational testing of screeners;[Footnote 10]
* requiring remedial training for any screener who fails an operational
test; and:
* screening all checked baggage for explosives using explosives
detection systems by December 31, 2002.[Footnote 11]
Passenger screening is a process by which authorized TSA personnel
inspect individuals and property to deter and prevent the carriage of
any unauthorized explosive, incendiary, weapon, or other dangerous item
aboard an aircraft or into a sterile area.[Footnote 12] Passenger
screeners must inspect individuals for prohibited items at designated
screening locations.[Footnote 13] The four passenger screening
functions are:
* X-ray screening of property,
* walk-through metal detector screening of individuals,
* hand-wand or pat-down screening of individuals, and:
* physical search of property and trace detection for explosives.
Checked baggage screening is a process by which authorized security
screening personnel inspect checked baggage to deter, detect, and
prevent the carriage of any unauthorized explosive, incendiary, or
weapon onboard an aircraft. Checked baggage screening is accomplished
through the use of explosive detection systems[Footnote 14] (EDS) or
explosive trace detection (ETD) systems,[Footnote 15] and through the
use of alternative means, such as manual searches, K-9 teams, and
positive passenger bag match,[Footnote 16] when EDS and ETD systems are
unavailable on a temporary basis. Figure 2 provides an illustration of
passenger and checked baggage screening operations.
Figure 2: Passenger Checkpoint and Checked Baggage Screening Operations:
[See PDF for image]
[End of figure]
There are several positions within TSA for employees that perform and
directly supervise passenger and checked baggage screening functions.
Figure 3 provides a description of these positions.
Figure 3: Description of Screening-Related Positions:
[See PDF for image]
[End of figure]
To prepare screeners to perform screening functions, to keep their
skills current, and to address performance deficiencies, TSA provides
three categories of required screener training. Table 1 provides a
description of the required training.
Table 1: Categories of Required Training Provided to TSA Screeners:
Category of training: Basic training;
Description of training requirement: ATSA requires newly hired
screeners to complete basic training composed of a minimum of 40 hours
of classroom instruction and 60 hours of on-the-job training before
they independently perform screening functions.
Category of training: Recurrent training;
Description of training requirement: TSA policy requires screeners to
complete 3 hours of skills refresher training per week averaged over
each quarter.
Category of training: Remedial training;
Description of training requirement: ATSA requires screeners who fail
any operational test to complete remedial training on the function they
failed before they resume performing that function.
Source: ATSA and TSA.
[End of table]
In September 2003, we reported on our preliminary observations of TSA's
efforts to ensure that screeners were effectively trained and
supervised and to measure screener performance.[Footnote 17] We found
that TSA had established and deployed a basic screener training program
and required remedial training but had not fully developed or deployed
a recurrent training program for screeners or supervisors. We also
reported that TSA had collected limited data to measure screener
performance. Specifically, TSA had conducted limited covert testing,
the Threat Image Projection System was not fully operational, and TSA
had not implemented the annual screener proficiency testing required by
ATSA. In subsequent products, we reported progress TSA had made in
these areas and challenges TSA continued to face in making training
available to screeners and in measuring and enhancing screener
performance. A summary of our specific findings is included in appendix
I.
TSA Has Enhanced and Expanded Training, but Some Screeners Have
Encountered Difficulty Accessing and Completing Recurrent Training:
TSA has taken a number of actions to enhance the training of screeners
and Screening Supervisors but has encountered difficulties in providing
access to recurrent training. TSA has enhanced basic training by, among
other things, adding a dual-function (passenger and checked baggage)
screening course for new employees. Furthermore, in response to the
need for frequent and ongoing training, TSA has implemented an Online
Learning Center with self-guided training courses available to
employees over TSA's intranet and the Internet and developed and
deployed a number of hands-on training tools.[Footnote 18] Moreover,
TSA now requires screeners to participate in 3 hours of recurrent
training per week, averaged over each quarter year. TSA has also
implemented leadership and technical training programs for Screening
Supervisors. However, some FSDs, in response to open-ended survey
questions, identified a desire for more training in specific areas,
including leadership, communication, and supervision. Further, despite
the progress TSA has made in enhancing and expanding screener and
supervisory training, TSA has faced challenges in providing access to
recurrent training. FSDs reported that insufficient staffing and a lack
of high-speed Internet/intranet connectivity at some training
facilities have made it difficult to fully utilize these programs and
to meet training requirements. TSA has acknowledged that challenges
exist in recurrent screener training delivery and is taking steps to
address these challenges, including factoring training requirements
into workforce planning efforts and distributing training through
written materials and CD-ROMs until full Internet/intranet connectivity
is achieved. However, TSA does not have a plan for prioritizing and
scheduling the deployment of high-speed connectivity to all airport
training facilities once funding is available. The absence of such a
plan limits TSA's ability to make prudent decisions about how to move
forward with deploying connectivity to all airports to provide
screeners access to online training.
TSA Has Enhanced Basic Screener Training:
TSA has enhanced its basic screener training program by updating the
training to reflect changes to standard operating procedures, deploying
a new dual-function (passenger and checked baggage screening) basic
training curriculum, and allowing the option of training delivery by
local staff. As required by ATSA, TSA established a basic training
program for screeners composed of a minimum of 40 hours of classroom
instruction and 60 hours of on-the-job training. TSA also updated the
initial basic screener training courses at the end of 2003 to
incorporate changes to standard operating procedures and directives,
which contain detailed information on how to perform TSA-approved
screening methods. However, a recent study by the DHS OIG found that
while incorporating the standard operating procedures into the
curricula was a positive step, a number of screener job tasks were
incompletely addressed in or were absent from the basic training
courses.[Footnote 19]
In addition to updates to the training curriculum, in April 2004, TSA
developed and implemented a new basic screener training program, dual-
function screener training that covers the technical aspects of both
passenger and checked baggage screening. Initially, new hire basic
training was performed by a contractor and provided a screener with
training in either passenger or checked baggage screening functions. A
screener could then receive basic training in the other function later,
at the discretion of the FSD, but could not be trained in both
functions immediately upon hire. The new dual-training program is
modular in design. Thus, FSDs can chose whether newly hired screeners
will receive instruction in one or both of the screening functions
during the initial training. In addition, the individual modules can
also be used to provide recurrent training, such as refreshing checked
baggage screening skills for a screener who has worked predominately as
a passenger screener. TSA officials stated that this new approach
provides the optimum training solution based on the specific needs of
each airport and reflects the fact that at some airports the FSD does
not require all screeners to be fully trained in both passenger and
checked baggage screening functions.
Some FSDs, particularly those at smaller airports, have made use of the
flexibility offered by the modular design of the new course to train
screeners immediately upon hire in both passenger and checked baggage
screening functions. Such training up front allows FSDs to use
screeners for either the passenger or the checked baggage screening
function, immediately upon completion of basic training. Figure 4 shows
that 58 percent (3,324) of newly hired screeners trained between April
1, 2004, and September 1, 2004, had completed the dual-function
training.
Figure 4: Newly Hired Screeners Trained as Dual-Function Screeners
between April 1, 2004, and September 1, 2004:
[See PDF for image]
[End of figure]
In April 2004, TSA also provided FSDs with the flexibility to deliver
basic screener training using local instructors. TSA's Workforce
Performance and Training Office developed basic screener training
internally, and initially, contractors delivered all of the basic
training. Since then, TSA has provided FSDs with the discretion to
provide the training using local TSA employees or to use contractors.
The flexibility to use local employees allows FSDs and members of the
screener workforce to leverage their first-hand screening knowledge and
experience and address situations unique to individual
airports.[Footnote 20] As of December 10, 2004, TSA had trained 1,021
local FSD staff (representing 218 airports) in how to instruct the dual-
function screener training course. TSA officials stated that they
expect the use of TSA-approved instructors to increase over time.
TSA Has Deployed Recurrent Screener Training and Provided Additional
Training Tools, but Some FSDs Identified the Need for More Courses:
TSA has taken several steps to deploy a recurrent screener training
program, including implementing the Online Learning Center--which
includes several recurrent training modules developed by TSA--
implementing a recurrent training requirement, and providing hands-on
training tools. According to TSA, comprehensive and frequent training
is key to a screener's ability to detect threat objects. In September
2003, we reported that TSA had not fully developed or deployed a
recurrent training program for passenger screeners. At that time, some
FSDs expressed concern with the lack of training available to the
screener workforce, and in the absence of headquarters-provided
training, they were developing and implementing locally based recurrent
training programs. Similar to our findings, TSA's April 2004 consultant
study reported that most FSDs and their staffs were generally
unsatisfied with the training support provided by
headquarters.[Footnote 21] Specifically, the study found that:
"Numerous interviews revealed concerns with training curriculum,
communication, and coordination issues that directly affect security
screening. Unsatisfied with the quantity and breadth of topics, many
Training Coordinators have developed supplementary lectures on both
security and non-security related topics. These additional
lectures—have been very highly received by screeners."
In October 2003, TSA introduced the Online Learning Center to provide
screeners with remote access to self-guided training courses. As of
September 14, 2004, TSA had provided access to over 550 training
courses via the Online Learning Center and made the system available
via the Internet and its intranet. TSA also developed and deployed a
number of hands-on training modules and associated training tools for
screeners at airports nationwide. These training modules cover topics
including hand-wanding and pat-down techniques, physical bag searches,
X-ray images, prohibited items, and customer service. Additionally, TSA
instituted another module for the Online Learning Center called Threat
in the Spotlight, that, based on intelligence TSA receives, provides
screeners with the latest in threat information regarding terrorist
attempts to get threat objects past screening checkpoints. Appendix III
provides a summary of the recurrent training tools TSA has deployed to
airports and the modules currently under development.
In December 2003, TSA issued a directive requiring screeners to receive
3 hours of recurrent training per week averaged over a quarter year.
One hour is required to be devoted to X-ray image interpretation and
the other 2 hours to screening techniques, review of standard operating
procedures, or other mandatory administrative training, such as ethics
and privacy act training.
In January 2004, TSA provided FSDs with additional tools to facilitate
and enhance screener training. Specifically, TSA provided airports with
at least one modular bomb set (MBS II) kit--containing components of an
improvised explosive device--and one weapons training kit, in part
because screeners had consistently told TSA's OIAPR inspectors that
they would like more training with objects similar to ones used in
covert testing.[Footnote 22]
Although TSA has made progress with the implementation of recurrent
training, some FSDs identified the need for several additional courses,
including courses that address more realistic threats. TSA acknowledged
that additional screener training is needed, and officials stated that
the agency is in the process of developing new and improved screener
training, including additional recurrent training modules (see app.
III).
TSA Provides Leadership and Technical Training for Supervisors, but
Some FSDs Would Like More Training for Screening Supervisors and Lead
Screeners:
TSA has arranged for leadership training for screening supervisors
through the Department of Agriculture Graduate School and has developed
leadership and technical training courses for screening supervisors.
However, some FSDs reported the need for more training for Screening
Supervisors and Lead Screeners. The quality of Screening Supervisors
has been a long-standing concern. In testifying before the 9/11
Commission in May 2003, a former FAA Assistant Administrator for Civil
Aviation Security stated that following a series of covert tests at
screening checkpoints to determine which were strongest, which were
weakest, and why, invariably the checkpoint seemed to be as strong or
as weak as the supervisor who was running it. Similarly, TSA's OIAPR
identified a lack of supervisory training as a cause for screener
covert testing failures. Further, in a July 2003 internal study of
screener performance, TSA identified poor supervision at the screening
checkpoints as a cause for screener performance problems. In
particular, TSA acknowledged that many Lead Screeners, Screening
Supervisors, and Screening Managers did not demonstrate supervisory and
management skills (i.e., mentoring, coaching, and positive
reinforcement) and provided little or no timely feedback to guide and
improve screener performance. In addition, the internal study found
that because of poor supervision at the checkpoint, supervisors or
peers were not correcting incorrect procedures, optimal performance
received little reinforcement, and not enough breaks were provided to
screeners. A September 2004 report by the DHS OIG supported these
findings, noting that Screening Supervisors and Screening Managers
needed to be more attentive in identifying and correcting improper or
inadequate screener performance.[Footnote 23]
TSA recognizes the importance of Screener Supervisors and has
established training programs to enhance their performance and
effectiveness. In September 2003, we reported that TSA had begun
working with the Department of Agriculture Graduate School to tailor
the school's off-the-shelf supervisory course to meet the specific
needs of Screening Supervisors, and had started training the existing
supervisors at that time through this course until the customized
course was fielded. According to TSA's training records, as of
September 2004, about 3,800 Screening Supervisors had completed the
course--approximately 92 percent of current Screening Supervisors. In
response to our survey, one FSD noted that the supervisory training was
long overdue because most of the supervisors had no prior federal
service or, in some cases, no leadership experience. This FSD also
noted that "leadership and supervisory skills should be continuously
honed; thus, the development of our supervisors should be an extended
and sequential program with numerous opportunities to develop skills--
not just a one-time class."
In addition to the Department of Agriculture Graduate School course,
TSA's Online Learning Center includes over 60 supervisory courses
designed to develop leadership and coaching skills. In April 2004, TSA
included in the Online Learning Center a Web-based technical training
course--required for all Lead Screeners and Screening Supervisors. This
course covers technical issues, such as resolving alarms at screening
checkpoints. TSA introduced this course to the field in March 2004, and
although the course is a requirement, TSA officials stated that they
have not set goals for when all Lead Screeners and Screening
Supervisors should have completed the course. In June 2004, TSA
training officials stated that a second supervisor technical course was
planned for development and introduction later in 2004. However, in
December 2004, the training officials stated that planned funding for
supervisory training may be used to support other TSA initiatives. The
officials acknowledged that this would reduce TSA's ability to provide
the desired type and level of supervisory training to its Lead
Screener, Screening Supervisor, and Screening Manager staff. TSA plans
to revise its plans to provide Lead Screener, Screening Supervisor, and
Screening Manager training based on funding availability.
Although TSA has developed leadership and technical courses for
Screening Supervisors, many FSDs, in response to our general survey,
identified additional types of training needed to enhance screener
supervision. Table 2 provides a summary of the additional training
needs that FSDs reported.
Table 2: Training Needed to Enhance Screener Supervision, as Reported
by FSDs:
Training topic: Leadership and management;
Percentage of FSDs indicating this need: 41%.
Training topic: Public speaking/communications;
Percentage of FSDs indicating this need: 26%.
Training topic: Conflict management;
Percentage of FSDs indicating this need: 24%.
Training topic: Human resource-related issues (hiring, termination,
discipline);
Percentage of FSDs indicating this need: 19%.
Training topic: Counseling, mentoring, and coaching;
Percentage of FSDs indicating this need: 18%.
Training topic: Writing skills;
Percentage of FSDs indicating this need: 16%.
Training topic: Additional supervisory training;
Percentage of FSDs indicating this need: 13%.
Training topic: Interpersonal skills;
Percentage of FSDs indicating this need: 13%.
Training topic: Crisis management and incident response;
Percentage of FSDs indicating this need: 6%.
Source: GAO general survey of FSDs.
[End of table]
TSA training officials stated that the Online Learning Center provides
several courses that cover these topics. Such courses include:
* Situation Leadership II;
* Communicating with Difficult People: Handling Difficult Co-Workers;
* Team Participation: Resolving Conflict in Teams;
* Employee Performance: Resolving Conflict;
* High Impact Hiring;
* Team Conflict: Overcoming Conflict with Communication;
* Correcting Performance Problems: Disciplining Employees;
* Team Conflict: Working in Diversified Teams;
* Correcting Performance Problems: Identifying Performance Problems;
* Resolving Interpersonal Skills;
* Grammar, Skills, Punctuation, Mechanics and Word Usage; and:
* Crisis in Organizations: Managing Crisis Situations.
TSA training officials acknowledged that for various reasons FSDs might
not be aware that the supervisory and leadership training is available.
For example, FSDs at airports without high-speed Internet/intranet
access to the Online Learning Center might not have access to all of
these courses. It is also possible that certain FSDs have not fully
browsed the contents of the Online Learning Center and therefore are
not aware that the training is available.[Footnote 24] Furthermore,
officials stated that online learning is relatively new to government
and senior field managers, and some of the FSDs may expect traditional
instructor-led classes rather than online software.
Some FSDs Reported Impediments to Screeners Receiving Recurrent
Training:
Some FSDs responded to our general survey that they faced challenges
with screeners receiving recurrent training, including insufficient
staffing to allow all screeners to complete training within normal duty
hours and a lack of high-speed Internet/intranet connectivity at some
training facilities. According to our guide for assessing training, to
foster an environment conducive to effective training and development,
agencies must take actions to provide sufficient time, space, and
equipment to employees to complete required training.[Footnote 25] TSA
has set a requirement for 3 hours of recurrent training per week
averaged over a quarter year, for both full-time and part-time
screeners. However, FSDs for about 18 percent (48 of 263) of the
airports in our airport-specific survey reported that screeners
received less than 9 to 12 hours of recurrent training per
month.[Footnote 26] Additionally, FSDs for 48 percent (125 of 263) of
the airports in the survey reported that there was not sufficient time
for screeners to receive recurrent training within regular work hours.
At 66 percent of those airports where the FSD reported that there was
not sufficient time for screeners to receive recurrent training within
regular work hours, the FSDs cited screener staffing shortages as the
primary reason. We reported in February 2004 that FSDs at 11 of the 15
category X airports we visited reported that they were below their
authorized staffing levels because of attrition and difficulties in
hiring new staff. In addition, three of these FSDs noted that they had
never been successful in hiring up to the authorized staffing levels.
We also reported in February 2004 that FSDs stated that because of
staffing shortages, they were unable to let screeners participate in
training because it affected the FSD's ability to provide adequate
coverage at the checkpoints. In response to our survey, FSDs across all
categories of airports reported that screeners must work overtime in
order to participate in training. A September 2004 DHS OIG report
recommended that TSA examine the workforce implications of the 3-hour
training requirement and take steps to correct identified imbalances in
future workforce planning to ensure that all screeners are able to meet
the recurrent training standard. The 3-hours-per-week training standard
represents a staff time commitment of 7.5 percent of full-time and
between 9 and 15 percent of part-time screeners' nonovertime working
hours. TSA headquarters officials have stated that because the 3-hours-
per-week requirement is averaged over a quarter, it provides
flexibility to account for the operational constraints that exist at
airports. However, TSA headquarters officials acknowledged that many
airports are facing challenges in meeting the 3-hour recurrent training
requirement. TSA data for the fourth quarter of fiscal year 2004
reported that 75 percent of airports were averaging less than 3 hours
of recurrent training per week per screener. The current screener
staffing model, which is used to determine the screener staffing
allocations for each airport, does not take the 3-hours-per-week
recurrent training requirement into account.[Footnote 27] However, TSA
headquarters officials said that they are factoring this training
requirement into their workforce planning efforts, including the
staffing model currently under development.[Footnote 28]
Another barrier to providing recurrent training is the lack of high-
speed Internet/intranet access at some of TSA's training
locations.[Footnote 29] TSA officials acknowledged that many of the
features of the Online Learning Center, including some portions of the
training modules and some Online Learning Center course offerings, are
difficult or impossible to use in the absence of high-speed
Internet/intranet connectivity. As one FSD put it, "the delayed
deployment of the high-speed Internet package limits the connectivity
to TSA HQ for various online programs that are mandated for passenger
screening operations including screener training." One FSD for a
category IV airport noted the lack of a high-speed connection for the
one computer at an airport he oversees made the Online Learning Center
"nearly useless."
TSA began deploying high-speed access to its training sites and
checkpoints in May 2003 and has identified high-speed connectivity as
necessary in order to deliver continuous training to screeners. TSA's
July 2003 Performance Improvement Study recommended accelerating high-
speed Internet/intranet access in order to provide quick and systematic
distribution of information and, thus, reduce uncertainty caused by the
day-to-day changes in local and national procedures and policy. In
October 2003, TSA reported plans to have an estimated 350 airports
online with high-speed connectivity within 6 months. However, in June
2004, TSA reported that it did not have the resources to reach this
goal.
TSA records show that as of October 2004, TSA had provided high-speed
access for training purposes to just 109 airports, where 1,726 training
computers were fully connected.[Footnote 30] These 109 airports had an
authorized staffing level of over 24,900 screeners, meaning that nearly
20,100 screeners (45 percent of TSA's authorized screening workforce)
still did not have high-speed Internet/intranet access to the Online
Learning Center at their training facility. In October 2004, TSA
officials stated that TSA's Office of Information Technology had
selected an additional 16 airport training facilities with a total of
205 training computers to receive high-speed connectivity by the end of
December 2004. As of January 19, 2005, TSA was unable to confirm that
these facilities had received high-speed connectivity. Additionally,
they could not provide a time frame for when they expected to provide
high-speed connectivity to all airport training facilities because of
funding uncertainties. Furthermore, TSA does not have a plan for
prioritizing and scheduling the deployment of high-speed connectivity
to all airport training facilities once funding is available. Without a
plan, TSA's strategy and timeline for implementing connectivity to
airport training facilities is unclear. The absence of such a plan
limits TSA's ability to make prudent decisions about how to move
forward with deploying connectivity once funding is available. Figure 5
shows the percentage of airports reported to have high-speed
connectivity for their training computers by category of airport as of
October 2004.
Figure 5: Percentage of Airports Reported to Have High-Speed
Connectivity for Training Purposes as of October 2004:
[See PDF for image]
Note: These data show airports with high-speed connectivity on training
computers. Some airports have high-speed connectivity, but for purposes
other than training.
[End of figure]
To mitigate airport connectivity issues in the interim, on April 1,
2004, TSA made the Online Learning Center courses accessible through
public Internet connections, which enable screeners to log on to the
Online Learning Center from home, a public library, or other locations.
However, TSA officials stated that the vast majority of screeners who
have used the Online Learning Center have logged in from airports with
connectivity at their training facilities. TSA also distributes new
required training products using multiple delivery channels, including
written materials and CD-ROMs for those locations where access to the
Online Learning Center is limited. Specifically, TSA officials stated
that they provided airports without high-speed connectivity with CD-
ROMs for the 50 most commonly used optional commercial courseware
titles covering topics such as information technology skills, customer
service, and teamwork. Additionally, officials stated that as technical
courses are added to the Online Learning Center, they are also
distributed via CD-ROM and that until full connectivity is achieved,
TSA will continue to distribute new training products using multiple
delivery channels.
TSA Lacks Adequate Controls to Provide Reasonable Assurance That
Screeners Receive Required Training:
Because of a lack of internal controls, TSA cannot provide reasonable
assurance that screeners are completing required training. First, TSA
policy does not clearly define responsibility for ensuring that
screeners have completed all required training. Additionally, TSA has
no formally defined policies or procedures for documenting completion
of remedial training, or a system designed to facilitate review of this
documentation for purposes of monitoring. Further, TSA headquarters
does not have formal policies and procedures for monitoring completion
of basic training and lacks procedures for monitoring recurrent
training. Finally, at airports without high-speed connectivity,
training records must be entered manually, making it challenging for
some airports to keep accurate and up-to-date training records.
TSA Policy Does Not Clearly Define Responsibility for Ensuring That
Screeners Are in Compliance with Training Requirements:
TSA's current guidance for FSDs regarding the training of the screener
workforce does not clearly identify responsibility for tracking and
ensuring compliance with training requirements. In a good control
environment, areas of authority and responsibility are clearly defined
and appropriate lines of reporting are established.[Footnote 31] In
addition, internal control standards also require that responsibilities
be communicated within an organization. The Online Learning Center
provides TSA with a standardized, centralized tool capable of
maintaining all training records in one system. It replaces an ad hoc
system previously used during initial rollout of federalized screeners
in which contractors maintained training records. A February 2004
management directive states that FSDs are responsible for ensuring the
completeness, accuracy, and timeliness of training records maintained
in the Online Learning Center for their employees. For basic and
recurrent training, information is to be entered into the Online
Learning Center within 30 days of completion of the training activity.
However, the directive does not clearly identify who is responsible for
ensuring that employees comply with training requirements. Likewise, a
December 2003 directive requiring that screeners complete 3 hours of
training per week averaged over a quarter states that FSDs are
responsible for ensuring that training records for each screener are
maintained in the Online Learning Center. Although both directives
include language that requires FSDs to ensure training records are
maintained in the Online Learning Center, neither specifies whether
FSDs or headquarters officials are responsible for ensuring compliance
with the basic, recurrent, and remedial training requirements. Even so,
TSA headquarters officials told us that FSDs are ultimately responsible
for ensuring screeners receive required training. However, officials
provided no documentation clearly defining this responsibility. Without
a clear designation of responsibility for monitoring training
completion, this function may not receive adequate attention, leaving
TSA unable to provide reasonable assurance that its screening workforce
receives required training. In April 2005, TSA officials responsible
for training stated that they were updating the February 2004
management directive on training records to include a specific
requirement for FSDs to ensure that screeners complete required
training. They expect to release the revised directive in May 2005.
TSA Lacks Formal Policies and Procedures for Monitoring Completion of
Required Training:
TSA has not established and documented policies and procedures for
monitoring completion of basic and recurrent training. Internal control
standards advise that internal controls should be designed so that
monitoring is ongoing and ingrained in agency operations.[Footnote 32]
However, TSA headquarters officials stated that they have no formal
policy for monitoring screeners' completion of basic training. They
also stated that they have neither informal nor formal procedures for
monitoring the completion of screeners' recurrent training
requirements, and acknowledged that TSA policy does not address what is
to occur if a screener does not meet the recurrent training
requirement. Officials further stated that individual FSDs have the
discretion to determine what action, if any, to take when screeners do
not meet this requirement.
In July 2004, TSA training officials stated that headquarters staff
recently began running a report in the Online Learning Center to review
training records to ensure that newly hired screeners had completed
required basic training. In addition, they stated that in June 2004,
they began generating summary-level quarterly reports from the Online
Learning Center to quantify and analyze hours expended for recurrent
screener training. Specifically, TSA training officials stated that
reports showing airport-level compliance with the 3-hour recurrent
requirement were generated for the third and fourth quarters of fiscal
year 2004 and delivered to the Office of Aviation Operations for
further analysis and sharing with the field. However, Aviation
Operations officials stated that they did not use these reports to
monitor the status of screener compliance with the 3-hour recurrent
training requirement and do not provide them to the field unless
requested by an FSD. TSA training officials said that while
headquarters intends to review recurrent training activity on an
ongoing basis at a national and airport level, they view FSDs and FSD
training staff as responsible for ensuring that individuals receive all
required training. Further, they acknowledged that weaknesses existed
in the reporting capability of the Online Learning Center and stated
that they plan to upgrade the Online Learning Center with improved
reporting tools by the end of April 2005. Without clearly defined
policies and procedures for monitoring the completion of training, TSA
lacks a structure to support continuous assurance that screeners are
meeting training requirements.
TSA Lacks Clearly Defined Policies and Procedures for Documenting
Completion of Remedial Training:
TSA has not established clear policies and procedures for documenting
completion of required remedial training. The Standards for Internal
Control state that agencies should document all transactions and other
significant events and should be able to make this documentation
readily available for examination.[Footnote 33] A TSA training bulletin
dated October 15, 2002, specifies that when remedial training is
required, FSDs must ensure the training is provided and a remedial
training reporting form is completed and maintained with the screener's
local records.[Footnote 34] However, when we asked to review these
records, we found confusion as to how and where they were to be
maintained. TSA officials stated that they are waiting for a decision
regarding how to maintain these records because of their sensitive
nature. In the meantime, where and by whom the records should be
maintained remains unclear.
In September 2004, officials from TSA's OIAPR--responsible for
conducting covert testing--stated that they maintain oversight to
ensure screeners requiring remedial training receive required training
by providing a list of screeners that failed covert testing and
therefore need remedial training to TSA's Office of Aviation
Operations. Aviation Operations is then to confirm via memo that each
of the screeners has received the necessary remedial training and
report back to OIAPR. Accordingly, we asked TSA for all Aviation
Operations memos confirming completion of remedial training, but we
were only able to obtain 1 of the 12 memos.[Footnote 35]
In addition, during our review, we asked to review the remedial
training reporting forms at five airports to determine whether
screeners received required training, but we encountered confusion
about requirements for maintaining training records and inconsistency
in record keeping on the part of local TSA officials. Because of the
unclear policies and procedures for recording completion of remedial
training, TSA does not have adequate assurance that screeners are
receiving legislatively mandated remedial training.
Lack of High-Speed Connectivity Limits TSA's Ability to Document and
Track the Completion of Screener Training:
Although training computers with high-speed Internet/intranet
connectivity automatically record completion of training in the Online
Learning Center, airports without high-speed access at their training
facility must have these records entered manually. The February 2004
management directive that describes responsibility for entering
training records into the Online Learning Center also established that
all TSA employees are required to have an official TSA training record
in the Online Learning Center that includes information on all official
training that is funded wholly or in part with government funds.
Without high-speed access, TSA officials stated that it can be a
challenge for airports to keep the Online Learning Center up to date
with the most recent training records. TSA headquarters officials
further stated that when they want to track compliance with mandatory
training such as ethics or civil rights training, they provide the
Training Coordinators with a spreadsheet on which to enter the data
rather than relying on the Online Learning Center. As one FSD told us,
without high-speed connectivity at several of the airports he oversees,
"this is very time consuming and labor intensive and strains my limited
resources." The difficulty that airports encounter in maintaining
accurate records when high-speed access is absent could compromise
TSA's ability to provide reasonable assurance that screeners are
receiving mandated basic and remedial training.
Progress Has Been Made in Implementing Tools to Measure and Enhance
Screener Performance, but Key Performance Targets Have Not Been
Finalized:
TSA has improved its efforts to measure and enhance screener
performance. However, these efforts have primarily focused on passenger
screening rather than checked baggage screening, and TSA has not yet
finalized performance targets for several key performance measures. For
example, TSA has increased the amount of covert testing it performs at
airports. These tests have identified that, overall, weaknesses and
vulnerabilities continue to exist in the passenger and checked baggage
screening systems. TSA also enabled FSDs to conduct local covert
testing, fully deployed the Threat Image Projection (TIP) system to
passenger screening checkpoints at commercial airports nationwide, and
completed the 2003/2004 annual screener recertification program for all
eligible screeners. However, not all of these performance measurement
and enhancement tools are available for checked baggage screening.
Specifically, TIP is not currently operational at checked baggage
screening checkpoints, and the recertification program does not include
an image recognition component for checked baggage screeners. However,
TSA is taking steps to address the overall imbalance in passenger and
checked baggage screening performance data, including working toward
implementing TIP for checked baggage screening and developing an image
recognition module for checked baggage screener recertification. To
enhance screener and screening system performance, TSA has also
conducted a passenger screener performance improvement study and
subsequently developed an improvement plan consisting of multiple
action items, many of which TSA has completed. However, TSA has not
conducted a similar study for checked baggage screeners. In addition,
TSA has established over 20 performance measures for the passenger and
checked baggage screening systems as well as two performance indexes
(one for passenger and one for checked baggage screening). However, TSA
has not established performance targets for each of the component
indicators within the indexes, such as covert testing. According to The
Office of Management and Budget, performance goals are target levels of
performance expressed as a measurable objective, against which actual
achievement can be compared. Performance goals should incorporate
measures (indicators used to gauge performance); targets
(characteristics that tell how well a program must accomplish the
measure), and time frames. Without these targets, TSA's performance
management system, and these performance indexes, specifically, may not
provide the agency with the complete information necessary to assess
achievements and make decisions about where to direct performance
improvement efforts. Although TSA has not yet established performance
targets for each of the component indicators, TSA plans to finalize
performance targets for the indicators by the end of fiscal year 2005.
TSA Has Increased Its Covert Testing and Allowed Local Covert Testing
at Passenger Screening Checkpoints:
TSA headquarters has increased the amount of covert testing it performs
and enabled FSDs to conduct additional local covert testing at
passenger screening checkpoints. TSA's OIAPR conducts unannounced
covert tests of screeners to assess their ability to detect threat
objects and to adhere to TSA-approved procedures. These tests, in which
undercover OIAPR inspectors attempt to pass threat objects through
passenger screening checkpoints and in checked baggage, are designed to
measure vulnerabilities in passenger and checked baggage screening
systems and to identify systematic problems affecting performance of
screeners in the areas of training, policy, and technology.[Footnote
36] TSA considers its covert testing as a "snapshot" of a screener's
ability to detect threat objects at a particular point in time and as
one of several indicators of system wide screener performance.
OIAPR conducts tests at passenger screening checkpoints and checked
baggage screening checkpoints. According to OIAPR, these tests are
designed to approximate techniques terrorists might use. These covert
test results are one source of data on screener performance in
detecting threat objects as well as an important mechanism for
identifying areas in passenger and checked baggage screening needing
improvement. In testimony before the 9/11 Commission, the Department of
Transportation Inspector General stated that emphasis must be placed on
implementing an aggressive covert testing program to evaluate
operational effectiveness of security systems and equipment.[Footnote
37]
Between September 10, 2002, and September 30, 2004, OIAPR conducted a
total of 3,238 covert tests at 279 different airports. In September
2003, we reported that OIAPR had conducted limited covert testing but
planned to double the amount of tests it conducted during fiscal year
2004, based on an anticipated increase in its staff from about 100 full-
time equivalents to about 200 full-time equivalents. TSA officials
stated that based on budget constraints, OIAPR's fiscal year 2004
staffing authorization was limited to 183 full-time-equivalents,
[Footnote 38] of which about 60 are located in the field.[Footnote 39]
Despite a smaller than expected staff increase, by the end of the
second quarter of fiscal year 2004, OIAPR had already surpassed the
number of tests it performed during fiscal year 2003, as shown in table
3.
Table 3: Checkpoint and Checked Baggage Tests Conducted by OIAPR,
September 10, 2002-September 30, 2004:
Testing period fiscal year/quarter: 2002 fourth;
Airports[A]: 2;
Passenger screening checkpoint tests: 30;
Checked baggage tests[B]: 3;
Total tests: 33.
Testing period fiscal year/quarter: 2003 first;
Airports[A]: 14;
Passenger screening checkpoint tests: 120;
Checked baggage tests[B]: 1;
Total tests: 121.
Testing period fiscal year/quarter: 2003 second;
Airports[A]: 31;
Passenger screening checkpoint tests: 231;
Checked baggage tests[B]: 27;
Total tests: 258.
Testing period fiscal year/quarter: 2003 third;
Airports[A]: 28;
Passenger screening checkpoint tests: 198;
Checked baggage tests[B]: 19;
Total tests: 217.
Testing period fiscal year/quarter: 2003 fourth;
Airports[A]: 25;
Passenger screening checkpoint tests: 217;
Checked baggage tests[B]: 23;
Total tests: 240.
Testing period fiscal year/quarter: 2004 first;
Airports[A]: 41;
Passenger screening checkpoint tests: 171;
Checked baggage tests[B]: 110;
Total tests: 281.
Testing period fiscal year/quarter: 2004 second;
Airports[A]: 111;
Passenger screening checkpoint tests: 770;
Checked baggage tests[B]: 182;
Total tests: 952.
Testing period fiscal year/quarter: 2004 third;
Airports[A]: 56;
Passenger screening checkpoint tests: 379;
Checked baggage tests[B]: 102;
Total tests: 481.
Testing period fiscal year/quarter: 2004 fourth;
Airports[A]: 64;
Passenger screening checkpoint tests: 527;
Checked baggage tests[B]: 128;
Total tests: 655.
Total;
Passenger screening checkpoint tests: 2,643;
Checked baggage tests[B]: 595;
Total tests: 3,238.
Source: GAO analysis of OIAPR data.
Note: Some airports have been tested more than once.
[A] OIAPR conducted covert testing at three additional airports in
September and October 2002. However, at the time of the testing,
federal screeners had not yet been deployed to these airports. We
excluded these tests from our analysis.
[B] TSA did not begin reporting the results of checked baggage tests
until January 2003. However, four of these tests were conducted in
September and October 2002.
[End of table]
In October 2003, OIAPR committed to testing between 90 and 150 airports
by April 2004 as part of TSA's short-term screening performance
improvement plan. OAIPR officials stated that this was a onetime goal
to increase testing. This initiative accounts for the spike in testing
for the second quarter of fiscal year 2004.
OIAPR has created a testing schedule designed to test all airports at
least once during a 3-year time frame. Specifically, the schedule calls
for OIAPR to test all category X airports once a year, category I and
II airports once every 2 years, and category III and IV airports at
least once every 3 years.
In September 2003 and April 2004, we reported that TSA covert testing
results had identified weaknesses in screeners' ability to detect
threat objects.[Footnote 40] More recently, in April 2005, we, along
with the DHS OIG, identified that screener performance continued to be
a concern. Specifically, our analysis of TSA's covert testing results
for tests conducted between September 2002 and September 2004
identified that overall, weaknesses still existed in the ability of
screeners to detect threat objects on passengers, in their carry-on
bags, and in checked baggage.[Footnote 41] Covert testing results in
this analysis cannot be generalized either to the airports where the
tests were conducted or to airports nationwide.[Footnote 42] These
weaknesses and vulnerabilities were identified at airports of all
sizes, at airports with federal screeners, and airports with private-
sector screeners. For the two-year period reviewed, overall failure
rates for covert tests (passenger and checked baggage) conducted at
airports using private-sector screeners were somewhat lower than
failure rates for the same tests conducted at airports using federal
screeners for the airports tested during this period.[Footnote 43]
Since these test results cannot be generalized as discussed above, each
airport's test results should not be considered a comprehensive
measurement of the airport's performance or any individual screener's
performance in detecting threat objects, or in determining whether
airports with private sector screeners performed better than airports
with federal screeners.
On the basis of testing data through September 30, 2004, we determined
that OIAPR had performed covert testing at 61 percent of the nation's
commercial airports. TSA has until September 30, 2005, to test the
additional 39 percent of airports and meet its goal of testing all
airports within 3 years. Although officials stated that they have had
to divert resources from airport testing to conduct testing of other
modes and that testing for other modes of transportation may affect
their ability to conduct airport testing, they still expect to meet the
goal.
In February 2004, TSA provided protocols to help FSDs conduct their own
covert testing of local airport passenger screening activities--a
practice that TSA had previously prohibited.[Footnote 44] Results of
local testing using these protocols are to be entered into the Online
Learning Center. This information, in conjunction with OAIPR covert
test results and TIP threat detection results, is intended to assist
TSA in identifying specific training and performance improvement
efforts. In February 2005, TSA released a general procedures document
for local covert testing at checked baggage screening locations.
TSA officials said that they had not yet begun to use data from local
covert testing to identify training and performance needs because of
difficulties in ensuring that local covert testing is implemented
consistently nationwide. These officials said that after a few months
of collecting and assessing the data, they will have a better idea of
how the data can be used.
TSA Has Reinstated the Threat Image Projection System and Plans to Use
Its Data to Improve the Screening System:
TSA has nearly completed the reactivation of the TIP system at airports
nationwide and plans to use data it is collecting to improve the
effectiveness of the passenger screening system.[Footnote 45] TIP is
designed to test passenger screeners' detection capabilities by
projecting threat images, including guns, knives, and explosives, onto
bags as they are screened during actual operations. Screeners are
responsible for identifying the threat image and calling for the bag to
be searched. Once prompted, TIP identifies to the screener whether the
threat is real and then records the screener's performance in a
database that could be analyzed for performance trends.[Footnote 46]
TSA is evaluating the possibility of developing an adaptive
functionality to TIP. Specifically, as individual screeners become
proficient in identifying certain threat images, such as guns or
knives, they will receive fewer of those images and more images that
they are less proficient at detecting, such as improvised explosive
devices.
TIP was activated by FAA in 1999 with about 200 threat images, but it
was shut down immediately following the September 11 terrorist attacks
because of concerns that it would result in screening delays and panic,
as screeners might think that they were actually viewing threat
objects. In October 2003, TSA began reactivating and expanding TIP. In
April 2004, we reported that TSA was reactivating TIP with an expanded
library of 2,400 images at all but one of the more than 1,800
checkpoint lanes nationwide. To further enhance screener training and
performance, TSA also plans to develop at least an additional 50 images
each month.
Despite these improvements, TIP is not yet available for checked
baggage screening. In April 2004, we reported that TSA officials stated
that they were working to resolve technical challenges associated with
using TIP for checked baggage screening on EDS machines and have
started EDS TIP image development. The DHS OIG reported in September
2004 that TSA plans to implement TIP on all EDS machines at checked
baggage stations nationwide in fiscal year 2005. However, in December
2004, TSA officials stated that because of severe budget reductions,
TSA will be unable to begin implementing a TIP program for checked
baggage in fiscal year 2005. They did not specify when such a program
might begin.
TSA plans to use TIP data to improve the passenger screening system in
two ways. First, TIP data can be used to measure screener threat
detection effectiveness by different threats. Second, TSA plans to use
TIP results to help identify specific recurrent training needs within
and across airports and to tailor screeners' recurrent training to
focus on threat category areas that indicate a need for improvement.
TSA considers February 2004 as the first full month of TIP reporting
with the new library of 2,400 images. TSA began collecting these data
in early March 2004 and is using the data to determine more precisely
how they can be used to measure screener performance in detecting
threat objects and to determine what the data identify about screener
performance.
TSA does not currently plan to use TIP data as an indicator of
individual screener performance because TSA does not believe that TIP
by itself adequately reflects a screener's performance. Nevertheless,
in April 2004, TSA gave FSDs the capability to query and analyze TIP
data in a number of ways, including by screener, checkpoint, and
airport. FSDs for over 60 percent of the airports included in our
airport-specific survey stated that they use or plan to use TIP data as
a source of information in their evaluations of individual screener
performance. Additionally, FSDs for 50 percent of the airports covered
in our survey reported using data generated by TIP to identify specific
training needs for individual screeners.
In September 2004, the DHS OIG reported that TSA is assessing the cost
and feasibility of modifying TIP so that it recognizes and responds to
specific threat objects with which individual screeners are most and
least competent in detecting, over time. This feature would increase
the utility of TIP as a training tool. The DHS OIG also reported that
TSA is considering linking TIP over a network, which would facilitate
TSA's collection, analysis, and information-sharing efforts around TIP
user results. The report recommended that TSA continue to pursue each
of these initiatives, and TSA agreed. However, in December 2004, TSA
officials stated that the availability of funding will determine
whether or not they pursue these efforts further.
TSA Has Completed Its First Round of Screener Recertification Testing,
but Testing for Checked Baggage Screeners Is Not as Comprehensive as
for Passenger Screeners:
TSA has completed its first round of the screener recertification
program, and the second round is now under way. However, TSA does not
currently include an image recognition component in the test for
checked baggage screener recertification. ATSA requires that each
screener receive an annual proficiency review to ensure he or she
continues to meet all qualifications and standards required to perform
the screening function. In September 2003, we reported that TSA had not
yet implemented this requirement.[Footnote 47] To meet this
requirement, TSA established a recertification program, and it began
recertification testing in October 2003 and completed the testing in
March 2004.[Footnote 48] The first recertification program was composed
of two assessment components, one of screeners' performance and the
other of screeners' knowledge and skills. During the performance
assessment component of the recertification program, screeners are
rated on both organizational and individual goals, such as maintaining
the nation's air security, vigilantly carrying out duties with utmost
attention to tasks that will prevent security threats, and
demonstrating the highest levels of courtesy to travelers to maximize
their levels of satisfaction with screening services. The knowledge and
skills assessment component consists of three modules: (1) knowledge of
standard operating procedures, (2) image recognition, and (3) practical
demonstration of skills. Table 4 provides a summary of these three
modules.
Table 4: Modules Included in Recertification Knowledge and Skills
Assessment:
Testing modules: Knowledge of standard operating procedures;
Description: Computerized 50-question multiple-choice test. It is
either passenger-or baggage-specific.
Testing modules: Image recognition;
Description: Computerized test that consists of 100 images and is used
to evaluate a screener's skill and ability in detecting threat or
prohibited objects within X-ray images.
Testing modules: Practical demonstration of skills;
Description: Hands-on simulated work sample to evaluate a screener's
knowledge, skills, and ability when performing specific screener tasks
along with ability to provide customer service.
Source: TSA.
[End of table]
To be recertified, screeners must have a rating of "met" or "exceeded"
standards on their annual performance assessments and have passed each
of the applicable knowledge and skills modules. Screeners that failed
any of the three modules were to receive study time or remedial
training as well as a second opportunity to take and pass the modules.
Screeners who failed on their second attempt were to be removed from
screening duties and subject to termination. Screeners could also be
terminated for receiving a rating of below "met" standards.
TSA completed its analysis of the recertification testing and
performance evaluations in May 2004. TSA's analysis shows that less
than 1 percent of screeners subject to recertification failed to
complete this requirement. Figure 6 shows the recertification results.
Figure 6: Screener Recertification Results, October 2003-March 2004:
[See PDF for image]
[End of figure]
Across all airports screeners performed well on the recertification
testing. Over 97 percent of screeners passed the standard operating
procedures test on their first attempt. Screeners faced the most
difficulty on the practical demonstration of skills component. However,
following remediation, 98.6 percent of the screeners who initially
failed this component passed on their second attempt. Table 5 shows the
results of the recertification testing by module.
Table 5: Screener Recertification Module Testing Percentage Pass Rates,
October 2003--March 2004:
First attempt;
Standard operating procedures: 97.4%;
Image recognition: 96.0%;
Practical skills demonstration: 75.2%.
Retest;
Standard operating procedures: 96.8%;
Image recognition: 84.3%;
Practical skills demonstration: 98.6%.
Overall;
Standard operating procedures: 99.9%;
Image recognition: 99.5%;
Practical skills demonstration: 99.7%.
Source: TSA.
[End of table]
As shown in table 6, screeners hired as checked baggage screeners were
not required to complete the image recognition module in the first
round of the recertification testing.[Footnote 49]
Table 6: Recertification Testing Modules by Screening Function:
Testing modules: Knowledge of standard operating procedures;
Passenger: Yes;
Checked baggage: Yes.
Testing modules: Image recognition;
Passenger: Yes;
Checked baggage: No.
Testing modules: Practical demonstration of skills;
Passenger: Yes;
Checked baggage: Yes.
Source: TSA.
[End of table]
In addition, during the first year of recertification testing, which
took place from October 2003 through May 2004, dual-function screeners
who were actively working as both passenger and checked baggage
screeners were required to take only the recertification test for
passenger screeners. They were therefore not required to take the
recertification testing modules required for checked baggage, even
though they worked in that capacity.[Footnote 50]
TSA began implementing the second annual recertification testing in
October 2004 and plans to complete it no later than June 2005. This
recertification program includes components for dual-function
screeners. However, TSA still has not included an image recognition
module for checked baggage screeners--which would include dual-function
screeners performing checked baggage screening. TSA officials stated
that a decision was made to not include an image recognition module for
checked baggage screeners during this cycle because not all checked
baggage screeners would have completed training on the onscreen
resolution protocol by the time recertification testing was conducted
at their airports.[Footnote 51] In December 2004, TSA officials stated
that they plan on developing an image recognition module for checked
baggage and dual-function screeners, and that this test should be
available for next year's recertification program. The development and
implementation of the image recognition test will be contingent, they
stated, upon the availability of funds.
TSA Has Identified and Implemented Efforts to Enhance Screener
Performance, but These Efforts Primarily Focused on Passenger
Screeners:
TSA has implemented a number of improvements designed to enhance
screener performance, based on concerns it identified in a July 2003
Passenger Screener Performance Improvement Study and recommendations
from OIAPR. To date, however, these efforts have primarily focused on
the performance of passenger screeners, and TSA has not yet undertaken
a comparable performance study for checked baggage screeners. The
Passenger Screener Performance Improvement Study relied in part on the
findings of OIAPR's covert testing. At the time the study was issued,
OIAPR had conducted fewer than 50 tests of checked baggage screeners.
The July 2003 study focused on and included numerous recommendations
for improving the performance of passenger screeners, but recommended
waiting to analyze the performance of checked baggage screeners until
some time after implementation of the recommendations, some of which
TSA indicated, also applied to checked baggage screeners. TSA officials
told us that this analysis has been postponed until they have reviewed
the impact of implementing the recommendations on passenger screening
performance.
In October 2003, to address passenger screener performance deficiencies
identified in the Screener Performance Improvement Study, TSA developed
a Short-Term Screening Performance Improvement Plan. This plan included
specific action items in nine broad categories--such as enhance
training, increase covert testing, finish installing TIP, and expedite
high-speed connectivity to checkpoints and training computers--that TSA
planned to pursue to provide tangible improvements in passenger
screener performance and security (see app. IV for additional
information on the action items). In June 2004, TSA reported that it
had completed 57 of the 62 specific actions. As of December 2004, two
of these actions still had not been implemented--full deployment of
high-speed connectivity and a time and attendance package--both of
which continue to be deferred pending the identification of appropriate
resources.[Footnote 52]
In addition to the Performance Improvement Study and corresponding
action plans, TSA's OIAPR makes recommendations in its reports on
covert testing results. These recommendations address deficiencies
identified during testing and are intended to improve screening
effectiveness. As of December 2004, OIAPR had issued 18 reports to TSA
management on the results of its checkpoint and checked baggage covert
testing.[Footnote 53] These reports include 14 distinct
recommendations,[Footnote 54] some of which were included in TSA's
screener improvement action plan. All but two of these reports included
recommendations on corrective actions needed to enhance the
effectiveness of passenger and checked baggage screening.
TSA Has Established Screening Performance Measures and Indexes but Has
Not Established Key Performance Targets:
TSA has established performance measures, indexes, and targets for the
passenger and checked baggage screening systems, but has not
established targets for the various components of the screening
indexes. The Government Performance and Results Act of 1993 provides,
among other things, that federal agencies establish program performance
measures, including the assessment of relevant outputs and outcomes of
each program activity[Footnote 55]. Performance measures are meant to
cover key aspects of performance and help decision makers to assess
program accomplishments and improve program performance. A performance
target is a desired level of performance expressed as a tangible,
measurable objective, against which actual achievement will be
compared. By analyzing the gap between target and actual levels of
performance, management can target those processes that are most in
need of improvement, set improvement goals, and identify appropriate
process improvements or other actions.
An April 2004 consultant study commissioned by TSA found that FSDs and
FSD staffs generally believed the lack of key performance indicators
available to monitor passenger and checked baggage screening
performance represented a significant organizational weakness. Since
then, TSA has established over 20 performance measures for the
passenger and checked baggage screening systems. For example, TSA
measures the percentage of screeners meeting a threshold score on the
annual recertification testing on their first attempt, the percentage
of screeners scoring above the national standard level on TIP
performance, and the number of passengers screened, by airport
category.
TSA also has developed two performance indexes to measure the
effectiveness of the passenger and checked baggage screening
systems.[Footnote 56] These indexes measure overall performance through
a composite of indicators and are derived by combining specific
performance measures relating to passenger and checked baggage
screening, respectively. Specifically, these indexes measure the
effectiveness of the screening systems through machine probability of
detection and covert testing results;[Footnote 57] efficiency through a
calculation of dollars spent per passenger or bag screened;
and customer satisfaction through a national poll, customer surveys,
and customer complaints at both airports and TSA's national call
center. According to TSA officials, the agency has finalized targets
for the two overall indexes, but these targets have not yet been
communicated throughout the agency. Further, TSA plans to provide the
FSDs with only the performance index score, not the value of each of
the components, because the probabilities of detection are classified
as secret and TSA is concerned that by releasing components, those
probabilities could be deduced.[Footnote 58] Table 7 summarizes the
components of the performance indexes developed by TSA.
Table 7: Components of TSA's Performance Indexes:
Performance index: Passenger screening program;
Components:
* Machine probability of detection x person probability of detection
(covert testing);
Weight (percent): 50.
Performance index: Passenger screening program;
Components:
* Cost per person screened;
Weight (percent): 25;
FY 2005-2010 performance target (1-5 scale): 3.3.
Performance index: Passenger screening program;
Components:
* Consumer satisfaction;
Weight (percent): 25.
Performance index: Checked baggage screening program;
Components:
* Machine probability of detection x person probability of detection
(covert testing);
Weight (percent): 50.
Performance index: Checked baggage screening program;
Components:
* Cost per bag screened;
Weight (percent): 25;
FY 2005-2010 performance target (1-5 scale): 3.2.
Performance index: Checked baggage screening program;
Components:
* Consumer satisfaction;
Weight (percent): 25.
Source: TSA.
[End of table]
TSA has not yet established performance targets for the various
components of the screening indexes, including performance targets for
covert testing (person probability of detection). TSA's strategic plan
states that the agency will use the performance data it collects to
make tactical decisions based on performance. The screening performance
indexes developed by TSA can be a useful analysis tool, but without
targets for each component of the index, TSA will have difficulty
performing meaningful analyses of the parts that add up to the index.
For example, without performance targets for covert testing, TSA will
not have identified a desired level of performance related to screener
detection of threat objects. Performance targets for covert testing
would enable TSA to focus its improvement efforts on areas determined
to be most critical, as 100 percent detection capability may not be
attainable. In January 2005, TSA officials stated that the agency plans
to track the performance of individual index components and establish
performance targets against which to measure these components. They
further stated that they are currently collecting and analyzing
baseline data to establish these targets and plan to finalize them by
the end of fiscal year 2005.
Conclusions:
It has been over 2 years since TSA assumed responsibility for passenger
and checked baggage screening operations at the nation's commercial
airports. TSA has made significant accomplishments over this period in
meeting congressional mandates related to establishing these screening
operations. With the congressional mandates now largely met, TSA has
turned its attention to assessing and enhancing the effectiveness of
its passenger and checked baggage screening systems. An important tool
in enhancing screener performance is ongoing training. As threats and
technology change, the training and development of screeners to ensure
they have the competencies--knowledge, skills, abilities, and
behaviors--needed to successfully perform their screening functions
become vital to strengthening aviation security. Without addressing the
challenges to delivering ongoing training, including installing high-
speed connectivity at airport training facilities, TSA may have
difficulty maintaining a screening workforce that possesses the
critical skills needed to perform at a desired level. In addition,
without adequate internal controls designed to help ensure screeners
receive required training that are also communicated throughout the
agency, TSA cannot effectively provide reasonable assurances that
screeners receive all required training. Given the importance of the
Online Learning Center in both delivering training and serving as the
means by which the completion of screener training is documented, TSA
would benefit from having a clearly defined plan for prioritizing the
deployment of high-speed Internet/intranet connectivity to all airport
training facilities. Such a plan would help enable TSA to move forward
quickly and effectively in deploying high-speed connectivity once
funding is available.
Additionally, history demonstrates that U.S. commercial aircraft have
long been a target for terrorist attacks through the use of explosives
carried in checked baggage, and covert testing conducted by TSA and DHS
OIG have identified that weaknesses and vulnerabilities continue to
exist in the passenger and checked baggage screening systems, including
the ability of screeners to detect threat objects. While covert test
results provide an indicator of screening performance, they cannot
solely be used as a comprehensive measure of any airport's screening
performance or any individual screener's performance, or in determining
the overall performance of federal versus private-sector screening.
Rather, these data should be considered in the larger context of
additional performance data, such as TIP and recertification test
results, when assessing screener performance. While TSA has undertaken
efforts to measures and strengthen performance, these efforts have
primarily focused on passenger screening and not on checked baggage
screening. TSA's plans for implementing TIP for checked baggage
screening, and establishing an image recognition component for the
checked baggage screeners recertification testing--plans made during
the course of our review--represent significant steps forward in its
efforts to strengthen checked baggage screening functions.
Additionally, although TSA has developed passenger and checked baggage
screening effectiveness measures, the agency has not yet established
performance targets for the individual components of these measures.
Until such targets are established, it will be difficult for TSA to
draw more meaningful conclusions about its performance and how to most
effectively direct its improvement efforts. For example, performance
targets for covert testing would enable TSA to focus its improvement
efforts on areas determined to be most critical, as 100 percent
detection capability may not be attainable. We are encouraged by TSA's
recent plan to establish targets for the individual components of the
performance indexes. This effort, along with the additional performance
data TSA plans to collect on checked baggage screening operations,
should assist TSA in measuring and enhancing screening performance and
provide TSA with more complete information with which to prioritize and
focus its screening improvement efforts.
Recommendations for Executive Action:
To help ensure that all screeners have timely and complete access to
screener training available in the Online Learning Center and to help
provide TSA management with reasonable assurance that all screeners are
receiving required passenger and checked baggage screener training, we
recommend that the Secretary of the Department of Homeland Security
direct the Assistant Secretary, Transportation Security Administration,
to take the following two actions:
* develop a plan that prioritizes and schedules the deployment of high-
speed Internet/intranet connectivity to all TSA's airport training
facilities to help facilitate the delivery of screener training and the
documentation of training completion, and:
* develop internal controls, such as specific directives, clearly
defining responsibilities for monitoring and documenting the completion
of required training, and clearly communicate these responsibilities
throughout the agency.
Agency Comments and Our Evaluation:
We provided a draft of this report to DHS for review and comment. On
February 4, 2005, we received written comments on the draft report,
which are reproduced in full in appendix V. DHS generally concurred
with the findings and recommendations in the report, and agreed that
efforts to implement our recommendations are critical to successful
passenger and checked baggage screening training and performance. With
regard to our recommendation that TSA develop a plan that prioritizes
and schedules the deployment of high-speed Internet/intranet
connectivity to all TSA's airport training facilities, DHS stated that
TSA has developed such a plan. However, although we requested a copy of
the plan several times during our review and after receiving written
comments from DHS, TSA did not provide us with a copy of the plan.
Therefore, we cannot assess the extent to which the plan DHS referenced
in its written comments fulfills our recommendation. In addition,
regarding our recommendation that TSA develop internal controls clearly
defining responsibilities for monitoring and documenting the completion
of required training, and clearly communicate those responsibilities
throughout TSA, DHS stated that it is taking steps to define
responsibility for monitoring the completion of required training and
to insert this accountability into the performance plans of all TSA
supervisors. TSA's successful completion of these ongoing and planned
activities should address the concerns we raised in this report. DHS
has also provided technical comments on our draft report, which we
incorporated where appropriate.
As agreed with your office, we will send copies of this report to
relevant congressional committees and subcommittees and to the
Secretary of the Department of Homeland Security. We will also make
copies available to others upon request. In addition, the report will
be made available at no charge on GAO's Web site at http://www.gao.gov.
If you have any questions about this report or wish to discuss it
further, please contact me at (202) 512-8777. Key contributors to this
report are listed in appendix VI.
Sincerely yours,
Signed by:
Cathleen A. Berrick:
Director, Homeland Security and Justice Issues:
[End of section]
Appendix I: Summary of Previous Findings Related to Screener Training
and Performance:
Product date: September 2003;
Summary of previous findings related to screener training and
performance: The Transportation Security Administration (TSA) had
deployed a basic screener training program and required remedial
training but had not fully developed or deployed a recurrent training
program for screeners or supervisors;
TSA had collected little information to measure screener performance in
detecting threat objects;
* TSA's Office of Internal Affairs and Program Review's (OIAPR) covert
testing was the primary source of information collected on screeners'
ability to detect threat objects. However, TSA did not consider the
covert testing a measure of screener performance;
* TSA was not using the Threat Image Projection system (TIP) but
planned to fully activate the system with significantly more threat
images than previously used in October 2003;
* TSA had not yet implemented an annual proficiency review to ensure
that screeners met all qualifications and standards required to perform
their assigned screening functions;
Title and GAO product number: Airport Passenger Screening: Preliminary
Observations Made and Challenges Remaining; GAO-03-1173.
Product date: November 2003;
Summary of previous findings related to screener training and
performance: Although little data existed on the effectiveness of
passenger screening, TSA was implementing several efforts to collect
performance data;
* TSA's OIAPR had conducted little covert testing of the screener
workforce but planned to double the number of tests it conducted during
fiscal year 2004;
* TSA only recently began activating TIP on a wide-scale basis and
expected it to be fully operational at every checkpoint at all airports
by April 2004;
* TSA only recently began implementing the annual recertification
program and did not expect to complete testing at all airports until
March 2004;
* TSA was developing performance indexes for individual screeners and
the screening system as a whole but had not fully established these
indexes. TSA expected to have them in place by the end of fiscal year
2004;
Title and GAO product number: Aviation Security: Efforts to Measure
Effectiveness and Address Challenges; GAO-04-232T.
Product date: November 2003;
Summary of previous findings related to screener training and
performance: Although TSA had not fully developed or deployed recurrent
or supervisory training programs, it was in the process of;
* deploying six recurrent training modules and was pilot-testing an
online learning management system, and;
* working with the U.S. Department of Agriculture's Graduate School to
tailor its off-the-shelf supervisory course to meet the specific
training needs of screening supervisors;
Title and GAO product number: Aviation Security: Efforts to Measure
Effectiveness and Strengthen Security Programs; GAO-04-285T.
Product date: February 2004;
Summary of previous findings related to screener training and
performance: While TSA had taken steps to enhance its screener training
programs, staffing imbalances, and lack of high-speed connectivity at
airport training facilities had made it difficult for screeners at some
airports to fully utilize these programs; Although TSA was making
progress in measuring the performance of passenger screeners, it had
collected limited performance data related to its checked baggage
screening operations. However, TSA had begun collecting additional
performance data related to its checked baggage screening operations
and planned to increase these efforts in the future; As part of its
efforts to develop performance indexes, TSA was developing baseline
data for fiscal year 2004 and planned to report the indexes to DHS in
fiscal year 2005;
Title and GAO product number: Aviation Security: Challenges Exist in
Stabilizing and Enhancing Passenger and Baggage Screening Operations;
GAO-04-440T.
Product date: April 2004;
Summary of previous findings related to screener training and
performance: With the exception of covert testing and recent TIP data,
data were not yet available to assess how well screeners were
performing and what steps if any TSA needed to take to improve
performance. Also, TSA was not using TIP as a formal indicator of
screening performance, but instead was using it to identify individual
screener training needs;
Title and GAO product number: Aviation Security: Private Screening
Contractors Have Little Flexibility to Implement Innovative Approaches;
GAO-04-505T.
Source: GAO.
[End of table]
[End of section]
Appendix II: Objectives, Scope, and Methodology:
To examine efforts by the Transportation Security Administration to
enhance their passenger and checked baggage screening programs, we
addressed the following questions: (1) What actions has TSA taken to
enhance training for screeners and supervisors? (2) How does TSA
monitor compliance with screener training requirements? (3) What is the
status of TSA's efforts to assess and enhance screener performance in
detecting threat objects?
To determine how TSA has enhanced training for screeners and
supervisors and how TSA has monitored compliance with screener training
requirements, we obtained and analyzed relevant legislation, as well as
TSA's training plans, guidance, and curriculum. We reviewed data from
TSA's Online Learning Center and assessed the reliability of the Online
Learning Center database. We compared TSA's procedures for ensuring
that screeners receive required training according to Standards for
Internal Controls in the Federal Government. We interviewed TSA
officials from the Office of Workforce Performance and Training and the
Office of Aviation Operations in Arlington, Virginia. At the airports
we visited, we interviewed Federal Security Directors and their staffs,
such as Training Coordinators. We also met with officials from four
aviation associations--the American Association of Airport Executives,
Airports Council International, the Air Transport Association, and the
Regional Airline Association. We did not assess the methods used to
develop TSA's screener training program, nor did we analyze the
contents of TSA's curriculum. Although we could not independently
verify the reliability of all of this information, we compared the
information with other supporting documents, when available, to
determine data consistency and reasonableness. We found the data to be
sufficiently reliable for our purposes.
To determine what efforts TSA has taken to assess and to enhance
screener performance in detecting threat objects, we reviewed related
reports from the Department of Transportation and the Department of
Homeland Security (DHS) Inspector General, Congressional Research
Service, and TSA, as well as prior GAO reports. We obtained and
reviewed TSA's covert test data and results of the annual
recertification testing. (Results of the covert testing are classified
and will be the subject of a separate classified GAO report.) We
discussed methods for inputting, compiling, and maintaining the data
with TSA officials. We also assessed the methodology of TSA's covert
tests and questioned OIAPR officials about the procedures used to
ensure the reliability of the covert test data. When we found
discrepancies between the data OIAPR maintained in spreadsheets and the
data included in the hard copy reports we obtained from TSA, we worked
with OIAPR to resolve the discrepancies. Further, we visited TSA
headquarters to review TSA's annual recertification testing modules and
discuss TSA's process for validating the recertification exams. As a
result, we determined that the data provided by TSA were sufficiently
reliable for the purposes of our review. We also reviewed TSA's
performance measures, targets, and indexes. Finally, we interviewed TSA
headquarters officials from several offices in Arlington, Virginia,
including Aviation Operations, Workforce Performance and Training,
Strategic Management and Analysis, and Internal Affairs and Program
Review.
In addition, in accomplishing our objectives, we also conducted site
visits at select airports nationwide to interview Federal Security
Directors and their staffs and conducted two Web-based surveys of
Federal Security Directors. Specifically, we conducted site visits at
29 airports (13 category X airports, 9 category I airports, 3 category
II airports, 3 category III airports, and 1 category IV airport) to
observe airport security screening procedures and discuss issues
related to the screening process with TSA, airport, and airline
officials. We chose these airports to obtain a cross-section of all
airports by size and geographic distribution. In addition, we selected
each of the five contract screening pilot airports. The results from
our airport visits provide examples of screening operations and issues
but cannot be generalized beyond the airports visited because we did
not use statistical sampling techniques in selecting the airports. The
category X airports we visited were Baltimore Washington International
Airport, Boston Logan International Airport, Chicago O'Hare
International Airport, Dallas/Fort Worth International Airport, Denver
International Airport, Washington Dulles International Airport, John F.
Kennedy International Airport, Los Angeles International Airport,
Newark Liberty International Airport, Orlando International Airport,
Ronald Reagan Washington National Airport, San Francisco International
Airport, Seattle-Tacoma International Airport. The category I airports
we visited were Burbank-Glendale-Pasadena Airport, John Wayne Airport,
Chicago Midway International Airport, Dallas Love Field, Kansas City
International Airport, Little Rock National Airport, Metropolitan
Oakland International Airport, Portland International Airport, and
Tampa International Airport. The category II airports we visited were
Jackson International Airport, Dane County Regional Airport, and
Greater Rochester International Airport. The category III airports we
visited were Idaho Falls Regional Airport, Jackson Hole Airport, and
Orlando Sanford International Airport. The category IV airport we
visited was Tupelo Regional Airport.
Further, we administered two Web-based surveys to all 155 Federal
Security Directors who oversee security at each of the airports falling
under TSA's jurisdiction. One survey, the general survey, contained
questions covering local and national efforts to train screeners and
supervisors and the status of TSA's efforts to evaluate screener
performance, including the annual recertification program and TIP. The
second survey attempted to gather more specific airport security
information on an airport(s) under the Federal Security Director's
supervision. For the airport-specific survey, each Federal Security
Director received one or two surveys to complete, depending on the
number of airports they were responsible for. Where a Federal Security
Director was responsible for more than two airports, we selected the
first airport based on the Federal Security Director's location and the
second airport to obtain a cross-section of all airports by size and
geographic distribution. In all, we requested information on 265
airports. However, two airports were dropped from our initial selection
because the airlines serving these airports suspended operations and
TSA employees were redeployed to other airports. As a result our sample
size was reduced to 263 airports, which included all 21 category X, and
60, 49, 73, and 60 category I through IV airports respectively. In that
we did not use probability sampling methods to select the sample of
airports that were included in our airport-specific survey, we cannot
generalize our findings beyond the selected airports.
A GAO survey specialist designed the surveys in combination with other
GAO staff knowledgeable about airport security issues. We conducted
pretest interviews with six Federal Security Directors to ensure that
the questions were clear, concise, and comprehensive. In addition, TSA
managers and an independent GAO survey specialist reviewed the survey.
We conducted these Web-based surveys from late March to mid-May 2004.
We received completed general surveys from all 155 Federal Security
Directors and completed airport-specific surveys for all 263 separate
airports for which we sought information, for 100 percent response
rates. The surveys' results are not subject to sampling errors because
all Federal Security Directors were asked to participate in the surveys
and we did not use probability-sampling techniques to select specific
airports. However, the practical difficulties of conducting any survey
may introduce other errors, commonly referred to as nonsampling errors.
For example, difficulties in how a particular question is interpreted,
in the sources of information that are available to respondents, or in
how the data are entered into a database or were analyzed can introduce
unwanted variability into the survey results. We took steps in the
development of the surveys, the data collection, and the data editing
and analysis to minimize these nonsampling errors. Also, in that these
were Web-based surveys whereby respondents entered their responses
directly into our database, there was little possibility of data entry
or transcription error. In addition, all computer programs used to
analyze the data were peer reviewed and verified to ensure that the
syntax was written and executed correctly.
[End of section]
We performed our work from May 2003 through April 2005 in accordance
with generally accepted government auditing standards. Certain
information we obtained and analyzed regarding screener training and
performance are classified or are considered by TSA to be sensitive
security information. Accordingly, the results of our review of this
information are not included in this report.[Footnote 59]
[End of section]
Appendix III: TSA Screener Training Tools Designed to Help Improve
Screener Performance:
Training tool: Hand Held Metal Detector and Pat Down Video;
Purpose: Provide an informative and effective learning tool to enhance
screeners' skills in the areas of hand-wanding and pat-down searches of
passengers;
Status reported by TSA: Deployed November 21, 2003.
Training tool: MBS II Weapons Kits;
Purpose: This tool allows screeners to touch actual improvised
explosive device (IED) components and build their own devices. This
experiential learning will enable screeners to more readily detect real
IEDs during screening. These weapons are also used to assist in
training by using them for live testing conducted by FSD staff;
Status reported by TSA: Deployed January 26, 2004.
Training tool: Firearms Weapons Kits;
Purpose: This tool allows screeners to touch actual firearms and begin
to understand how they can be broken down into various parts. By
understanding this and experiencing it, screeners are better able to
see the components of a firearm during actual screening. These weapons
are also used to assist in training by using them for live testing
conducted by FSD staff;
Status reported by TSA: Deployed January 26, 2004.
Training tool: X-Ray Operator Video;
Purpose: Maintain and enhance the screeners' X-ray image operational
skills;
Status reported by TSA: Deployed February 5, 2004.
Training tool: X-Ray Tutor Version 1;
Purpose: Provide a tool that includes about 14,000 image combinations
to practice threat identification;
Status reported by TSA: Deployed February 15, 2004.
Training tool: Basic Supervisory Technical Web-Based Training 1;
Purpose: Assist screener supervisors to understand additional roles
they must perform, including enhanced technical skills needed to
adequately supervise screening functions and resolve alarms using
interactive, performance-based training tool;
Status reported by TSA: Deployed April 16, 2004.
Training tool: Mobile Training Assist Teams (MTAT);
Purpose: These teams go into airports where data shows performance
needs attention. The team offers a variety of services to assist in
improving the performance, such as on-the-spot training and consulting
services. Team visits can be initiated by FSDs, Internal Affairs
reports, Quality Assurance trips, or MTAT Supervisors proactively
visiting the airport and FSD;
Status reported by TSA: Site visits completed from October 2003 through
December 3, 2004:
* North Central (37 visits);
* South Central (51 visits);
* Northeast (25 visits);
* Southeast (60 visits);
* Western (53 visits);
* 54 FSD Stakeholder Relations Meetings.
Training tool: Basic Screener Supervisory Classroom Training;
Purpose: Improve screener supervisors' knowledge of federal government
and TSA personnel rules and how to effectively coach and communicate
with employees;
Status reported by TSA: Approximately 3,800 supervisors have been
trained.
Training tool: ETD Maintenance Course;
Purpose: Certification of screeners to perform supervisory maintenance
tasks above and beyond operator training;
Status reported by TSA: Delivered April 1, 2004.
Training tool: Credential Verification Training;
Purpose: Provide students with basic skills needed to verify the
identity of flying armed law enforcement officers;
Status reported by TSA: Deployed April 15, 2004.
Training tool: Threat in the Spotlight;
Purpose: This weekly product brings to light actual cases of weapons
being found by law enforcement, with an explanation of how those
weapons could be used to attack aviation;
Status reported by TSA: Deployed weekly in 2004.
Training tool: EDS Operations Web-Based Training;
Purpose: Provide interactive, performance based recurrent Web-based
training modules for checked baggage explosive detection systems (EDS);
Status reported by TSA: Deployed April 30, 2004.
Training tool: Handwanding and Pat Down Web-Based Training;
Purpose: Improve screener performance by providing an interactive tool
complementary to Hand Held Metal Detector and Pat Down Video that
allows the screener to practice proper techniques and receive immediate
feedback;
Status reported by TSA: Deployed April 30, 2004.
Training tool: Customer Service Web-Based Training;
Purpose: Reinforces TSA's customer service principles and places the
screener in various situations requiring effective customer service
responses;
Status reported by TSA: Deployed April 30, 2004.
Training tool: Checkpoint and Checked Baggage Operations Web-Based
Training;
Purpose: Provide interactive, performance-based recurrent training
modules for checkpoint and checked baggage operations;
Status reported by TSA: Deployed April 30, 2004.
Training tool: Physical Bag Search Video;
Purpose: Maintain and enhance screeners' explosive trace detection
(ETD) and physical bag search skills for carry-on and checked baggage;
Status reported by TSA: Deployed April 30, 2004.
Training tool: ETD and Physical Bag Search Web-Based Training;
Purpose: Provide interactive recurrent Web-based training modules for
ETD and physical bag search;
Status reported by TSA: Deployed April 30, 2004.
Training tool: Prohibited Items Web-Based Training;
Purpose: Provide an interactive, performance-based training tool to
enhance screener's ability to identify prohibited items;
Status reported by TSA: Deployed June 25, 2004.
Training tool: Effectively Screening Prosthetics Video;
Purpose: Provide an informative and effective learning tool to maintain
and enhance the skills of screeners in the areas of persons with
prosthetics;
Status reported by TSA: Deployed December 16, 2004.
Training tool: X-Ray Tutor Version 2;
Purpose: Provide a tool to practice threat identification with about
10,000,000 image combinations;
Status reported by TSA: Scheduled to be deployed during the second
quarter of fiscal year 2005.
Training tool: X-Ray Simulator Training;
Purpose: Sharing the X-Ray Tutor Version 2 library, this tool will
allow screeners to practice finding threat items using the full
capabilities of the TIP-ready X-ray machines;
Status reported by TSA: Scheduled to be deployed during the second
quarter of fiscal year 2005.
Training tool: Basic Supervisory Technical Web-Based Training 2;
Purpose: Provide an interactive, performance-based tool to convey how
the supervisor is to handle screening situations, handed off by the
screening, following standard operator procedures;
Status reported by TSA: To be developed.
Training tool: Safety, Lifting, and Twisting Web-Based Training;
Purpose: Provide a Web-based training that will engage the student with
3-dimensional representations of the muscular frame, showing proper
lifting techniques and the results of improper techniques;
Status reported by TSA: Scheduled to be deployed during the second
quarter of fiscal year 2005.
Source: TSA.
[End of table]
[End of section]
Appendix IV: Summary of TSA's Short-Term Action Items for Strengthening
Passenger Screener Performance:
People: 1;
Action item: Increase FSD support and accountability;
Description: Hold FSDs accountable for screening performance and
delivery of security;
Benefit: Management accountability is driven down to the local airport;
FSD performance is linked to screener performance, creating incentives
for maintaining and improving security.
People: 2;
Action item: Enhance training;
Description: Provide ongoing training for screeners and supervisors to
maintain their skills and provide new skills and techniques based on
evolving threats and lessons learned;
Benefit: Maintains and improves knowledge base of screeners; Ensures
proper oversight by supervisors; Ensures that screeners are capable of
addressing evolving threats.
People: 3;
Action item: Increase Internal Affairs covert testing;
Description: Increase the frequency of TSA covert testing;
Benefit: Improved identification of systemic vulnerabilities in airport
security systems; Immediate implementation of limited remedial actions.
People: 4;
Action item: Continue to pursue human performance improvements;
Description: Better understand reasons and causes for human errors and
interactions with technology in order to identify opportunities for
performance improvements, with a goal of identifying optimum work
conditions;
Benefit: Reduces human-based errors; Increases workforce morale and
working conditions, leading to improved performance.
Technology: 5;
Action item: Continue to identify screening technology improvements;
Description: Continue to research alternative technologies and seek
short-term technological solutions, especially for potential vectors;
Benefit: Identifies threats more accurately and quickly; Decreases
number of false positives from equipment.
Technology: 6;
Action item: Finish installing TIP;
Description: The TIP system is a series of 2,400 images of threat
objects that can be automatically fed into X-ray machines during actual
screening;
Benefit: Maintains alertness of screeners; Identifies individual
screener performance issues.
Technology: 7;
Action item: Expedite high-speed connectivity to checkpoints and
training computers;
Description: Connect all TSA offices, checkpoints and screening
equipment (X-rays, EDS machines) to the Internet in order to automate
and improve processes that are currently done manually or not at all;
Benefit: Provides immediate feedback on and response to screener
performance issues; Improves communication with managers in the field.
Process: 8;
Action item: Refresh aviation operations policy, procedures, and
practice;
Description: Conduct a thorough and expedited review of all policies
and procedures developed during the rollout of TSA with a focus on
increasing screening performance and capabilities;
Benefit: Maintains "freshness" of standard operating procedures based
on most recent intelligence about security threats; Removes or updates
outdated or unnecessary screening techniques based on lessons learned.
Process: 9;
Action item: Improve workforce management;
Description: Determine the optimal workforce staffing levels based on
latest passenger flows and other factors;
Benefit: Maximizes utilization of existing resources.
Source: TSA.
[End of table]
[End of section]
Appendix V: Comments from the Department of Homeland Security:
U.S. Department of Homeland Security:
Washington, DC 20528:
February 4, 2005:
Ms. Cathleen Berrick:
Director, Homeland Security & Justice Issues:
U.S. Government Accountability Office:
441 G Street, N.W.:
Washington, D.C. 20548:
Dear Ms. Berrick:
RE: GAO-05-143SU, Aviation Security: Screener Training and Performance
Measurement Strengthened, but More Work Remains:
(GAO Job Code 440256):
Thank you for the opportunity to comment on the subject draft report.
The Department of Homeland Security (DHS) appreciates the work done to
identify areas for improvement in the Transportation Security
Administration's training and performance measurement of passenger and
baggage screeners. We generally concur with the report and its
recommendations and appreciate the discussion of challenges, and
related on-going and planned work designed to fully meet our
responsibilities.
The report acknowledges the substantial progress the Transportation
Security Administration (TSA) has made in enhancing Screener training,
and measuring and enhancing performance. However, there are areas
within the report about which DHS would like to comment.
The U.S. Government Accountability Office (GAO) made the following two
recommendations regarding passenger and baggage screening training and
performance:
1. Develop a plan that prioritizes and schedules the deployment of high-
speed internet/intranet connectivity to all TSA's airport training
facilities to help facilitate the delivery of Screener training and the
documentation of training completion.
2. Develop internal controls, such as specific directives, clearly
defining responsibilities for monitoring and documenting the completion
of required training, and clearly communicate these responsibilities
throughout [TSA].
DHS agrees that efforts to implement these recommendations are critical
to successful passenger and baggage screening training and performance.
We believe that we have fulfilled the first recommendation because TSA
has developed such a plan.
Specifically, TSA has developed a plan to facilitate connectivity to
all of TSA's airport training facilities. The High-Speed Operational
Connectivity (HiSOC) program is a detailed plan and corresponding
schedule for ensuring that training centers in airports receive high
speed connectivity. The $174 million of funding necessary to complete
HiSOC installation is included in the President's FY 2006 budget
request. To date in FY 2005, $3 million has been allocated to the HiSOC
effort. As additional funds are made available for the HiSOC program,
connectivity can be deployed to additional training centers.
The HiSOC program includes a detailed plan for Wide Area Network (WAN)
connectivity to TSA Airports including the following services:
* Local Area Networking (LAN) linking Operation Centers, Training
Centers and Break Rooms, Checkpoint/Passenger Screening Areas and
Baggage Screening Areas, and Federal Security Directors:
* XP Migration:
* Intelligent Phone Deployment:
* Email Migration:
* Remote Access via Virtual Private Network (VPN):
TSA is already taking steps to address the second recommendation. The
TSA Executive Leadership is drafting a memorandum to advise all
Assistant Administrators and Federal Security Directors that managers
and supervisors will be held accountable for their subordinates
completing all mandatory training requirements. This accountability
will be inserted into the performance plans of all TSA supervisors.
In February 2005, TSA's Office of Workforce Performance and Training
will be updating the existing Management Directive on Training Records.
This directive will be revised to strengthen and clarify recordkeeping
requirements. By the end of March 2005, the TSA Online Learning Center
(OLC) should be enhanced to include a robust reporting tool that will
produce a number of detailed and summary level training performance
accountability reports directly accessible to local training
administrators. By the summer 2005, management plans to expand TSA's
Performance Information Management System to include select OLC
training summary data. This data will be visible to managers and will
include the ability to correlate training performance data with other
TSA source data for cause and effect and trending analyses.
Sincerely,
Signed by:
Steven Pecinovsky:
Acting Director:
Departmental GAO/OIG Liaison Office:
MMcP:
[End of section]
Appendix VI: GAO Contacts and Acknowledgments:
GAO Contacts:
Cathleen A. Berrick (202) 512-8777:
Maria D. Strudwick (202) 512-5419:
Staff Acknowledgments:
In addition to those named above, David Alexander, Leo Barbour, Lisa
Brown, Elizabeth Curda, Kevin Dooley, Kathryn Godfrey, David Hooper,
Christopher Jones, Stuart Kaufman, Kim Gianopoulos, Thomas Lombardi,
Cady S. Panetta, Minette Richardson, Sidney Schwartz, Su Jin Yon, and
Susan Zimmerman were key contributors to this report.
FOOTNOTES
[1] The Federal Security Director is the ranking TSA authority
responsible for the leadership and coordination of TSA security
activities at the nation's commercial airports. We sent two surveys--a
general survey and an airport-specific survey--to all 155 Federal
Security Directors on March 23, 2004. In the general survey, we asked
each Federal Security Director to answer security-related questions
that pertain to all of the airports for which he/she is responsible. In
the airport-specific survey, we asked Federal Security Directors a
number of airport-specific questions about screening and other security
concerns for one or two airports, depending on the number of airports
they were responsible for. By early May 2004, we had received responses
from 100 percent of the Federal Security Directors for both surveys.
[2] Pursuant to section 108 of the Aviation and Transportation Security
Act (ATSA), TSA conducted a 2-year private screening pilot program at
five airports--one in each airport security category (Pub. L. No. 107-
71). The mission of the pilot, as defined by TSA, was to test the
effectiveness of using private screening contractors in a post-
September 11 environment. The pilot concluded on November 18, 2004. On
November 19, 2004, consistent with ATSA, TSA began allowing airports to
apply to opt out of using federal screeners in favor of private
contractors. For additional information on TSA's progress in developing
the opt-out program, see GAO, Aviation Security: Preliminary
Observations on TSA's Progress to Allow Airports to Use Private
Passenger and Baggage Screening Services, GAO-05-126 (Washington, D.C.:
Nov. 19, 2004).
[3] GAO, Standards for Internal Control in the Federal Government,
GAO/AIMD-00-21.3.1 (Washington, D.C.: November 1999).
[4] The Threat Image Projection system is designed to test screeners'
detection capabilities by projecting threat images, including images of
guns and explosives, into bags as they are screened. Screeners are
responsible for positively identifying the threat image and calling for
the bag to be searched.
[5] ATSA requires that TSA collect performance information on all
passengers and baggage screeners by conducting an annual proficiency
evaluation to ensure each screener continues to meet all qualifications
and standards related to the functions he or she performs.
[6] TSA's performance indexes show how well screening systems are
functioning on a scale of 1 to 5, using a weighted average of the
values of four distinct performance indicators.
[7] We issued two additional reports detailing the results of our
review, which discuss information deemed to be classified or sensitive
security information. The report containing sensitive security
information is GAO, Aviation Security: Screener Training and
Performance Measurement Strengthened, but More Work Remains, GAO-05-
143SU (Washington D.C.: Feb. 28, 2005). The report containing
classified and sensitive security information is GAO, Aviation
Security: Results of Transportation Security Administration's Covert
Testing for Passenger and Checked Baggage Screening for September 2002
through September 2004, GAO-05-437C (Washington D.C.: Apr. 7, 2005).
[8] The National Commission on Terrorist Attacks Upon the United
States: The 9/11 Commission Report (Washington, D.C., July 2004).
[9] Consistent with ATSA, TSA was created as an agency within the
Department of Transportation (DOT) with responsibility for securing all
modes of transportation, including aviation. The Homeland Security Act
of 2002, signed into law on November 25, 2002, transferred TSA from the
DOT to the new Department of Homeland Security (Pub. L. No. 107-296).
[10] TSA defines an operational screening test as any covert test of a
screener conducted by TSA, on any screening function, to assess the
screener's threat item detection ability and/or adherence to TSA-
approved procedures.
[11] Pursuant to the Homeland Security Act, the deadline for screening
all checked baggage using explosive detection systems was, in effect,
extended until December 31, 2003.
[12] Sterile areas are located within the terminal where passengers
wait after screening to board departing aircraft. Access to these areas
is controlled by TSA screeners at checkpoints where they conduct
physical screening of passengers and their carry-on baggage for weapons
and explosives.
[13] Screeners must deny passage beyond the screening location to any
individual or property that has not been screened or inspected in
accordance with passenger screening standard operating procedures. If
an individual refuses to permit inspection of any item, that item must
not be allowed into the sterile area or aboard an aircraft.
[14] Explosive detection systems use probing radiation to examine
objects inside baggage and identify the characteristic signatures of
threat explosives. EDS equipment operates in an automated mode.
[15] Explosive trace detection works by detecting vapors and residues
of explosives. Human operators collect samples by rubbing bags with
swabs, which are chemically analyzed to identify any traces of
explosive materials.
[16] Positive passenger bag match is an alternative method of screening
checked baggage, which requires that the passenger be on the same
aircraft as the checked baggage.
[17] GAO, Airport Passenger Screening: Preliminary Observations on
Progress Made and Challenges Remaining, GAO-03-1173 (Washington, D.C.:
Sept. 24, 2003).
[18] The Online Learning Center is TSA's central, official electronic
source of all training and related performance/development
accomplishments for TSA employees. It serves as the delivery platform
for online training and is the official repository for TSA training
records. TSA launched the Online Learning Center on October 31, 2003.
[19] Department of Homeland Security, Office of Inspector General: An
Evaluation of the Transportation Security Administration's Screener
Training and Methods of Testing, OIG-04-045 (Washington, D.C., Sept.
2004).
[20] For local employees to be approved as instructors by TSA, they
must be nominated by an FSD and have a current or prior instructor
certification by a recognized training and development organization or
have had at least 2 years of experience as an instructor. In addition,
local TSA instructors must have successfully completed the course of
instruction they will be teaching and demonstrate instructional skills
by assisting a TSA-approved instructor in classroom instruction and
monitoring actual classroom instruction.
[21] In April 2004, a TSA-sponsored independent evaluation of screening
operations was completed by a consultant (Private Screening Operations
Performance Evaluation Summary Report, BearingPoint, Apr. 16, 2004).
The study was designed to evaluate the performance of federal and
private screening operations.
[22] The MBS II and weapons training kits were provided to airports to
address the identified training gap by allowing screeners to see and
feel the threat objects that they are looking for. These kits contain
some of the test objects used by TSA's OAIPR to conduct the covert
testing. In February 2004, TSA issued guidance to FSDs on use of these
kits to conduct local screener testing. These guidelines were updated
in June 2004.
[23] Department of Homeland Security Office of Inspector General, Audit
of Passenger and Baggage Screening Procedures at Domestic Airports, OIG-
04-37 (Washington, D.C., September 2004).
[24] TSA headquarters training officials stated that the Training
Coordinators at airports should be aware of the availability of the
supervisory and leadership training courses.
[25] GAO, Human Capital: A Guide for Assessing Strategic Training and
Development Efforts in the Federal Government, GAO-04-546G (Washington,
D.C., March 2004).
[26] Methods used by FSDs for determining and reporting recurrent
training time have not been verified by GAO.
[27] The staffing model took into account factors such as the number of
screening checkpoints and lanes at an airport; originating passengers;
projected air carrier service increases and decreases during calendar
year 2003; and hours needed to accommodate some screener training,
leave, and breaks.
[28] In May 2003, TSA hired a contractor to develop a staffing model
for its screening workforce. TSA officials told us that the contractor
completed the staffing model in June 2004, and all airports now have
the capability to use the contractor's stand-alone software. TSA
completed installation of the software on its intranet in March 2005.
This installation provides TSA headquarters with access to the staffing
models used by airports. The Intelligence Reform and Terrorism
Prevention Act of 2004 requires TSA to develop and submit to the
appropriate congressional committees, standards for determining
aviation security staffing at commercial airports no later than 90 days
after December 17, 2004, the date of the act's enactment, and GAO to
conduct an analysis of these standards (Pub. L. No. 108-458). As of
April 15, 2005, these standards had not been submitted.
[29] High-speed Internet/intranet access is provided by a series of
technologies that give users the ability to send and receive data at
volumes and speeds far greater than Internet access over traditional
telephone lines. In addition to offering speed, the technology provides
a continuous, "always on" connection (no need to dial up) and a "two-
way" capability, that is, the ability to both receive and transmit data
at high speeds.
[30] TSA defines fully connected as a training computer with the new
network image installed and connected to the TSA broadband network.
[31] GAO/AIMD-00-21.3.1.
[32] GAO/AIMD-00-21.3.1.
[33] GAO-AIMD-00-21.3.1.
[34] Consistent with ATSA, TSA requires remedial training for any
passenger or baggage screener who fails an operational (covert) test
and prohibits screeners from performing the screening function related
to the test they failed until they successfully complete the training.
[35] According to TSA officials, between September 2002 and December 8,
2004, the Office of Internal Affairs and Program Review issued 12
memorandums to Aviation Operations that identified screeners requiring
remedial training based on covert testing conducted from September 2002
through September 30, 2004.
[36] OIAPR designs its covert testing methods based, in part, on
intelligence regarding the most recent threats.
[37] Statement before the National Commission on Terrorist Attacks upon
the United States, Statement of the Honorable Kenneth M. Mead,
Inspector General, U.S. Department of Transportation, May 22, 2003.
[38] Covert testing is an ancillary duty and not a full-time assignment
for the majority of OIAPR staff. According to OIAPR, of the
approximately 123 full-time-equivalent positions in headquarters, 14
are dedicated fully to the covert testing program, which includes
covert testing of all modes of transportation, not just airports. These
14 full-time-equivalents are in OIAPR's Special Operations group and
form the core of team leaders for the covert testing trips. In
addition, two full-time-equivalents from OIAPR's Office of Program
Analysis support the covert testing program full-time in terms of data
analysis, report writing, and quality assurance. The remaining OIAPR
staff in headquarters and the field are responsible for conducting
criminal and noncriminal investigations of employee misconduct;
conducting program reviews, inspections, and special inquiries into
security incidents; and managing OIAPR.
[39] TSA established five mission support centers staffed with OIAPR,
training, and other personnel. These centers are located in Atlanta,
Dallas, Detroit, Philadelphia, and San Francisco. In fiscal year 2004,
OIAPR began to use field staff to support its covert testing
activities. In addition, OIAPR has 12 investigators located at seven
airports.
[40] GAO, Airport Passenger Screening: Preliminary Observations on
Progress Made and Challenges Remaining, GAO-03-1173 (Washington, D.C.:
Sept. 24, 2003). GAO, Aviation Security: Private Screening Contractors
Have Little Flexibility to Implement Innovative Approaches, GAO-04-544T
(Washington, D.C.: Apr 22, 2004).
[41] Results of TSA's covert testing of passenger and checked baggage
screening are classified and are discussed in a separate classified
report, GAO, Results of Transportation Security Administration's Covert
Testing for Passenger and Checked Baggage Screening for September 2002
through September 2004, GAO-05-437C (Washington, D.C.: Apr. 5, 2005).
[42] Test results cannot be generalized because sample tests were not
identified using the principles of probability sampling. In a
probability sample to assess screener detection of threat objects, each
screening of a passenger or baggage would have to have a chance of
being selected. A well-designed probability sample would enable failure
rates to be generalized to all airports. However, for cost and
operational reasons, probability sampling may not be feasible for
passenger and checked baggage screening because it would require a very
large sample size and an exhaustive examination of each sampled
passenger or baggage to determine if there was a threat object to
detect.
[43] Test failure rates are classified and have been excluded from this
report.
[44] The local covert testing protocols were updated in June 2004 and
August 2004 to provide information on alternative testing methods.
[45] TIP is not yet operational at one airport because of construction
at the screening checkpoint to prepare for its installation. However,
the TIP-ready X-ray machines have already been procured for the airport
and are expected to be installed once the construction issues have been
resolved.
[46] The TIP database records both the TIP hit rate and TIP false alarm
rate. These two results are used to determine the probability of
detection and probability of false alarm, which determine overall TIP
performance.
[47] GAO-03-1173.
[48] Some screeners, such as those on extended leave, leave without
pay, military leave, or leave because of an on-the-job injury were not
tested. These screeners are retested as they return to work.
[49] According to TSA records, between October 2003 and May 2004,
13,516 screeners completed the recertification testing as baggage
screeners.
[50] As of January 7, 2005, TSA reported that its workforce included
approximately 25,947 dual-trained screeners who were certified to serve
as passenger or baggage screeners.
[51] TSA's onscreen resolution protocol requires that when an EDS
machine alarm goes off, indicating the possibility of explosives, TSA
screeners, by reviewing computer-generated images of the inside of the
bag, attempt to determine whether or not a suspect item or items are in
fact explosive materials. If the screener is unable to make this
determination, the bag is diverted from the main conveyor belt into an
area where it receives a secondary screening by a screener with an ETD
machine.
[52] TSA officials stated that in early fiscal year 2005, TSA's Office
of Information Technology committed to extending high-speed
connectivity to an additional 16 locations, but is still awaiting
funding to further expand network coverage and to extend the network to
checkpoints.
[53] OIAPR has issued its reports to the TSA Administrator; the TSA
Administrator's Chief of Staff; Associate Undersecretary for Aviation
Operations; Associate Undersecretary for Workforce, Performance and
Training; Assistant Secretary and Chief Technology Officer; Assistant
Administrator and Chief Support Systems Officer; Chief Operating
Officer, the Office of Transportation Security and Intelligence, and
the Office of Security Policy. The report recommendations are directed
to the office(s) responsible for taking the corrective action.
[54] Some recommendations appear repeatedly in multiple reports issued
by OIAPR.
[55] According to the Government Performance and Results Act, the
Office of Management and Budget, and GAO, outcomes assess actual
results as compared with the intended results or consequences that
occur from carrying out a program or activity. Outputs count the goods
and services produced by a program or organization.
[56] TSA analyzed data from fiscal years 2003 and 2004 to establish
baselines and establish performance targets.
[57] According to TSA, the machine probabilities of detection are
established by the certification standards for each particular model of
machines, and machines are not deployed unless they have met those
standards.
[58] TSA headquarters officials stated that their intent is to provide
FSDs with various mechanisms to assess their screening effectiveness at
the airports for which they are responsible. Specifically, they stated
that FSDs are provided with TIP data and the results of OIAPR's covert
testing at their airports. Additionally, they have access to the
results of local covert testing at their airports.
[59] We issued two additional reports detailing the results of our
review, which discuss information deemed to be classified or sensitive
security information. The report containing sensitive security
information is GAO, Aviation Security: Screener Training and
Performance Measurement Strengthened, but More Work Remains, GAO-05-
143SU (Washington D.C.: Feb. 28, 2005). The report containing
classified and sensitive security information is GAO, Results of
Transportation Security Administration's Covert Testing for Passenger
and Checked Baggage Screening for September 2002 through September
2004, GAO-05-437C (Washington D.C.: Apr. 7, 2005).
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
