GAO Review of the Department of Homeland Security's Certification of the Secure Flight Program--Cost and Schedule Estimates
Gao ID: GAO-10-535R April 5, 2010The matching of airline passenger information against terrorist watchlist records (watchlist matching) is a frontline defense against acts of terrorism that target the nation's civil aviation system. In general, passengers identified as matches to the No-Fly list are prohibited from boarding commercial flights, while those matched to the Selectee list are required to undergo additional screening. Historically, airline passenger prescreening against watchlist records has been performed by commercial air carriers. As required by the Intelligence Reform and Terrorism Prevention Act of 2004, the Department of Homeland Security's (DHS) Transportation Security Administration (TSA) has developed an advanced passenger prescreening program--known as Secure Flight--to assume from air carriers the function of matching passenger information against terrorist watchlist records. Since fiscal year 2004, TSA has received $358 million in appropriated funds for the development and implementation of Secure Flight, according to program officials. Also, since fiscal year 2004, GAO has been mandated to assess the development and implementation of the Secure Flight program. We have reported on numerous challenges the program has faced, including those related to protecting passenger privacy, completing performance testing, fully defining and testing security requirements, and establishing reliable cost and schedule estimates, among other things. We have made recommendations to address these challenges, and TSA has generally agreed with them and has taken corrective actions.
TSA has generally achieved the statutory condition related to the appropriateness of Secure Flight's life-cycle cost and schedule estimates, and thus has generally achieved all 10 statutory conditions related to the development and implementation of the program. Although the program's cost and schedule estimates do not fully meet all related best practices, TSA has demonstrated that it completed all key activities and our overall assessment found that the agency had substantially satisfied best practices for developing the cost and schedule estimates. According to TSA, the Secure Flight program's estimated life-cycle cost is $1.36 billion through fiscal year 2020. TSA plans to complete assumption of the watchlist-matching function from air carriers for all domestic flights in May 2010 and to assume this function for all international flights departing to and from the United States by December 2010. If effectively maintained and updated, TSA's cost and schedule estimates should help ensure oversight and accountability of the Secure Flight program and provide assurance that it will be delivered within estimated costs and time frames.
GAO-10-535R, GAO Review of the Department of Homeland Security's Certification of the Secure Flight Program--Cost and Schedule Estimates
This is the accessible text file for GAO report number GAO-10-535R
entitled 'GAO Review of the Department of Homeland Security's
Certification of the Secure Flight Program--Cost and Schedule
Estimates' which was released on April 6, 2010.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
GAO-10-535R:
United States Government Accountability Office:
Washington, DC 20548:
April 5, 2010:
Congressional Committees:
Subject: GAO Review of the Department of Homeland Security's
Certification of the Secure Flight Program--Cost and Schedule
Estimates:
The matching of airline passenger information against terrorist
watchlist records (watchlist matching) is a frontline defense against
acts of terrorism that target the nation's civil aviation system. In
general, passengers identified as matches to the No-Fly list are
prohibited from boarding commercial flights, while those matched to
the Selectee list are required to undergo additional
screening.[Footnote 1] Historically, airline passenger prescreening
against watchlist records has been performed by commercial air
carriers.
As required by the Intelligence Reform and Terrorism Prevention Act of
2004, the Department of Homeland Security's (DHS) Transportation
Security Administration (TSA) has developed an advanced passenger
prescreening program--known as Secure Flight--to assume from air
carriers the function of matching passenger information against
terrorist watchlist records.[Footnote 2] Since fiscal year 2004, TSA
has received $358 million in appropriated funds for the development
and implementation of Secure Flight, according to program officials.
Also, since fiscal year 2004, GAO has been mandated to assess the
development and implementation of the Secure Flight program.[Footnote
3] We have reported on numerous challenges the program has faced,
including those related to protecting passenger privacy, completing
performance testing, fully defining and testing security requirements,
and establishing reliable cost and schedule estimates, among other
things.[Footnote 4] We have made recommendations to address these
challenges, and TSA has generally agreed with them and has taken
corrective actions.
Section 522(a) of the Department of Homeland Security Appropriations
Act, 2005, set forth 10 conditions related to the development and
implementation of the Secure Flight program that the Secretary of
Homeland Security must certify have been successfully met before the
program may be implemented or deployed on other than a test basis.
[Footnote 5] Although DHS certified that it had satisfied all 10
conditions in September 2008, our initial assessment found that TSA
generally had not achieved 5 of the 10 statutory conditions and that
the agency had not demonstrated Secure Flight's operational readiness.
In response, TSA took additional actions and, in late January 2009, we
found that the agency had generally achieved 6 of the 10 conditions,
conditionally achieved 3 conditions--subject to the timely completion
of planned activities--and generally had not achieved 1 condition. We
also concluded that the actions TSA had taken were sufficient to
support beginning Secure Flight initial operations. We continued to
review the program and reported in May 2009 that TSA had generally
achieved 9 of the 10 statutory conditions and had conditionally
achieved 1 condition, subject to the timely completion of planned
activities for developing appropriate cost and schedule estimates.
[Footnote 6] Table 1 shows the status of the 10 conditions as of April
2009.
Table 1: GAO Assessment of Whether DHS Had Achieved the 10 Statutory
Conditions, as of April 2009:
Legislative condition topic: System of due process (redress);
Generally achieved[A]: [Check];
Conditionally achieved[B]: [Empty];
Generally not achieved[C]: [Empty].
Legislative condition topic: Extent of false-positive errors;
(misidentifications);
Generally achieved[A]: [Check];
Conditionally achieved[B]: [Empty];
Generally not achieved[C]: [Empty].
Legislative condition topic: Performance of stress testing and;
efficacy and accuracy of search tools;
Generally achieved[A]: [Check];
Conditionally achieved[B]: [Empty];
Generally not achieved[C]: [Empty].
Legislative condition topic: Establishment of an internal oversight;
board;
Generally achieved[A]: [Check];
Conditionally achieved[B]: [Empty];
Generally not achieved[C]: [Empty].
Legislative condition topic: Operational safeguards to reduce abuse;
opportunities;
Generally achieved[A]: [Check];
Conditionally achieved[B]: [Empty];
Generally not achieved[C]: [Empty].
Legislative condition topic: Substantial security measures to prevent;
unauthorized access by hackers;
Generally achieved[A]: [Check];
Conditionally achieved[B]: [Empty];
Generally not achieved[C]: [Empty].
Legislative condition topic: Effective oversight of system use and;
operation;
Generally achieved[A]: [Check];
Conditionally achieved[B]: [Empty];
Generally not achieved[C]: [Empty].
Legislative condition topic: No specific privacy concerns with the;
system's technological architecture;
Generally achieved[A]: [Check];
Conditionally achieved[B]: [Empty];
Generally not achieved[C]: [Empty].
Legislative condition topic: Accommodation of states with unique;
transportation needs;
Generally achieved[A]: [Check];
Conditionally achieved[B]: [Empty];
Generally not achieved[C]: [Empty].
Legislative condition topic: Appropriateness of life-cycle cost;
estimates and program plans;
Generally achieved[A]: [Empty];
Conditionally achieved[B]: [Check];
Generally not achieved[C]: [Empty].
Source: GAO analysis.
[A] For generally achieved, TSA had demonstrated that it completed all
key activities related to the condition in accordance with applicable
federal guidelines and related best practices, which should reduce the
risk of the program experiencing cost, schedule, or performance
shortfalls.
[A] For conditionally achieved, TSA had completed some key activities
and had defined plans for completing remaining activities that if
effectively implemented as planned, should result in a reduced risk of
the program experiencing cost, schedule, or performance shortfalls.
[C] For generally not achieved, TSA had not demonstrated that it
completed all key activities related to the condition in accordance
with applicable federal guidelines and related best practices and did
not have defined plans for completing the remaining activities, and
the uncompleted activities result in an increased risk of the program
experiencing cost, schedule, or performance shortfalls.
[End of table]
In our May 2009 report, we concluded that the actions TSA had
completed and those planned had reduced the risks associated with
implementing the program. We noted, however, that while TSA's ability
to fully achieve the statutory condition related to developing
appropriate cost and schedule estimates did not affect the Secure
Flight system's operational readiness, having reliable cost and
schedule estimates would allow for better insight into the management
of program resources and time frames as the program is deployed. DHS
concurred with our assessment and noted that TSA would continue to
work on the Secure Flight program's cost and schedule estimates until
the statutory condition is generally achieved.
The Conference Report accompanying the Department of Homeland Security
Appropriations Act, 2010, directed GAO to continue its review of the
Secure Flight program until all 10 statutory conditions are generally
achieved.[Footnote 7] In accordance with that mandate, this report
addresses the extent to which TSA met the Secure Flight condition
related to the appropriateness of cost and schedule estimates and any
shortfalls or limitations in meeting the requirements in GAO's Cost
Estimating and Assessment Guide.[Footnote 8]
Our overall methodology included (1) identifying key activities
related to the cost and schedule condition; (2) identifying federal
guidance and related best practices that are relevant to successfully
meeting the condition; (3) analyzing whether TSA has demonstrated
through verifiable analysis and documentation, as well as oral
explanation, that the guidance has been followed and best practices
have been met; and (4) assessing any risks associated with not fully
following applicable guidance and meeting best practices.
Specifically, we reviewed the program's life-cycle cost estimate,
integrated master schedule,[Footnote 9] and other relevant
documentation against best practices, including those contained in our
Cost Estimating and Assessment Guide. We also interviewed key program
officials overseeing these activities and consulted with a scheduling
expert to identify risks to the integrated master schedule. We
assessed the status of TSA's efforts to develop appropriate life-cycle
cost estimates and program plans based on the agency's plan of action
developed in early 2009 and related documentation that TSA provided to
us through February 2010. The plan detailed the Secure Flight program
management office's proposed activities and time frames for addressing
weaknesses that we identified in the program's cost and schedule
estimates.
We conducted this performance audit from October 2009 to April 2010 in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.
Results in Brief:
TSA has generally achieved the statutory condition related to the
appropriateness of Secure Flight's life-cycle cost and schedule
estimates, and thus has generally achieved all 10 statutory conditions
related to the development and implementation of the program. Although
the program's cost and schedule estimates do not fully meet all
related best practices, TSA has demonstrated that it completed all key
activities and our overall assessment found that the agency had
substantially satisfied best practices for developing the cost and
schedule estimates, as shown in table 2. In general, GAO's methodology
for assessing a program's cost and schedule estimates is based on the
extent to which an agency has "satisfied" best practices for
developing the estimates. For purposes of this review, our assessment
that TSA had substantially satisfied best practices equates to the
agency generally achieving the statutory condition.
Table 2: GAO Final Assessment of Secure Flight Cost and Schedule
Estimates versus Best Practices, as of February 2010:
Reliable cost estimate:
Best practice[A]: Comprehensive - The estimate should include all
costs over the life of the program;
Extent satisfied: Partially satisfied[B];
Overall assessment: Substantially satisfied[C].
Best practice[A]: Well documented - The estimate should clearly define
all key program or system characteristics;
Extent satisfied: Satisfied[D];
Overall assessment: Substantially satisfied[C].
Best practice[A]: Accurate - The estimate should provide for results
that are unbiased and should not be overly conservative or optimistic;
Extent satisfied: Partially satisfied;
Overall assessment: Substantially satisfied[C].
Best practice[A]: Credible - The estimate should discuss any
limitations because of uncertainty surrounding data or assumptions;
Extent satisfied: Satisfied;
Overall assessment: Substantially satisfied[C].
Reliable schedule:
Best practice[A]: Capturing key activities - The schedule should
reflect all key government and contractor activities;
Extent satisfied: Partially satisfied;
Overall assessment: Substantially satisfied[C].
Best practice[A]: Sequencing key activities - The schedule should be
planned so that critical program dates can be met;
Extent satisfied: Substantially satisfied;
Overall assessment: Substantially satisfied[C].
Best practice[A]: Establishing the duration of key activities - The
schedule should realistically reflect how long each activity will take
to execute;
Extent satisfied: Substantially satisfied;
Overall assessment: Substantially satisfied[C].
Best practice[A]: Assigning resources to key activities - The schedule
should reflect what resources (e.g., labor and materials) are needed
to do the work;
Extent satisfied: Satisfied;
Overall assessment: Substantially satisfied[C].
Best practice[A]: Integrating key activities horizontally and
vertically - The schedule should sequentially link activities
horizontally and show relationships between tasks and subtasks
vertically;
Extent satisfied: Substantially satisfied;
Overall assessment: Substantially satisfied[C].
Best practice[A]: Establishing the critical path for key activities -
The scheduling software should identify the longest duration path
through the sequenced list of key activities;
Extent satisfied: Partially satisfied;
Overall assessment: Substantially satisfied[C].
Best practice[A]: Identifying the float time between key activities -
The schedule should identify the time that a predecessor activity can
slip before the delay affects successor activities;
Extent satisfied: Partially satisfied;
Overall assessment: Substantially satisfied[C].
Best practice[A]: Schedule risk analysis should be performed - The
schedule risk analysis should predict the level of confidence in
meeting a program's completion date;
Extent satisfied: Satisfied;
Best practice[A]: Distributing reserves to high-risk activities - The
schedule should include a buffer or reserve of extra time for
contingencies;
Extent satisfied: Substantially satisfied;
Overall assessment: Substantially satisfied[C].
Source: GAO analysis.
[A] Enclosure I contains additional information on each cost and
schedule best practice.
[B] For partially satisfied, TSA officials provided evidence that
satisfies about half of the criterion.
[C] For substantially satisfied, TSA officials provided evidence that
satisfies the majority of the criterion.
[D] For satisfied, TSA officials provided complete evidence that
satisfies the entire criterion.
[End of table]
According to TSA, the Secure Flight program's estimated life-cycle
cost is $1.36 billion through fiscal year 2020. TSA plans to complete
assumption of the watchlist-matching function from air carriers for
all domestic flights in May 2010 and to assume this function for all
international flights departing to and from the United States by
December 2010. If effectively maintained and updated, TSA's cost and
schedule estimates should help ensure oversight and accountability of
the Secure Flight program and provide assurance that it will be
delivered within estimated costs and time frames.
TSA Has Generally Achieved the Statutory Condition Related to
Appropriate Life-Cycle Cost and Schedule Estimates:
In May 2009, we reported that TSA had conditionally achieved the
statutory requirement related to Secure Flight's life-cycle cost and
schedule estimates, based on the agency's plan of action for
addressing weaknesses we identified. Since then, TSA has taken several
steps to improve these estimates and implement our prior
recommendations; thus, we now consider the legislative requirement to
be generally achieved.
Secure Flight's Life-Cycle Cost Estimate Substantially Satisfies GAO
Best Practices:
A reliable cost estimate is critical to the success of any program
since it provides the basis for informed investment decision making,
realistic budget formulation and program resourcing, meaningful
progress measurement, proactive course correction when warranted, and
accountability for results.
In our May 2009 report, we noted that Secure Flight's $1.36 billion
life-cycle cost estimate was well documented in that it clearly stated
the purpose, source, assumptions used, and calculations made. However,
it was not comprehensive (it did not include all costs), fully
accurate, or credible. As a result, the life-cycle cost estimate did
not provide a meaningful baseline from which to track progress, hold
TSA accountable, and provide a basis for sound investment decision
making. To address recommendations that were in the draft of our May
2009 report,[Footnote 10] TSA established a plan of action to, among
other things, (1) provide more detail in the work necessary to
accomplish the program's objectives, (2) properly align the cost
estimate with the schedule of work to be performed, (3) develop an
independent cost estimate performed by a contractor, (4) have the DHS
Cost Analysis Division assess the life-cycle cost estimate, and (5)
perform cost uncertainty and sensitivity analyses on the estimate.
[Footnote 11]
Over the past year, TSA has significantly improved the Secure Flight
program's life-cycle cost estimate and our overall assessment found
that the agency has substantially satisfied GAO best practices related
to the four characteristics of a reliable cost estimate--
comprehensive, well documented, accurate, and credible. For example,
in October 2009, a TSA contractor developed an independent cost
estimate for the program, which the agency used to validate the
credibility of the existing life-cycle cost estimate. TSA has also
conducted a cost uncertainty analysis and a sensitivity analysis on
the independent cost estimate.
The DHS Cost Analysis Division also reviewed the Secure Flight life-
cycle cost estimate in late 2009 and found that the estimate generally
met GAO best practices for cost estimating. One of the reasons the
estimate still did not fully meet GAO best practices was because the
estimate was based on a functional breakdown of the work to be
performed (the work breakdown structure, or WBS), instead of a product-
oriented structure. While functional activities--for example,
engineering or quality control--are necessary for supporting a
product's development, a WBS generally should not be organized around
them. Rather, best practices recommend that a product-oriented WBS be
used that reflects the cost, schedule, and technical performance on
specific portions of a program so that project performance can be
directly related to developed products. However, we recognize that it
is not feasible to reorganize the WBS for Secure Flight at this point
in the program, since the time to reorganize it would exceed the time
phase of the program and rebaselining the program--which would be
required for a new WBS--would be cost prohibitive.[Footnote 12] DHS
Cost Analysis Division officials stated that they expect the life-
cycle cost estimate to adhere to a product-oriented WBS for the next
major acquisition milestone, which is scheduled to be completed by
February 2011. While it is important for TSA to develop a product-
oriented WBS for Secure Flight, the other actions the agency has taken
are sufficient for us to conclude that TSA has substantially satisfied
GAO's best practices for developing a reliable cost estimate for the
program.
Enclosure I contains additional information on our final assessment of
Secure Flight's life-cycle cost estimate relative to GAO's best
practices.
Secure Flight's Program Schedule Substantially Satisfies GAO Best
Practices:
The success of any program depends in part on having a reliable
schedule specifying when the program's set of work activities will
occur, how long they will take, and how they relate to one another. As
such, the schedule not only provides a road map for the systematic
execution of a program, but it also provides the means by which to
gauge progress, identify and address potential problems, and promote
accountability.
In our May 2009 report, we noted that Secure Flight's schedule was
developed using some of GAO's best practices for developing schedules,
but several key practices were not fully employed that are fundamental
to having a schedule that provides a sufficiently reliable basis for
estimating costs, measuring progress, and forecasting slippages. In
addition, best practices require that a schedule identify the longest
duration path through the sequenced list of key activities--known as
the schedule's critical path--where if any activity slips along this
path, the entire program will be delayed. TSA's schedule did not
include a critical path, which could help program managers better
understand the effect of any delays. We also noted that updating the
Secure Flight program's schedule is important because of the
significant cost and time that remained to be incurred for TSA to
assume the watchlist-matching function for all domestic flights and to
develop, test, and deploy the functionality to assume the watchlist
matching function for international flights.
To address recommendations that were in the draft of our May 2009
report,[Footnote 13] TSA established a plan of action to develop,
among other things, (1) a sequenced and logical schedule to accurately
calculate float time and a critical path;[Footnote 14] (2) a schedule
that fully identifies the resources needed to complete key
activities;[Footnote 15] (3) a schedule that includes realistic
estimates of activity duration; and (4) a schedule risk analysis that
will be used by TSA leadership to distribute resources to high-risk
activities on the critical path, which if delayed would delay the
entire program. According to TSA, this revised schedule would better
forecast the completion date for the project.
Since we issued our May 2009 report, TSA has taken several steps to
improve the Secure Flight program's schedule, and our overall
assessment found that the agency has substantially satisfied GAO's
best practices related to the nine characteristics of a reliable
schedule estimate--capturing key activities, sequencing key
activities, establishing duration of key activities, assigning
resources to key activities, integrating key activities horizontally
and vertically,[Footnote 16] establishing a critical path, identifying
float time, performing a schedule risk analysis, and distributing
reserves to high-risk activities.[Footnote 17] For example, TSA has
assigned resources for each activity, conducted a schedule risk
analysis, and undertaken strategies to mitigate risks to the program
that could affect the schedule. To minimize the risk of schedule
delay, TSA prioritized the schedule for assuming the watchlist-
matching function from air carriers and reallocated staff resources.
In addition, TSA hired a contractor to complete an independent
schedule risk analysis and implemented some of the recommendations
made in that analysis. However, best practices for scheduling are not
fully satisfied, because scheduling constraints prevent the scheduling
software from automatically calculating the start and finish dates of
remaining activities. These constraints also prevent the software from
automatically creating a valid critical path, which hampers the
ability of decision makers to optimally allocate resources (e.g., move
resources from low-risk activities that are not on the critical path
to high-risk activities). However, the Secure Flight program office
has justified the use of these constraints, as the start dates for
activities related to TSA's assumption of the watchlist-matching
function from air carriers are legally binding between the Secure
Flight program office and the airlines.
On January 27, 2009, the Secure Flight program began initial
operations--assuming the watchlist-matching function for a limited
number of domestic flights for one airline--and has since phased in
additional flights and airlines. According to TSA, Secure Flight plans
to assume this function for a total of over 70 U.S. air carriers and
about 150 foreign carriers.[Footnote 18] As of March 31, 2010, TSA was
working with 74 U.S. air carriers and 19 foreign carriers.
Specifically,
* Secure Flight had fully assumed the watch-list matching function for
39 U.S. air carriers (domestic flights only), had partially assumed
this function for another 11 U.S. carriers,[Footnote 19] and was in
testing with another 24 U.S. carriers, and:
* Secure Flight had fully assumed the matching function for 5 foreign
air carriers (international flights departing to and from the United
States) and was in testing with another 14 foreign carriers.
TSA plans to complete assumption of the watchlist-matching function
from air carriers for all domestic flights in May 2010 and to assume
this function for all international flights departing to and from the
United States by December 2010, assuming the air carriers make the
necessary system changes as required to be compliant with the Secure
Flight Final Rule.[Footnote 20] Specifically, the rule contains
requirements for air carriers to follow as TSA implements and operates
Secure Flight, including the collection of full name and date-of-birth
information from airline passengers to facilitate watch-list matching.
To date, TSA has not experienced any unexpected challenges with
aircraft operators currently testing with Secure Flight that would
necessitate an extension to the schedule, according to program
officials.
Enclosure I contains additional information on our final assessment of
Secure Flight's schedule estimate relative to GAO's best practices.
Agency Comments:
On March 29, 2010, we provided a draft of this report to DHS for
comment. DHS did not provide written agency comments. However, TSA
provided technical comments, which we incorporated in this report
where appropriate. TSA also noted that it appreciated the assistance
we have provided in helping TSA meet Congress' 10 statutory conditions
related to the Secure Flight program, and our public recognition that
TSA has generally achieved the single outstanding statutory condition
related to the program's appropriate life-cycle cost and schedule
estimates.
We are sending copies of this report to the appropriate congressional
committees, the Secretary of Homeland Security, and other interested
parties. This report also is available at no charge on the GAO Web
site at [hyperlink, http://www.gao.gov].
Should you or your staff have any questions about this report, please
contact me at (202) 512-4379 or lords@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. Key contributors to this report were
Karen Richey, Assistant Director; Eric Erdman, Assistant Director;
Jason Lee; and Victoria Miller.
Signed by:
Stephen M. Lord:
Director, Homeland Security and Justice Issues:
Enclosures - 1:
List of Committees:
The Honorable John D. Rockefeller, IV:
Chairman:
The Honorable Kay Bailey Hutchison:
Ranking Member:
Committee on Commerce, Science, and Transportation:
United States Senate:
The Honorable Joseph I. Lieberman:
Chairman:
The Honorable Susan M. Collins:
Ranking Member:
Committee on Homeland Security and Governmental Affairs:
United States Senate:
The Honorable Patrick J. Leahy:
Chairman:
The Honorable Jeff Sessions:
Ranking Member:
Committee on the Judiciary:
United States Senate:
The Honorable Robert C. Byrd:
Chairman:
The Honorable George Voinovich:
Ranking Member:
Subcommittee on Homeland Security:
Committee on Appropriations:
United States Senate:
The Honorable Bennie G. Thompson:
Chairman:
The Honorable Peter T. King:
Ranking Member:
Committee on Homeland Security:
House of Representatives:
The Honorable Edolphus Towns:
Chairman:
The Honorable Darrell Issa:
Ranking Member:
Committee on Oversight and Government Reform:
House of Representatives:
The Honorable James L. Oberstar:
Chairman:
The Honorable John L. Mica:
Ranking Member:
Committee on Transportation and Infrastructure:
House of Representatives:
The Honorable David E. Price:
Chairman:
The Honorable Harold Rogers:
Ranking Member:
Subcommittee on Homeland Security:
Committee on Appropriations:
House of Representatives:
[End of section]
Enclosure I: GAO Analyses of Secure Flight's Life-Cycle Cost and
Schedule Estimates against Best Practices:
Our research has identified several best practices that serve as the
basis for effective program cost and schedule estimating.[Footnote 21]
Our assessments of the extent to which Secure Flight's cost and
schedule estimates satisfied best practices were based on the
following criteria:
* Not satisfied: Project officials provided no evidence that satisfies
any part of the criterion.
* Minimally satisfied: Project officials provided evidence that
satisfies less than half of the criterion.
* Partially satisfied: Project officials provided evidence that
satisfies about half of the criterion.
* Substantially satisfied: Project officials provided evidence that
satisfies the majority of the criterion.
* Satisfied: Project officials provided complete evidence that
satisfies the entire criterion.
Specifically, we have identified four characteristics of a reliable
cost estimate: comprehensive, well documented, accurate, and credible.
Table 3 summarizes the results of our final assessment of the Secure
Flight program's cost estimate relative to the four characteristics of
a reliable cost estimate, as of February 2010.
Table 3: GAO Final Assessment of Secure Flight Cost Estimate Compared
to Best Practices for a Reliable Cost Estimate, as of February 2010:
Best practice: Comprehensive;
Explanation: The cost estimates should include both government and
contractor costs over the program's full life cycle, from the
inception of the program through design, development, deployment, and
operation and maintenance to retirement. They should also provide an
appropriate level of detail to ensure that cost elements are neither
omitted nor double counted and include documentation of all cost-
influencing ground rules and assumptions;
Satisfied? Partially satisfied;
GAO analysis: The Department of Homeland Security's (DHS) Cost
Analysis Division (CAD) reviewed the Secure Flight life-cycle cost
estimate in 2009. CAD concluded that one of the reasons the estimate
did not entirely meet GAO best practices for cost estimating was
because the cost estimate adhered to a functional work breakdown
structure (WBS) instead of the best practice standard product-oriented
WBS. However, CAD accepted the estimate conditionally, determining
that the time to correct the WBS would exceed the time phase of the
program, and that rebaselining the program--required for a new WBS--
would be cost prohibitive. CAD officials stated that they expected the
life cycle cost estimate to adhere to a product-oriented WBS for the
next major acquisition milestone. GAO agreed with CAD's conclusions.
Best practice: Well;
documented;
Explanation: The cost estimates should have clearly defined
descriptions of key program or system characteristics. Additionally,
they should capture in writing such things as the source data used and
their significance, the calculations performed and their results, and
the rationale for choosing a particular estimating method. Moreover,
this information should be captured in such a way that the data used
to derive the estimate can be traced back to, and verified against,
their sources. The final cost estimate should be reviewed and accepted
by management;
Satisfied? Satisfied;
GAO analysis: TSA has fully met this criterion.
Best practice: Accurate;
Explanation: The cost estimates should provide for results that are
unbiased and not overly conservative or optimistic. In addition, the
estimates should be updated regularly to reflect material changes in
the program, and steps should be taken to minimize mathematical
mistakes and their significance. Among other things, an estimate
should be grounded in a historical record of cost estimating and
actual experiences on comparable programs;
Satisfied? Partially satisfied;
GAO analysis: As in the case of the "Comprehensive" best practice, CAD
concluded that one of the reasons the estimate did not entirely meet
GAO best practices for cost estimating was because the cost estimate
adhered to a functional WBS instead of the best practice standard
product-oriented WBS. However, CAD accepted the estimate
conditionally, determining that the time to correct the WBS would
exceed the time phase of the program, and that rebaselining the
program--required for a new WBS--would be cost prohibitive. CAD
officials stated that they expected the life cycle cost estimate to
adhere to a product-oriented WBS for the next major acquisition
milestone. GAO agreed with CAD's conclusions.
Best practice: Credible;
Explanation: The cost estimates should discuss any limitations in the
analysis performed due to uncertainty surrounding data or assumptions.
Further, the estimates' derivation should provide for varying any
major assumptions and recalculating outcomes based on sensitivity
analyses, and their associated risks/uncertainty should be disclosed.
Also, the estimates should be verified based on cross-checks using
other estimating methods and by comparing the results with those of
independent cost estimates;
Satisfied? Satisfied;
GAO analysis: CAD had performed assessments of the Secure Flight life-
cycle cost estimate prior to 2009 and identified several areas of
deficiency in the estimate. These deficiencies included the estimate
not being well documented, an inability to trace data back to their
original sources, a functional WBS instead of the best practice
standard product-oriented WBS, and the lack of a risk analysis.
However, CAD accepted the estimates conditionally, determining that
the time to correct these deficiencies would exceed the time phase of
the program, and rebaselining the program--required for a new WBS--
would be cost prohibitive. CAD officials reviewed an updated Secure
Flight life-cycle cost estimate from April 2009 and determined that
the estimate generally adhered to best practices in GAO's Cost
Estimating and Assessment Guide. CAD officials told us that they will
require the Secure Flight program to adhere to a product-oriented WBS
for the next major acquisition milestone point. The Secure Flight
program office hired a contractor to develop an independent cost
estimate, which was completed in October 2009. This independent
estimate was reviewed by the program office and used to assess the
reasonableness of the Secure Flight life-cycle cost estimate. The
independent cost estimate included a sensitivity analysis on key
variables that might affect the long-term cost of the program, as well
as a cost risk uncertainty analysis that determined the level of
confidence associated with the point estimate.
Source: GAO analysis.
[End of table]
GAO has also identified nine best practices that characterize a
reliable schedule estimate--capturing key activities, sequencing key
activities, establishing duration of key activities, assigning
resources to key activities, integrating key activities horizontally
and vertically, establishing critical path, identifying float time,
performing a schedule risk analysis, and distributing reserves to high-
risk activities. Table 4 summarizes the results of our final
assessment of the Secure Flight program's schedule relative to the
nine schedule estimating best practices, as of February 2010.
Table 4: GAO Final Assessment of Secure Flight Schedule Compared to
Best Practices for Schedule Estimating, as of February 2010:
Best practice;
Explanation;
Satisfied?;
GAO analysis.
Best practice: Capturing key activities;
Explanation: The schedule should reflect all key activities as defined
in the program's WBS, to include activities to be performed by both
the government and its contractors;
Satisfied? Partially satisfied;
GAO analysis: The updated integrated master schedule (IMS) includes
baselined international deployment dates. DHS's CAD reviewed the
Secure Flight WBS in 2009 and concluded that it was functionally based
rather than being product oriented, which is the best practice
standard. However, CAD accepted the WBS conditionally, determining
that the time to correct the WBS would exceed the time phase of the
program, and that rebaselining the program--required for a new WBS--
would be cost prohibitive. CAD officials stated that they expected the
program to adhere to a product-oriented WBS for the next major
acquisition milestone. GAO agreed with CAD's conclusions.
Best practice: Sequencing key activities;
Explanation: The schedule should be planned so that it can meet
critical program dates. To meet this objective, key activities need to
be logically sequenced in the order that they are to be carried out.
In particular, activities that must finish prior to the start of other
activities (i.e., predecessor activities), as well as activities that
cannot begin until other activities are completed (i.e., successor
activities), should be identified. By doing so, interdependencies
among activities that collectively lead to the accomplishment of
events or milestones can be established and used as a basis for
guiding work and measuring progress;
Satisfied? Substantially satisfied;
GAO analysis: The logic in the updated IMS is straightforward, and
none of the remaining activities have missing dependencies. However,
the use of "Start No Earlier Than" constraints for each airline
testing phase 1 activity prevents the schedule from being able to
dynamically calculate the completion date. The program office has
justified the use of these constraints, as airline testing phase start
dates are legally binding dates negotiated between the program office
and each of the approximately 200 airlines.
Best practice: Establishing the duration of key activities;
Explanation: The schedule should realistically reflect how long each
activity will take to execute. In determining the duration of each
activity, the same rationale, historical data, and assumptions used
for cost estimating should be used. Durations should be as short as
possible and have specific start and end dates. Excessively long
periods needed to execute an activity should prompt further
decomposition so that shorter execution durations will result. The
schedule should be continually monitored to determine when forecasted
completion dates differ from the planned dates, which can determine
whether schedule variances will affect downstream work;
Satisfied? Substantially satisfied;
GAO analysis: The remaining activities in the updated IMS generally
met best practices for activity duration. For example, 75 percent of
the remaining activities have duration of 4 days or less, and only 1
percent of remaining activities have duration of greater than 13 days.
However, because 48 percent of the remaining activities have duration
of 1 day, concern remains that the schedule is assuming too much
productivity for an 8-hour day.
Best practice: Assigning resources to key activities;
Explanation: The schedule should reflect what resources (e.g., labor,
material, and overhead) are needed to do the work, whether all
required resources will be available when needed, and whether any
funding or time constraints exist;
Satisfied? Satisfied;
GAO analysis: Based on our analysis of the December 2009 IMS and
interviews with program office officials, we found that the IMS is
resource loaded and the program office is actively assessing the
loaded resources.
Best practice: Integrating key activities horizontally and vertically;
Explanation: The schedule is horizontally integrated, meaning that it
linked the products and outcomes associated with already sequenced
activities. These links are commonly referred to as "handoffs" and
serve to verify that activities are arranged in the right order to
achieve aggregated products or outcomes. The schedule should also be
vertically integrated, meaning that traceability exists among varying
levels of activities and supporting tasks and subtasks. Such mapping
or alignment among levels enables different groups to work to the same
master schedule;
Satisfied? Substantially satisfied;
GAO analysis: The updated IMS is fully integrated vertically and
somewhat integrated horizontally. The use of Start No Earlier Than
constraints for each airline testing phase 1 activity prevents the
schedule from being able to dynamically calculate the completion date.
However, as airline testing phase start dates are legally binding
dates negotiated between the program office and each of the
approximately 200 airlines, the program office has justified the use
of these constraints.
Best practice: Establishing the critical path for key activities;
Explanation: Using scheduling software, the critical path--the longest
duration path through the sequenced list of key activities--should be
identified. The establishment of a program's critical path is
necessary for examining the effects of any activity slipping along
this path. Potential problems that might occur along or near the
critical path should also be identified and reflected in the
scheduling of the time for high-risk activities;
Satisfied? Partially satisfied;
GAO analysis: The updated IMS does not contain a valid critical path
as calculated by the software because of the Start No Earlier Than
constraints on each airline testing phase beginning milestone.
However, as airline testing phase start dates are legally binding
dates negotiated between the program office and each of the
approximately 200 airlines, the program office has justified the use
of these constraints.
Best practice: Identifying the float time between key activities;
Explanation: The schedule should identify float time--the time that a
predecessor activity can slip before the delay affects successor
activities--so that schedule flexibility can be determined. Generally,
activities along the critical path typically have the least amount of
float time. Total float describes the amount of time flexibility an
activity has without delaying the project completion (if everything
else goes according to plan). Total float is used to find out which
activities or paths are crucial to project completion;
Satisfied? Partially satisfied;
GAO analysis: The updated IMS schedule does not reflect realistic
float values because of the high number of Start No Earlier Than
constraints on the on each airline testing phase beginning milestone.
However, as airline testing phase start dates are legally binding
dates negotiated between the program office and each of the
approximately 200 airlines, the program office has justified the use
of these constraints. The IMS shows that 159 activities have from 250
to 300 days of float, suggesting that these activities can slip up to
300 days and not affect the completion date of the project. The
updated IMS contains baselined international deployment dates.
Best practice: Schedule risk analysis should be performed;
Explanation: A schedule risk analysis should be performed using
statistical techniques to predict the level of confidence in meeting a
program's completion date. This analysis focuses not only on critical
path activities but also on activities near the critical path, since
they can potentially affect program status;
Satisfied? Satisfied;
GAO analysis: A schedule risk analysis was performed by an independent
contractor and delivered in August 2009. The schedule risk analysis
made several recommendations regarding the technical aspects of the
schedule as well as programmatic risk reduction recommendations. Among
the risk reduction recommendations made were (1) that the amount of
work required for each task be estimated before resources are assigned
and that an attempt be made to capture the actual work performed
throughout the project; (2) that aircraft operator start dates be
moved earlier when resources are available; (3) that because the
schedule may be delayed if less than 17 aircraft operators are
deployed concurrently, the program office should determine the maximum
number of aircraft operators that can be deployed concurrently;
and (4) that additional resources be assigned to the testing
environment.
Best practice: Distributing reserves to high-risk activities;
Explanation: The baseline schedule should include a buffer or a
reserve of extra time. Schedule reserve for contingencies should be
calculated by performing a schedule risk analysis. Generally, the
reserve should be applied to high-risk activities, which are typically
found along the critical path;
Satisfied? Substantially satisfied;
GAO analysis: According to TSA's plan of action, the schedule risk
analysis was completed in August 2009, and program leadership took
related actions to distribute reserves.
Source: GAO analysis.
[End of table]
[End of section]
Footnotes:
[1] The No-Fly and Selectee lists contain the names of individuals
with known or suspected links to terrorism. These lists are subsets of
the consolidated terrorist watchlist that is maintained by the Federal
Bureau of Investigation's Terrorist Screening Center.
[2] See Pub. L. No. 108-458, § 4012(a), 118 Stat. 3638, 3714-18 (2004)
(codified at 49 U.S.C. § 44903(j)(2)(C)).
[3] GAO has performed this work in accordance with statutory mandates,
beginning in fiscal year 2004 with the Department of Homeland Security
Appropriations Act, 2004, Pub. L. No. 108-90, § 519, 117 Stat. 1137,
1155-56 (2003) (establishing the initial mandate that GAO assess the
Computer-Assisted Passenger Prescreening System II, the precursor to
Secure Flight, and setting forth the original eight statutory
conditions related to the development and implementation of the
prescreening system), and pursuant to the requests of various
congressional committees.
[4] See, for example, GAO, Aviation Security: Transportation Security
Administration Has Strengthened Planning to Guide Investments in Key
Aviation Security Programs, but More Work Remains, [hyperlink,
http://www.gao.gov/products/GAO-08-456T] (Washington, D.C.: Feb. 28,
2008); Aviation Security: Significant Management Challenges May
Adversely Affect Implementation of the Transportation Security
Administration's Secure Flight Program, [hyperlink,
http://www.gao.gov/products/GAO-06-374T] (Washington, D.C.: Feb. 9,
2006); and Aviation Security: Secure Flight Development and Testing
Under Way, but Risks Should Be Managed as System Is Further Developed,
[hyperlink, http://www.gao.gov/products/GAO-05-356] (Washington, D.C.:
Mar. 28, 2005).
[5] See Pub. L. No. 108-334, § 522, 118 Stat. 1298, 1319-20 (2004).
The appropriations acts for each subsequent fiscal year through fiscal
year 2009 included the same requirement, referring back to the 10
conditions from fiscal year 2005. See Department of Homeland Security
Appropriations Act, 2006, Pub. L. No. 109-90, § 518(a), 119 Stat.
2064, 2085 (2005); Department of Homeland Security Appropriations Act,
2007, Pub. L. No. 109-295, § 514(a), 120 Stat. 1355, 1379 (2006);
Consolidated Appropriations Act, 2008, Pub. L. No. 110-161, div. E, §
513(a), 121 Stat. 1844, 2072 (2007); and Department of Homeland
Security Appropriations Act, 2009, Pub. L. No. 110-329, div. D, §
512(a), 122 Stat. 3652, 3682 (2008).
[6] GAO, Aviation Security: TSA Has Completed Key Activities
Associated with Implementing Secure Flight, but Additional Actions Are
Needed to Mitigate Risks, [hyperlink,
http://www.gao.gov/products/GAO-09-292] (Washington, D.C.: May 13,
2009).
[7] See H.R. Rep. No. 111-298, at 80 (2009) (Conf. Rep.).
[8] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for
Developing and Managing Capital Program Costs, [hyperlink,
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009).
[9] In general, an integrated master schedule contains all of the
detailed work and planning activities necessary to support key program
milestones.
[10] Specifically, we recommended that TSA update the Secure Flight
program's life-cycle cost estimate to include all costs, compare the
updated estimate to an independent cost estimate, align cost estimates
with the program's schedule, quantify the risks facing the program,
and determine a level of confidence in meeting the cost estimate.
Because TSA provided us with its plan of action to address these
recommendations in April 2009, we did not include the recommendations
in our May 2009 report.
[11] In general, an uncertainty analysis provides a level of
confidence about the program's cost estimate, while a sensitivity
analysis allows decision makers to understand the impact of changing
one assumption or cost driver at a time (e.g., the effect of a program
milestone delay on the total cost of the program).
[12] At times, an organization may conclude that the remaining budget
and schedule targets for completing a program are significantly
insufficient and that the current baseline is no longer valid for
realistic performance measurement. If so, the program may be
rebaselined.
[13] Specifically, we recommended that TSA establish a reliable
benchmark schedule for the Secure Flight program's remaining
activities using scheduling best practices, maintain the schedule
throughout the program's life cycle using proper scheduling methods,
rely on calculated dates in the schedule rather than imposing target
dates, add projected resources to the program's schedule to better
track progress, and periodically assess the risks to the schedule.
Because TSA provided us with its plan of action to address these
recommendations in April 2009, we did not include the recommendations
in our May 2009 report.
[14] In general, float is the amount of time an activity can slip
before affecting successor activities.
[15] The schedule should reflect what resources (e.g., labor,
material, and overhead) are needed to do the work, whether all
required resources will be available when needed, and whether any
funding or time constraints exist.
[16] The schedule should be horizontally integrated, meaning that it
links the products and outcomes associated with already sequenced
activities, and should be vertically integrated, meaning that
traceability exists among varying levels of activities and supporting
tasks and subtasks.
[17] The schedule should include a buffer or a reserve of extra time
to complete work. Generally, the reserve should be applied to high-
risk activities, which are typically found along the critical path.
[18] Because of fluctuations in passenger service, the total number of
covered carriers in the Secure Flight program will vary. For example,
a new carrier could either start new service or cancel existing
service.
[19] Partial assumption involves air carriers that are phasing in
their flights or provide service to multiple airlines.
[20] See 73 Fed. Reg. 64,018 (Oct. 28, 2008) (codified at 49 C.F.R.
pt. 1560).
[21] [hyperlink, http://www.gao.gov/products/GAO-09-3SP].
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
