National Airspace System
Better Cost Data Could Improve FAA's Management of the Standard Terminal Automation Replacement System Gao ID: GAO-03-343 January 31, 2003To enhance the capacity and safety of the national airspace system, the Federal Aviation Administration (FAA), within the Department of Transportation, is acquiring 74 Standard Terminal Automation Replacement Systems (STARS). STARS will replace some outdated air traffic control equipment. Since 1996, when FAA initiated this major computer hardware and software acquisition, the scope and estimated costs of STARS have changed many times. FAA now estimates that STARS's remaining costs will total about $2.54 billion. GAO was asked to assess the reliability of FAA's life-cycle cost estimate for STARS, determine the impact of STARS's estimated costs on future FAA budgets, and identify any alternatives to STARS that FAA is considering. GAO based its analysis on published FAA cost data and the guidance FAA uses for managing major acquisitions.
The reliability of FAA's life cycle cost estimate for STARS is uncertain. This estimate includes estimates of the program's development, operation, and technology upgrade costs, shown for the next 6 fiscal years in the figure below. The development cost estimate is based on the contractor's projections, which FAA has not yet independently analyzed as its guidance directs. Furthermore, baseline data in cost performance reports that FAA obtains from the contractor are not accurate because the data do not reflect the current status of the contract. As a result, FAA is limited in its ability to manage the contract effectively. FAA plans to address these problems. In addition, the program's operation and technology upgrade cost estimates are based on limited experience with STARS and extend many years into the future. However, the estimates do not reflect these uncertainties. For fiscal years 2004 through 2007, the years for which FAA provided budget information, STARS's estimated costs should have a declining impact on FAA's budgets because the program's development is nearly over and its operations are still limited. For fiscal years 2008 through 2030, the impact of STARS's estimated costs on FAA's budgets is unknown because it is still too soon in the budget cycle for FAA to have developed detailed budgets for these years. After deploying STARS at the 74 terminal and support facilities included in the program, FAA could use STARS, another contractor's technology, or a combination of the two technologies for the nearly 100 remaining facilities. FAA is exploring the feasibility of combining the technologies and expects to announce its plans in April 2003.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-03-343, National Airspace System: Better Cost Data Could Improve FAA's Management of the Standard Terminal Automation Replacement System
This is the accessible text file for GAO report number GAO-03-343
entitled 'National Airspace System: Better Cost Data Could Improve
FAA's Management of the Standard Terminal Automation Replacement
System' which was released on February 04, 2003.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products‘ accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
Report to the Honorable Ellen O. Tauscher, House of Representatives:
January 2003:
NATIONAL AIRSPACE SYSTEM:
Better Cost Data Could Improve FAA‘s Management of the Standard
Terminal Automation Replacement System:
GAO-03-343:
GAO Highlights:
Highlights of GAO-03-343, a report to the Honorable Ellen O. Tauscher,
House of Representatives.
NATIONAL AIRSPACE SYSTEM
Better Cost Data Could Improve FAA‘s Management of the Standard
Terminal Automation Replacement System.
Why GAO Did This Study:
To enhance the capacity and safety of the national airspace system,
the Federal Aviation Administration (FAA), within the Department of
Transportation, is acquiring 74 Standard Terminal Automation
Replacement Systems (STARS). STARS will replace some outdated air
traffic control equipment. Since 1996, when FAA initiated this major
computer hardware and software acquisition, the scope and estimated
costs of STARS have changed many times. FAA now estimates that STARS‘s
remaining costs will total about $2.54 billion. GAO was asked to assess
the reliability of FAA‘s life-cycle cost estimate for STARS, determine
the impact of STARS‘s estimated costs on future FAA budgets, and
identify any alternatives to STARS that FAA is considering. GAO based
its analysis on published FAA cost data and the guidance FAA uses for
managing major acquisitions.
What GAO Found:
The reliability of FAA‘s life-cycle cost estimate for STARS is
uncertain. This estimate includes estimates of the program‘s
development, operation, and technology upgrade costs, shown for the
next 6 fiscal years in the figure below. The development cost estimate
is based on the contractor‘s projections, which FAA has not yet
independently analyzed as its guidance directs. Furthermore, baseline
data in cost performance reports that FAA obtains from the contractor
are not accurate because the data do not reflect the current status of
the contract. As a result, FAA is limited in its ability to manage the
contract effectively. FAA plans to address these problems. In addition,
the program‘s operation and technology upgrade cost estimates are based
on limited experience with STARS and extend many years into the future.
However, the estimates do not reflect these uncertainties.
Estimated STARS Funding, Fiscal Years 2004-2009
[See PDF for Image]
[End of Figure]
For fiscal years 2004 through 2007, the years for which FAA provided
budget information, STARS‘s estimated costs should have a declining
impact on FAA‘s budgets because the program‘s development is nearly
over and its operations are still limited. For fiscal years 2008
through 2030, the impact of STARS‘s estimated costs on FAA‘s budgets
is unknown because it is still too soon in the budget cycle for FAA to
have developed detailed budgets for these years. After deploying STARS
at the 74 terminal and support facilities included in the program, FAA
could use STARS, another contractor‘s technology, or a combination of
the two technologies for the nearly 100 remaining facilities. FAA is
exploring the feasibility of combining the technologies and expects to
announce its plans in April 2003.
What GAO Recommends:
To improve FAA‘s management of STARS and of subsequent terminal
modernization programs and to provide the Congress with more reliable
information for oversight, GAO recommends that the Secretary of
Transportation direct the FAA Administrator to maintain accurate,
current baseline data; review baseline data within 6 months of any
major modification to ensure that the data reflect the current status
of the contract; and prepare a rigorous life-cycle cost estimate that
identifies the level of uncertainty.
To view the full report, including the scope
and methodology, click on the link above.
For more information, contact Gerald L. Dillingham at (202) 512-5555
or Keith A. Rhodes at (202) 512-6412.
Contents:
Letter:
Results in Brief:
Background:
Reliability of FAA‘s Life-Cycle Cost Estimate for STARS Is
Uncertain:
Impact of STARS‘s Estimated Costs on FAA‘s Budgets Is Expected to
Decline through Fiscal Year 2007:
After Deploying STARS at 74 Facilities, FAA May Modify Its Approach for
the Remaining Facilities:
Conclusions:
Recommendations for Executive Action:
Agency Comments:
Scope and Methodology:
Appendix:
Appendix I: GAO Contacts and Staff Acknowledgements:
GAO Contacts:
Staff Acknowledgments:
Table:
Table 1: Proposed Funding for FAA‘s Total Facilities and Equipment
Budget and for STARS‘s Development and Deployment:
Figures:
Figure 1: Standard Terminal Automation Replacement System (STARS):
Figure 2: Life-Cycle Cost Estimate for STARS, Fiscal Years 2004-2030:
Figure 3: Allocation of Facilities and Equipment Funding for Fiscal
Year 2004 in FAA‘s Capital Investment Plan:
Abbreviations:
ARTS: Automated Radar Terminal System:
DCMA: Defense Contract Management Agency:
DOD: Department of Defense:
FAA: Federal Aviation Administration:
STARS: Standard Terminal Automation Replacement System:
This is a work of the U.S. Government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. It may contain
copyrighted graphics, images or other materials. Permission from the
copyright holder may be necessary should you wish to reproduce
copyrighted materials separately from GAO‘s product.
Letter January 31, 2003:
The Honorable Ellen O. Tauscher
House of Representatives:
Dear Ms. Tauscher:
On November 17, 2002, the Federal Aviation Administration (FAA) began
using the Standard Terminal Automation Replacement System (STARS) to
control air traffic at the Philadelphia air route traffic control
center, the first ’busy“ FAA facility to use this version of the new
system.[Footnote 1] FAA‘s current plan is to procure 74 STARS systems,
including 70 for terminal facilities and 4 for support
facilities.[Footnote 2] STARS will replace outdated computer equipment
that is used to control air traffic within 5 to 50 nautical miles of an
airport.[Footnote 3] With STARS, air traffic controllers at these
facilities will receive new hardware and software that produce color
displays of aircraft position and flight information. In the future,
FAA will be able to upgrade the software to provide air traffic control
tools, such as those that will allow better spacing of aircraft as they
descend into airports. STARS is complex, costly, and software-
intensive. Since 1996, when FAA initiated STARS, the number of systems
scheduled to be procured has ranged from as many as 188 to as few as
74,[Footnote 4] and the program‘s cost and schedule have also varied
considerably. Over the years, we have reported on these changes, most
recently in September 2002.[Footnote 5]
As agreed with your office, this report addresses the following
questions:
* How reliable is FAA‘s estimate of the life-cycle costs to develop and
deploy, operate and maintain, and upgrade STARS?
* What impact will STARS‘s estimated cost have on future FAA budgets
given competing demands for funds to enhance aviation safety, security,
and capacity?
* What alternatives to STARS is FAA considering?
In addition, you asked whether STARS falls under the termination
provisions of Public Law 104-264, as an acquisition that is more than
50 percent over its cost goal or behind schedule. Our analysis of the
law and its legislative history indicates that STARS is not subject to
these termination provisions. According to our analysis, the
termination provisions apply to acquisitions initiated after October 9,
1996, the date of the law‘s enactment. Because FAA approved the initial
acquisition plan for STARS in March 1996 and signed the contract with
Raytheon Corporation in September 1996, STARS is not subject to those
provisions.
This report covers cost and performance issues related to FAA‘s
procurement of STARS for 74 terminal and support facilities.[Footnote
6] To conduct our work, we reviewed FAA‘s 5-year Capital Investment
Plan, which proposes funding for programs to modernize the national
airspace system. We also analyzed data from cost performance reports
that the STARS contractor developed for FAA. However, we did not
independently verify these cost and performance data.
Results in Brief:
The reliability of FAA‘s life-cycle cost estimate for STARS is
uncertain. According to FAA, the costs to develop and deploy,[Footnote
7] operate and maintain, and upgrade STARS will amount to about $2.54
billion for 74 systems for fiscal years 2004 through 2030. More
specifically, the costs of completing STARS‘s development and
deployment will amount to about $153 million for fiscal years 2004
through 2008, the costs of operating and maintaining the systems at
those facilities over their useful lives will add another $1.46 billion
for fiscal years 2004 through 2030,[Footnote 8] and the costs of
upgrading STARS technology will amount to about $930 million over the
same period. FAA‘s development cost estimate is based largely on the
contractor‘s recent proposals and projections, which incorporate the
costs of new work specified in major modifications to the STARS
contract. FAA has not yet independently analyzed these proposals and
projections as its guidance directs and therefore does not know whether
the development cost estimate of $153 million is reliable. Furthermore,
FAA has not worked with the contractor to incorporate the requirements
and costs of the new work into cost performance reports, which it
receives from the contractor. These reports are intended to provide FAA
with accurate, current data for monitoring and overseeing the
contractor‘s progress and for estimating the program‘s remaining costs,
but FAA is not using the reports because they do not reflect the
current status of the contract. FAA is now analyzing the contractor‘s
cost data and working with the contractor to align the cost performance
reports with the current status of the contract. FAA expects to
complete these tasks in the spring of 2003 and then should be able to
use the cost performance reports as intended. The reliability of FAA‘s
estimates of the life-cycle costs to operate, maintain, and upgrade
STARS technology is unknown, primarily because FAA has limited
experience with STARS equipment and the estimates extend nearly 30
years into the future. However, despite these uncertainties, FAA has
expressed its cost estimates as point values, rather than as ranges. As
a result, the estimates may imply more certainty than is appropriate.
Moreover, in this instance, the use of point values limits disclosure
of the program‘s investment risks. We are making recommendations to
strengthen FAA‘s management of STARS, and FAA officials indicated that
the recommendations were in line with the agency‘s ongoing and planned
efforts. These recommendations, we believe, will also help FAA better
manage the planned modernization of terminal facilities that are no
longer included in the STARS program.
According to FAA‘s latest budget planning documents, the impact of
STARS‘s estimated costs on FAA‘s budgets will decline for fiscal years
2004 through 2007; for later years, however, the impact of these
estimated costs is unknown. Since FAA has nearly finished developing
STARS, the program‘s development costs are expected to decrease over
the next 4 fiscal years, while its operation and maintenance costs are
expected to grow with increased deployment. According to FAA‘s
documents, the funding proposed to develop and deploy STARS represents
4 percent of FAA‘s proposed $3.06 billion facilities and equipment
budget account for fiscal year 2004 and smaller percentages for fiscal
years 2005 through 2007. This account funds the development,
procurement, and installation of equipment to help increase the
capacity and improve the safety, efficiency, and security of the
national airspace system. Given these small and declining percentages
for STARS, the program‘s estimated costs should, over this period, have
a small and declining impact on other aviation safety, security, and
capacity efforts funded from the facilities and equipment account.
FAA‘s budget planning documents also show that as FAA deploys STARS at
more facilities, more funding will come from the agency‘s operations
account, which supports training and compensation for the controllers
and technicians who operate and maintain STARS. However, the impact of
STARS‘s estimated costs on FAA‘s budgets for later years is currently
unknown because it is too soon in the budget cycle for FAA to have
developed detailed budgets beyond fiscal year 2007. In addition, as we
previously noted, FAA has limited operational experience for projecting
STARS‘s budgetary impact, and long-range cost estimates are inherently
uncertain.
FAA is committed to deploying STARS at the 74 terminal and support
facilities included in the STARS program, but for the nearly 100 other
facilities that remain to be modernized, the agency could deploy STARS,
another contractor‘s hardware and software, or a combination of STARS
and the other contractor‘s technologies that are currently being used
in FAA terminal facilities. FAA knows that each contractor‘s technology
works independently, and FAA is assessing the feasibility of operating
Raytheon‘s STARS display system with the processing system from
Lockheed Martin Corporation‘s Common Automated Radar Terminal System,
which was recently installed at some terminal facilities. Combining the
two technologies could be cost-effective because it would allow FAA to
use both the customized display system that accounted for a substantial
portion of STARS‘s development costs and the recently acquired
processing system. FAA plans to announce the results of this assessment
in April 2003.
Background:
STARS will replace controller workstations with new color displays,
processors, and computer software at FAA and Department of Defense
terminal air traffic control facilities. (See fig. 1.) FAA‘s goal for
STARS is to provide an open, expandable terminal automation platform
that can accommodate future air traffic growth and allow for the
introduction of new hardware-and software-based tools to promote
safety, maximize operational efficiency, and improve controllers‘
productivity. FAA believes that STARS will facilitate efforts to
optimally configure the terminal airspace around the country, exchange
digital information between pilots and controllers, and introduce new
position and surveillance capabilities for pilots.
Figure 1: Standard Terminal Automation Replacement System (STARS):
[See PDF for image] - graphic text:
[End of figure] - graphic text:
FAA has given high priority to STARS. Both the past and the current FAA
Administrator have emphasized the program‘s importance to enhancing the
capacity of the national airspace system. In addition, STARS is 1 of 19
programs on FAA‘s list of top programs. Recently, FAA gave priority
consideration to STARS so that it could meet its commitment to the
Congress to deploy STARS at the Philadelphia terminal, the first busy
facility, on November 17, 2002. Furthermore, according to FAA
officials, the agency remains committed to funding STARS until
completion, even if that means postponing the funding for other
programs. Agency officials have indicated, for example, that if the
fiscal year 2004 budget falls below the level shown in FAA‘s planning
documents, funds from other programs would likely be cut to fully fund
STARS.
For each acquisition program that FAA undertakes, it officially
estimates the program‘s life-cycle costs (from development and
deployment through operations and maintenance), schedule, benefits, and
performance in a formal budget document called the acquisition program
baseline. FAA uses this document--which its acquisition decision-making
body, the Joint Resources Council, must approve--to decide whether to
fund the program and, if it is funded, to monitor its progress. FAA
also uses the approved acquisition program baseline to develop a 5-year
budget-planning document, called the Capital Investment Plan. Program
managers rely on the acquisition program baseline to oversee the
program‘s progress and to ensure that no action is taken that would
breach the approved baseline. To support their oversight, program
managers typically require contractors to deliver cost performance
reports, each of which includes a performance measurement baseline for
assessing the contractor‘s progress in meeting the contract‘s cost,
schedule, and technical performance goals. For STARS, FAA has one
contract with Raytheon, which accounts for 82 percent of the funding
approved in the acquisition program baseline. Most of the remaining
funding is used for contracts that support FAA‘s internal program
management.
For guidance in managing its major acquisition programs, FAA relies
largely on two documents--its own Acquisition System Toolset, a ’one-
stop acquisition information system“ on FAA‘s Web site that contains
the agency‘s official acquisition policy and guidance, and the Defense
Contract Management Agency‘s (DCMA) guide on implementing earned value
management.[Footnote 9] The Acquisition System Toolset includes policy,
guidance, instructions, examples, best practices, lessons learned,
references, and other related information tailored to each type of
procurement contract. The earned value management guide provides
information on how to use cost and performance measurement to manage
acquisition programs.
FAA funds STARS primarily through two of its budget accounts: (1)
facilities and equipment and (2) operations. The facilities and
equipment account covers the costs to develop, procure, and place the
new equipment in operation.[Footnote 10] After the equipment has been
fully operating for at least a year, the funding source shifts to the
operations account, which covers the costs to support and maintain the
equipment over its life cycle. Operations costs are, in large part,
personnel costs--for the controllers who operate and the technicians
who maintain the equipment. Planned product improvements and technology
upgrades are primarily funded from the facilities and equipment
account.
Since 1996, when FAA initiated STARS, it has spent approximately $1.2
billion, or about 86 percent of the funding budgeted for the program,
and it has twice approved major changes to the program‘s cost and
schedule estimates. First, in October 1999, FAA modified its
acquisition approach (from off-the-shelf software only to a combination
of customized and off-the-shelf software) and increased to 188 the
number of facilities scheduled to receive STARS. We reported on these
changes in September 2002.[Footnote 11] At that time, FAA also
concluded that it did not have adequate funding to deploy STARS to all
188 facilities with the remainder of the STARS funding. Instead, FAA
decided to deploy STARS to 74 terminal and support facilities. The
selected facilities had frequent equipment failures, were new, or had
the digital radar needed to operate STARS. FAA has since reduced the
total number of facilities to about 170. The agency is currently
studying options for modernizing display systems at the nearly 100
remaining facilities and is identifying the additional costs of
upgrading the other facilities with STARS or an alternative system. The
STARS program office stated that these additional costs, together with
the funds already committed to STARS, would more than likely exceed the
$1.4 billion originally planned. The program office is in the process
of developing options and estimates for these additional sites.
Reliability of FAA‘s Life-Cycle Cost Estimate for STARS Is Uncertain:
FAA has estimated the life-cycle cost for STARS, including the costs to
develop and deploy systems to 74 terminal and support facilities and to
operate, maintain, and upgrade the systems. However, the reliability of
these cost estimates is uncertain. If FAA‘s estimates are not reliable,
both the agency and the Congress will be limited in their ability to
project and compare the costs and benefits of completing STARS and
modernizing other facilities as well as in their ability to budget
realistically for other capital investments.
FAA Has Estimated STARS‘s Remaining Life-Cycle Costs:
According to FAA‘s estimates, the agency will need about $2.54 billion
for STARS over the remaining life of the program. As indicated in
figure 2, FAA will need about (1) $153 million for fiscal years 2004
through 2008 to complete the development and deployment of 74 systems;
(2) $1.46 billion for fiscal years 2004 through 2030--or about $54
million a year, on average--to operate and maintain the
systems;[Footnote 12] and (3) $930 million to upgrade STARS technology
over the same period. FAA‘s operating cost estimate assumes that
STARS‘s operating costs will grow with deployment through about fiscal
year 2007 and will subsequently keep pace with inflation, at least
through fiscal year 2009. FAA expects the costs of technology upgrades
to more than triple between fiscal years 2007 and 2008 because of a
technology update cycle, which will peak when the hardware and software
are due for replacement. This projection assumes that STARS software
will be upgraded every 3 years.
Figure 2: Life-Cycle Cost Estimate for STARS, Fiscal Years 2004-2030:
[See PDF for image] - graphic text:
[End of figure] - graphic text:
Note: These data are from FAA‘s March 2002 acquisition program baseline
for STARS and have been adjusted to reflect the effects of inflation.
Numbers may not add because of rounding.
FAA does not yet know to what extent its estimate of STARS‘s remaining
development costs is reliable. The estimate is derived from the
contractor‘s proposals and projections, which reflect the costs of
major modifications to the contract made since May 2000. However, FAA
has not yet independently analyzed and validated the proposals and
projections and therefore cannot assess the reliability of the
development cost estimate.
FAA obtains monthly cost performance reports from the contractor, which
the agency should be able to use both to oversee the contractor‘s
performance and to estimate the program‘s remaining development costs.
However, FAA does not use these reports because they are not current.
More specifically, their central component, the performance management
baseline--which establishes performance, cost, and schedule milestones
for the contract--has not been updated since May 2000 and therefore
does not incorporate the effects of major contract modifications
approved since that date. For example, the September 2002 cost
performance report does not reflect FAA‘s March 2002 reduction in the
scope of STARS from 188 to 74 systems, and the report does not include
the costs of new work that FAA authorized between May 2000 and
September 2002. Consequently, the report indicates that STARS is on
schedule and within 1 percent of budget, even though, compared with the
program envisioned in May 2000, FAA is now under contract to modernize
fewer than half as many facilities at more than twice the cost per
facility.
Cost Performance Reports Do Not Meet FAA‘s Criteria:
The cost performance reports for STARS do not meet the criteria for
such reports established in the guidance for managing major
acquisitions that FAA has adopted from DCMA.[Footnote 13] According to
this guidance, cost performance reports are valid only when:
* a reliable performance measurement baseline is established and
maintained (i.e., regularly updated and validated),
* changes to the baseline are carefully controlled (i.e., negotiated
and approved before being authorized),
* an integrated baseline review[Footnote 14] takes place within 6
months of a contract‘s award or significant modification, and:
* contract oversight occurs regularly.
FAA and the contractor have established a performance measurement
baseline for the STARS contract, but they have not satisfied the
remaining criteria. The baseline has not been updated since May 2000,
and FAA has not validated it. For example, FAA has not asked an
independent organization, such as the Defense Contract Audit Agency, to
verify the costs in the contractor‘s cost performance reports by
tracing the costs back to the contractor‘s accounting system. In
addition, FAA has not controlled changes to the baseline because,
between May 2000 and September 2002, it approved up to $179 million in
authorized, unpriced work--that is, additional work that FAA agreed to
let the contractor perform without first negotiating or independently
verifying the costs. Furthermore, the additional tasks and costs have
not been incorporated in the baseline, even though the DCMA guidance
calls for processing contract modifications expeditiously and
incorporating them in a timely manner to maintain the baseline‘s
integrity. Although the DCMA guidance does not define ’a timely
manner,“ experts generally agree that this term means no longer than 3
months. FAA has not maintained and controlled the baseline because,
according to program officials, it has been ’schedule driven“--
committed to deploying STARS at the Philadelphia terminal by November
17, 2002. FAA is currently analyzing the contractor‘s cost data and
working with the contractor to incorporate modifications in the
performance measurement baseline.
Although the DCMA guidance calls for performing an integrated baseline
review within 6 months of awarding or significantly modifying a
contract, FAA has not performed such a review of STARS since August
2000, even though it has subsequently made two major modifications to
the contract. An integrated baseline review is important to ensure that
the contract‘s cost data are aligned with the current status of the
program after a major contract modification. According to the Manager
of Terminal Automation, who oversees STARS, FAA had planned to initiate
an integrated baseline review of the STARS contract in November 2002
and expects to begin this effort as soon as funding is available.
To provide regular contract oversight as the DCMA guidance requires,
FAA and Raytheon meet monthly to, among other things, discuss the cost
performance reports. However, discussions of these reports do not, in
our view, constitute regular contract oversight because, without a
current, valid performance measurement baseline, FAA cannot compare
what the contractor has done with what the contractor agreed to do in
the contract. With a current, valid baseline, the reports would
indicate when cost or schedule thresholds had been exceeded, and FAA
could then require the contractor to explain the reasons for the
variances and to identify and take appropriate corrective actions. But
because the baseline has not been maintained and is not aligned with
the current status of the program, the reports are not useful for
evaluating the contractor‘s performance or for projecting the
contract‘s remaining costs. The Manager of Terminal Automation agreed
that the current cost performance reports are not useful for these
purposes and said that the agency therefore uses the monthly meetings
with the contractor to discuss other program control and financial
issues.
FAA‘s Cost Estimates Do Not Reflect Levels of Risk:
FAA has limited information for determining the reliability of the
costs it has estimated to operate and maintain STARS and to upgrade its
technology. The agency first projected the costs of operating and
maintaining STARS in March 2002 and said that if these costs were not
funded, essential STARS maintenance would not be completed. However,
FAA has just begun to operate STARS and has limited experience for
projecting future operating and maintenance costs. In addition, FAA did
not perform a risk assessment, as its Acquisition System Toolset
guidance specifies, to identify the minimum, most likely, and maximum
expected costs for the entire program. Although FAA performed such an
assessment for one facility, it did not extend this effort--again
because of time pressures, according to FAA officials. Because FAA has
just begun to operate STARS, it also has limited experience for
projecting the costs of technology upgrades over the life of the
program. Additionally, both the operating and the technology upgrade
cost estimates are uncertain because they extend many years into the
future.
Despite the uncertainty inherent in estimates--especially long-range
ones--FAA has expressed its cost estimates for STARS as point values.
As we reported in 1997,[Footnote 15] point values imply certainty and
therefore are not suitable for expressing estimates. Instead, ranges or
numbers with confidence levels would be more appropriate. For instance,
a cost estimate of $1 million could be presented either as a range of
$900,000 to $1.1 million or as $1 million with a confidence interval of
90 percent, indicating that there is a 10 percent chance that the cost
will exceed the estimate.[Footnote 16] Because FAA did not perform a
risk assessment for the entire STARS program, it did not develop ranges
or confidence levels that it could use to express its cost estimates
for STARS. And because FAA used point values instead, it limited its
disclosure of the program‘s investment risks.
The reliability of FAA‘s development and operating cost estimates is
important not only for managing STARS effectively but also for planning
appropriately for other terminal modernization efforts and other FAA
capital investments. Without reliable estimates for comparing STARS‘s
costs and benefits and for anticipating future expenditures, FAA and
the Congress cannot make informed decisions about how best to modernize
the nearly 100 facilities that are not currently scheduled to receive
STARS. Furthermore, because FAA has given high priority to completing
STARS and has said that it will fund the program even if it has to
postpone the funding for other programs, FAA‘s estimates for STARS will
influence the agency‘s plans for funding other capital investments.
Impact of STARS‘s Estimated Costs on FAA‘s Budgets Is Expected to
Decline through Fiscal Year 2007:
According to our analysis of budgetary data for fiscal years 2004
through 2007 that we obtained from FAA, the estimated costs of STARS
should have a declining impact on FAA‘s budgets during these years
because the program‘s development phase is nearly over and its
operations will still be limited. For later years, the impact of these
estimated costs on FAA‘s budgets is unknown because it is too soon in
the budget cycle for FAA to have developed detailed budgets beyond
fiscal year 2007.
Because FAA has nearly completed the development of STARS and has begun
to install the equipment, the agency is budgeting less from the
facilities and equipment account for STARS. Therefore, the proposed
funding for STARS does not have a significant impact on the funding for
other efforts to improve the safety, security, and capacity of the
national airspace system. For fiscal year 2004, the $94.1 million
planned to develop and deploy STARS represents 3 percent of the total
funding for facilities and equipment proposed in FAA‘s Capital
Investment Plan. When the $19.9 million planned for technology upgrades
is included in the projections for fiscal year 2004, the estimated cost
for STARS is about 4 percent of FAA‘s planned total facilities and
equipment budget (see fig. 3).
Figure 3: Allocation of Facilities and Equipment Funding for Fiscal
Year 2004 in FAA‘s Capital Investment Plan:
[See PDF for image] - graphic text:
[End of figure] - graphic text:
Notes:
GAO analysis of data from FAA.
Beginning with the fiscal year 2003 budget, FAA showed its facilities
and equipment budget in broad performance (mission) outcome areas.
Individual programs support these broad performance areas. For example,
STARS supports several mission areas, such as safety, security, and
capacity. To illustrate the impact of STARS on the other broad mission
areas, this figure consolidates the allocation for STARS in one
percentage and compares this percentage with the allocations to the
broad mission areas.
According to FAA‘s Capital Investment Plan, the proposed funding for
FAA‘s total facilities and equipment budget will increase slightly
during fiscal years 2005 through 2007, while the proposed funding for
STARS‘s development and deployment will continue to decline (see table
1). Therefore, the impact of these estimated STARS costs on the
facilities and equipment budget for safety, security, and capacity
would be even less during these fiscal years. The estimated costs of
technology upgrades, if included in the facilities and equipment cost
estimates for STARS for these 3 fiscal years, would likewise have a
similarly small impact on the facilities and equipment budget for these
fiscal years.
Table 1: Proposed Funding for FAA‘s Total Facilities and Equipment
Budget and for STARS‘s Development and Deployment:
Dollars in millions.
Total facilities and equipment budget; Dollars in millions: Fiscal year
2005: $3,129; Dollars in millions: Fiscal year 2006: $3, 202; Dollars
in millions: Fiscal year 2007: $3, 277.
STARS development and deployment; Dollars in millions: Fiscal year
2005: 43.5; Dollars in millions: Fiscal year 2006: 12.5; Dollars in
millions: Fiscal year 2007: 2.0.
STARS technology upgrades; Dollars in millions: Fiscal year 2005: 13.6;
Dollars in millions: Fiscal year 2006: 12.6; Dollars in millions:
Fiscal year 2007: 16.5.
[End of table]
Source: FAA.
Note: These data are from the Capital Investment Plan that FAA
submitted with its fiscal year 2003 budget request and FAA‘s March 2002
acquisition program baseline for STARS.
As FAA begins to operate STARS equipment at more terminal facilities,
the primary funding source for the program will shift from the
facilities and equipment budget account to the operations budget
account. Although FAA has projected that it will increasingly need
funding from (1) the operations account to cover the costs of operating
and maintaining STARS and (2) the facilities and equipment account to
upgrade STARS technology, it has not yet developed detailed estimates
of the program‘s operations and maintenance and technology upgrade
costs beyond fiscal year 2007. FAA officials noted that FAA, like most
federal agencies, develops its budget in 5-year plans and has not yet
begun to develop detailed budgets beyond fiscal year 2007.
Consequently, FAA does not currently know what impact the estimated
costs of operating, maintaining, and upgrading STARS will have on the
agency‘s future budgets. Furthermore, as we previously noted, FAA has
limited operational experience with STARS to use in estimating and
budgeting for the costs of the program‘s operations, maintenance, and
technology upgrades. Finally, the uncertainty of long-range estimates
makes it difficult to determine the impact of STARS‘s estimated costs
on FAA‘s budgets for outlying fiscal years.
After Deploying STARS at 74 Facilities, FAA May Modify Its Approach for
the Remaining Facilities:
FAA is committed to deploying STARS at the 74 terminal and support
facilities included in the STARS program, but for the nearly 100 other
facilities that remain to be modernized, the agency has at least three
options: It could deploy Raytheon‘s STARS, it could procure Lockheed
Martin Corporation‘s Common Automated Radar Terminal System (Common
ARTS) technology, or it could combine Raytheon‘s STARS display system
with Lockheed Martin‘s Common ARTS processing system. Between 1997 and
1999, FAA deployed Common ARTS to 131 small to medium-sized facilities
and to 5 larger facilities. From its experience with both systems, FAA
knows that each works independently. However, by combining the two
technologies, FAA could both (1) take advantage of the customized
software developed to resolve complex computer-human (controller and
technician) interface issues that accounted for a significant portion
of STARS‘s development costs and (2) continue to use portions of the
Common ARTS equipment that it recently deployed.
In October 2002, FAA asked Raytheon to consider the feasibility of
merging the STARS display system with the Common ARTS processing
system. According to the Manager of Terminal Automation, Raytheon
finished negotiating a subcontract with Lockheed Martin in January
2003, and FAA anticipates that the two contractors will now begin
analyzing the feasibility of a merger of components of the two systems.
After FAA receives the results of this analysis, which it expects in
April 2003, it plans to determine the cost and schedule for modernizing
the remaining terminal facilities.
Conclusions:
FAA lacks accurate, valid, current data on the STARS program‘s costs
and progress. Without such data, FAA is limited in its ability to
effectively oversee the contractor‘s performance and reliably estimate
future costs. FAA cannot use the contractor‘s cost performance reports
for these purposes until the contract‘s performance measurement
baseline has been revised to incorporate the results of contract
modifications, and FAA has verified that the revised baseline is
aligned with the current status of the contract. Furthermore, without
performing a risk assessment to identify the program‘s minimum, most
likely, and maximum expected costs, FAA cannot reliably determine the
level of uncertainty inherent in its cost estimates. Finally, by using
point values, rather than ranges or other appropriate measures, to
express its cost estimates, FAA is not revealing the extent of their
uncertainty. Its current estimates, expressed as point values, are
misleading because they convey undue certainty and limit disclosure of
the program‘s investment risks. As such, the estimates‘ usefulness to
program managers and congressional overseers is limited.
The earned value management guidance that FAA has adopted from DCMA and
the Acquisition System Toolset that it compiled for itself establish
clear procurement management policies and procedures that are
applicable to STARS and to subsequent terminal modernization programs.
In light of FAA‘s incomplete or inconsistent application of this
guidance to the STARS program thus far and the resulting cost overruns
and schedule delays, we believe it is essential that the agency revisit
the guidance, not only when it updates its baseline and performs an
integrated baseline review this spring, but also throughout the
remainder of STARS and throughout subsequent terminal modernization
programs.
Recommendations for Executive Action:
To improve FAA‘s management of STARS and of subsequent terminal
modernization programs and to provide the Congress with more reliable
information for overseeing these programs, we recommend that the
Secretary of Transportation direct the FAA Administrator to follow the
agency‘s guidance for managing major acquisition systems by:
* establishing, maintaining, and controlling an accurate, valid, and
current performance measurement baseline, which would include
negotiating all authorized, unpriced work within 3 months;
* conducting an integrated baseline review of any major contract
modifications within 6 months; and:
* preparing a rigorous life-cycle cost estimate, including a risk
assessment, in accordance with the Acquisition System Toolset‘s
guidance and identifying the level of uncertainty inherent in the
estimate.
Agency Comments:
We requested comments on a draft of this report from the Secretary of
Transportation or his designee. On January 23, 2003, FAA officials,
including the Manager of Terminal Automation, provided us with the
following oral comments on the draft. FAA agreed with the general tone
and intent of the draft report as well as with our conclusions and
recommendations. FAA noted that, in focusing its time and energy on
meeting the technical performance and schedule requirements of STARS,
it spent less time and energy on its business management of the
program. However, the manager noted, the agency is now taking steps to
reduce the program‘s cost risks, such as defining and negotiating with
the prime contractor all of the contract work planned for fiscal years
2003 through 2005. FAA also plans to conduct an integrated baseline
review when it receives its appropriation for fiscal year 2003.
According to FAA, our recommendations will further strengthen FAA‘s
commitment to improve the business management of the STARS program. FAA
provided additional clarifying and technical information, which we
incorporated as appropriate.
Scope and Methodology:
To conduct our work, we reviewed FAA‘s 5-year Capital Investment Plan,
which proposes funding for programs to modernize the national airspace
system. We also analyzed data from cost performance reports that the
STARS contractor developed for FAA. We interviewed FAA officials
responsible for air traffic control modernization planning and
budgeting. We also reviewed data from the FAA STARS program,
contracting officials, and the Department of Transportation‘s Office of
the Inspector General. We did not independently verify the cost and
performance data we received from FAA. We performed our work from
October 2002 through January 2003 in accordance with generally accepted
government auditing standards.
As arranged with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 5 days
after the date of this letter. At that time, we will send copies to
interested Members of Congress, the Secretary of Transportation, and
the FAA Administrator. We will also make copies available to others on
request. In addition, the report will be available at no charge on the
GAO Web site at http://www.gao.gov.
Should you or your staff have questions on matters discussed in this
report, please contact Gerald L. Dillingham at (202) 512-2834 or Keith
A. Rhodes at (202) 512-6412. We can also be reached by E-mail at
dillinghamg@gao.gov and rhodesk@gao.gov, respectively. GAO contacts
and key contributors to this report are listed in appendix I.
Sincerely yours,
Gerald L. Dillingham, Ph.D.
Director, Physical Infrastructure Issues:
Signed by Gerald L. Dillingham:
Keith A. Rhodes
Chief Technologist, Applied Research and Methods:
Signed by Keith A. Rhodes
[End of section]
Appendixes:
Appendix I: GAO Contacts and Staff Acknowledgments:
GAO Contacts:
Belva Martin (202) 512-2834
Madhav Panwar (202) 512-6228:
Staff Acknowledgments:
In addition to those individuals named above, Yvette Banks, Geraldine
Beard, Jennifer Echard, Elizabeth Eisenstadt, Elizabeth Marchak, and
Karen Richey made key contributions to this report.:
FOOTNOTES
[1] A previous version of STARS has been in use at smaller facilities
since 1999. Philadelphia is the 14th-largest terminal facility in terms
of handling operations under instrument flight rules. FAA‘s data show
that Philadelphia handled 686,000 operations between January and
December 2000.
[2] To support STARS operations at terminal facilities, FAA currently
has four systems dedicated to maintenance.
[3] The Department of Defense (DOD) is also procuring STARS for 153 of
its facilities.
[4] FAA currently plans to deploy STARS to a total of about 170
terminal and support facilities.
[5] U.S. General Accounting Office, National Airspace System: Status of
FAA‘s Standard Terminal Automation Replacement System, GAO-02-1071
(Washington, D.C.: Sept. 17, 2002).
[6] The report does not address DOD‘s efforts to deploy the equipment.
[7] Throughout this report, we use the term ’deploy“ to denote efforts
by FAA to put STARS software, hardware, and other supporting equipment
into a facility to test it and eventually use it to control traffic.
[8] FAA noted that the $1.46 billion would be predominantly for labor
costs that the agency would incur for STARS or a similar automation
system.
[9] Defense Contract Management Agency, Earned Value Management
Implementation Guide (October 1997).
[10] This account funds security activities related mainly to the
security of FAA facilities and equipment. Funding for some airport
security comes from another FAA account--Grants-in-Aid for Airports.
[11] GAO-02-1071.
[12] As previously noted, FAA officials said the $1.46 billion would be
labor costs that the agency would incur for STARS or a similar
automation system.
[13] Earned Value Management Guide (October 1997).
[14] The purpose of this review, conducted jointly by the agency and
the contractor, is to understand and assess the adequacy, accuracy, and
risks of a performance measurement baseline with respect to the
contract‘s work scope, schedule, technical requirements, and resource
availability.
[15] U.S. General Accounting Office, Air Traffic Control: Improved Cost
Information Needed to Make Billion Dollar Modernization Investment
Decisions, GAO/AIMD-97-20 (Washington, D.C.: Jan. 22, 1997).
[16] The 90 percent confidence level is based on an analysis of the
estimate‘s uncertainty made by using a Monte Carlo model.
GAO‘s Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO‘s commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO‘s Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ’Today‘s Reports,“ on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select ’Subscribe to daily E-mail alert for newly
released products“ under the GAO Reports heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW,
Room LM Washington,
D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.
General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.
20548:
