Air Traffic Control
FAA's Acquisition Management Has Improved, but Policies and Oversight Need Strengthening to Help Ensure Results Gao ID: GAO-05-23 November 12, 2004The Federal Aviation Administration's (FAA) multibillion-dollar effort to modernize the nation's air traffic control (ATC) system has resulted in cost, schedule, and performance shortfalls for over two decades and has been on GAO's list of high-risk federal programs since 1995. According to FAA, performance shortfalls were due, in part, to restrictions imposed by federal acquisition and personnel regulations. In response, Congress granted FAA exemptions in 1995 and directed it to develop a new acquisition management system. In this report, GAO compared FAA's AMS with (1) the FAR and (2) commercial best practices for major acquisitions, and (3) examined FAA's implementation of AMS and its progress in resolving problems with major acquisitions.
FAA's Acquisition Management System (AMS) is broader and less prescriptive than the Federal Acquisition Regulation (FAR), but both afford managers flexibility. AMS establishes an acquisition life-cycle management system, including both a contracting and program management system, whereas the FAR is primarily a contracting system. In addition, AMS takes the form of guidance--it is not regulatory, while the FAR is a set of published regulations--a legal foundation that has the force and effect of law that most federal agencies are required to follow. AMS provides some discipline for acquiring major ATC systems; however, it does not ensure a knowledge-based approach to acquisition found in the best commercial practices for managing commercial and DOD product developments that we have identified in numerous past reports. Best practices call for (1) use of explicit written criteria to attain specific knowledge at key decision points and (2) use of this knowledge by executives at the corporate level to determine whether a product is ready to move forward. Attainment and use of such knowledge by executives helps to avoid cost, schedule, and performance shortfalls that can occur if they commit to a system design prematurely. While AMS has some good features, including calling for key decision points, it falls short of best practices. GAO's review of seven major ATC systems and analysis of FAA's performance in acquiring major systems found that AMS has not resolved longstanding problems it experienced prior to its implementation of AMS--including developing requirements and managing software--and is just beginning to focus on how these acquisitions will improve the efficiency of ATC operations. While FAA has made progress by providing guidance for avoiding past weaknesses, it has not applied these improvements consistently. According to FAA officials, reorganization under and improved oversight by FAA's new performance-based Air Traffic Organization should help ensure greater consistency and an increased focus on results. Past GAO reports have demonstrated that the success of an acquisition process depends on good management, whether it be under AMS or the FAR.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-05-23, Air Traffic Control: FAA's Acquisition Management Has Improved, but Policies and Oversight Need Strengthening to Help Ensure Results
This is the accessible text file for GAO report number GAO-05-23
entitled 'Air Traffic Control: FAA's Acquisition Management Has
Improved, but Policies and Oversight Need Strengthening to Help Ensure
Results' which was released on December 13, 2004.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Chairman and Ranking Minority Member, Committee on
Government Reform, House of Representatives:
November 2004:
AIR TRAFFIC CONTROL:
FAA's Acquisition Management Has Improved, but Policies and Oversight
Need Strengthening to Help Ensure Results:
GAO-05-23:
GAO Highlights:
Highlights of GAO-05-23, a report to the Chairman and Ranking Minority
Member, Committee on Government Reform, House of Representatives:
Why GAO Did This Study:
The Federal Aviation Administration‘s (FAA) multibillion-dollar effort
to modernize the nation‘s air traffic control (ATC) system has resulted
in cost, schedule, and performance shortfalls for over two decades and
has been on GAO‘s list of high-risk federal programs since 1995.
According to FAA, performance shortfalls were due, in part, to
restrictions imposed by federal acquisition and personnel regulations.
In response, Congress granted FAA exemptions in 1995 and directed it to
develop a new acquisition management system.
In this report, GAO compared FAA‘s AMS with (1) the FAR and (2)
commercial best practices for major acquisitions, and (3) examined
FAA‘s implementation of AMS and its progress in resolving problems with
major acquisitions.
What GAO Found:
FAA‘s Acquisition Management System (AMS) is broader and less
prescriptive than the Federal Acquisition Regulation (FAR), but both
afford managers flexibility. AMS establishes an acquisition life-cycle
management system, including both a contracting and program management
system, whereas the FAR is primarily a contracting system. In addition,
AMS takes the form of guidance”it is not regulatory, while the FAR is a
set of published regulations”a legal foundation that has the force and
effect of law that most federal agencies are required to follow.
[See PDF for image]
[End of figure]
AMS provides some discipline for acquiring major ATC systems; however,
it does not ensure a knowledge-based approach to acquisition found in
the best commercial practices for managing commercial and DOD product
developments that we have identified in numerous past reports. Best
practices call for (1) use of explicit written criteria to attain
specific knowledge at key decision points and (2) use of this knowledge
by executives at the corporate level to determine whether a product is
ready to move forward. Attainment and use of such knowledge by
executives helps to avoid cost, schedule, and performance shortfalls
that can occur if they commit to a system design prematurely. While AMS
has some good features, including calling for key decision points, it
falls short of best practices.
GAO‘s review of seven major ATC systems and analysis of FAA‘s
performance in acquiring major systems found that AMS has not resolved
longstanding problems it experienced prior to its implementation of
AMS”including developing requirements and managing software”and is just
beginning to focus on how these acquisitions will improve the
efficiency of ATC operations. combined found that long-standing
problems While FAA has made progress by providing guidance for avoiding
past weaknesses, it has not applied these improvements consistently.
According to FAA officials, reorganization under and improved oversight
by FAA‘s new performance-based Air Traffic Organization should help
ensure greater consistency and an increased focus on results. Past GAO
reports have demonstrated that the success of an acquisition process
depends on good management, whether it be under AMS or the FAR.
What GAO Recommends:
GAO recommends that the Secretary of Transportation advise FAA to,
among other things, (1) improve its development of requirements and
management of software and (2) more closely align AMS with commercial
best practices.
In commenting on a draft of this report, FAA generally agreed with the
report‘s contents and said that our recommendations would be helpful to
them as they continue to refine AMS. They also provided us with
technical comments, which we have incorporated as appropriate.
www.gao.gov/cgi-bin/getrpt?GAO-05-23.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact JayEtta Z. Hecker
512-2834 or heckerj@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
AMS Is Broader and Less Prescriptive Than the FAR:
AMS Provides Some Discipline but Does Not Ensure a Knowledge-Based
Approach to Acquisition:
As Implemented, AMS Has Not Resolved Long-standing Acquisition
Problems, but FAA Is Beginning to Focus More on Results:
Conclusions:
Recommendations for Executive Action:
Agency Comments:
Appendixes:
Appendix I: FAA Has Begun Analyzing Spending Trends to Take a More
Strategic Approach to Procurement:
Appendix II: Objectives, Scope, and Methodology:
Appendix III: Comparison of the Scope and Flexibility of FAA's
Acquisition Management System and the Federal Acquisition Regulation
Process:
Background:
AMS Defines an Investment/Life-Cycle Project Management System:
Only a Portion of AMS Deals Directly with the Procurement Process:
AMS States a Nonregulatory FAA Policy:
AMS Chapter 3 Parallels a Subset of the FAR:
AMS Includes a Less Rigorous Competition Requirement Than Does the FAR:
The FAR Gives Procurement Professionals Tighter Control over
Procurement Decisions:
Although FAA Project Managers View AMS as More Efficient and Flexible
Than the FAR, Some Procurement Officials We Interviewed Do Not Agree:
Appendix IV: How FAA's Acquisition Policy Adapted Key Recommendations
Made by GAO and DOT (1996-2003):
FAA Refined AMS in Response to Recommendations:
Appendix V: Status of the Seven ATC Modernization Acquisitions That GAO
Reviewed:
Appendix VI: Key Contributors:
GAO Contacts:
Staff Acknowledgments:
Tables Tables:
Table 1: Structure of Best Practices Model for Major Product
Developments:
Table 2: Structure of AMS:
Table 3: Knowledge-Based Approach Called for in Our Best Practices
Model:
Table 4: AMS's Decision Points, Information Sources, and Oversight
Reviews:
Table 5: Description and Status of Seven Selected ATC Acquisitions:
Table 6: Internal and External Reviews of FAA's Use of AMS for
Acquiring Major ATC Systems:
Table 7: Program Costs for the Seven Systems We Reviewed:
Table 8: Comparison of AMS and the FAR:
Table 9: Key Recommendations Made to Improve FAA's Acquisition
Processes:
Figures:
Figure 1: Scope of AMS and the FAR:
Figure 2: Review Process under Our Best Practices Model and under AMS:
Figure 3: Key Milestones for Selected ATC Acquisitions Initiated before
and after AMS:
Figure 4: Our Analysis of FAA's Progress as of 2003 in Implementing Key
Elements of Training for Its Acquisition Workforce:
Abbreviations:
AMS: Acquisition Management System:
ASR-11: Airport Surveillance Radar, Model 11:
ATC: air traffic control:
ATO: Air Traffic Organization:
ATOP: Advanced Technologies and Oceanic Procedures:
BOE: Cost Basis of Estimate:
CFO: Chief Financial Officer:
CIO: Chief Information Officer:
COO: Chief Operating Officer:
CMMI: CMMI Capability Maturity Model Integration:
COTS: Commercial-off-the-shelf:
DOD: Department of Defense:
DOT: Department of Transportation:
DOTIG: Department of Transportation Office of Inspector General:
ERAM: En Route Automation Modernization:
F&E: Facilities and Equipment:
FAA: Federal Aviation Administration:
FAR: Federal Acquisition Regulation:
FMFIA: Federal Managers' Financial Integrity Act of 1982:
iCMM: integrated Capability Maturity Model:
ISC: Initial System Configuration:
IT: Information Technology:
ITIM: Information Technology Investment Management framework:
ITWS: Integrated Terminal Weather System:
JRC: Joint Resources Council:
LAAS: Local Area Augmentation System:
NEXCOM: Next Generation Air/Ground Communications System:
NDI: Non-developmental Item:
OMB: Office of Management and Budget:
RE&D: Research, Engineering and Development:
SEI: Software Engineering Institute:
SFFAS 4: Statements of Federal Financial Accounting Standards no. 4:
STARS: Standard Terminal Automation Replacement System:
WAAS: Wide Area Augmentation System:
Letter November 12, 2004:
The Honorable Tom Davis:
Chairman:
The Honorable Henry A. Waxman:
Ranking Minority Member:
Committee on Government Reform:
House of Representatives:
In late 1981, the Federal Aviation Administration (FAA) began a
modernization program to replace and upgrade the National Airspace
System's (NAS) equipment and facilities to meet the expected increase
in traffic volume, enhance the margin of safety, and increase the
efficiency of the air traffic control (ATC) system--the principal
component of the NAS. Historically, the modernization program has
experienced cost overruns, schedule delays, and performance shortfalls
of large proportions and has been on our list of high-risk programs
since 1995. To date, FAA has spent $41 billion and expects to spend an
additional $7.6 billion through fiscal year 2007 to, among other
things, finalize key modernization projects designed to replace radar,
navigation, communications, and information-processing
systems.[Footnote 1]
According to FAA, the performance shortfalls in its modernization
program were due, in part, to restrictions imposed by federal
acquisition and personnel requirements. In response, Congress passed
legislation in 1995 that granted FAA unique acquisition and personnel
exemptions, or flexibilities, and directed FAA to develop a new
acquisition management policy. FAA issued its new acquisition
management policy, called the Acquisition Management System (AMS), in
1996 and began using the new system instead of the Federal Acquisition
Regulation (FAR). To further address long-standing weaknesses in the
ATC modernization program, the President and Congress in 2000 directed
FAA to reorganize and establish a new organization. FAA has just begun
to do so.
Now that FAA has had several years to implement the earlier procurement
flexibilities, as well as some time to reorganize, some results of its
acquisition reform should be discernable. Moreover, FAA's experiences
in exercising its acquisition flexibilities could provide valuable
information to Congress in overseeing the use of these flexibilities.
You asked us to review the steps that FAA has taken to reform its
acquisition of major ATC systems and the impact of the reforms on FAA's
acquisition outcomes. Specifically, you asked us to (1) compare the
scope and flexibility of AMS and the FAR, (2) compare AMS with
commercial best practices for major acquisitions, and (3) examine FAA's
implementation of AMS and progress in addressing long-standing problems
with major acquisitions. In addition, you asked us to review FAA's
general procurement of goods and services; we cover this topic in
appendix I.
To address the first objective, we compared the topics addressed by,
and the implementation options afforded to contracting and procurement
officials under AMS and the FAR. To address the second objective, we
used a model of best practices that we derived from our body of work on
how leading private firms manage costly and complex product
developments and how the Department of Defense (DOD) manages major
weapon systems acquisitions.[Footnote 2] We used this model to assess
the extent to which FAA's acquisition management policy mirrors the
acquisition policies of high-performing organizations in the public and
private sectors. This model consists of four phases: (1) concept and
technology development; (2) product development, which includes both
integration and demonstration activities; (3) production; and (4)
operations and support. In between these four phases are three key
knowledge decision points at which commercial firms and the government
must have sufficient knowledge to make large investment decisions. To
address the third objective, we selected the seven ATC systems with the
largest budgets to explore the results of FAA's implementation of its
acquisition management policy and procedures and to determine how FAA
has addressed issues found to have contributed to cost, schedule, or
performance problems. In selecting these seven systems, we ensured that
some were initiated before and some after April 1996, when FAA
implemented AMS. While the results of these analyses are not
generalizable to all of FAA's major ATC acquisitions, they indicate the
extent to which the agency has made progress in addressing long-
standing problems we have identified. To further assess both the
implementation and the impact of FAA's acquisition reforms, we reviewed
our work on FAA's major ATC acquisition efforts since 1996 as well as
the work of the Department of Transportation's Office of Inspector
General (DOTIG), FAA, and others. We also reviewed the actions that FAA
has taken to refine AMS in response to internal and external reviews.
Finally, to review FAA's procurement of goods and services across the
agency, we used a commercial best-practices model for taking a more
strategic approach to procurement, along with interviews with key
agency officials, to determine whether FAA has begun to analyze
spending trends to identify opportunities to leverage its buying power.
We conducted our work from December 2003 through November 2004 in
accordance with generally accepted government auditing standards. See
appendix II for additional information on our objectives, scope, and
methodology.
Results in Brief:
AMS consists of broad guidance for acquisition life-cycle management--
from defining the requirements for a system through fielding
(deploying) and decommissioning it (removing it from service). This
broad guidance contrasts with the rather more detailed and prescriptive
contract-formation and contract-administration requirements contained
in the FAR. AMS is broader in scope because it addresses, among other
areas of life-cycle management, both contract and program management,
providing both policies and procedures for contracting and a toolset of
recommended practices for managing individual acquisition projects over
their life cycles. By contrast, the FAR focuses in far greater detail
on contracting policies and procedures. FAA managers believe they have
greater flexibility in interpreting and applying AMS than they would
have under the FAR, in part because, in areas addressed by both, AMS is
less directive than the FAR. For example, although AMS states a
"preference" for competition, FAA personnel may use single-source
contracting when necessary to fulfill FAA's mission. By contrast, other
federal agency contracting officials operating under the FAR are
generally required to seek "full and open competition"--a more rigorous
standard. These other agency officials can generally use sole-source or
limited-competition contracting only after higher-level agency
procurement officials have approved a written justification. In
addition, FAA contracting personnel operate as part of acquisition
teams that are responsible to program managers; under the FAR,
contracting decisions are made by contracting personnel who are
responsible only to contracting officials. Nonetheless, the FAR also
affords flexibility because it encourages innovation and addresses a
wide selection of contracting methods; therefore, procurement officials
can choose the approach that they consider most appropriate to their
procurement. According to some current and former FAA procurement
officials with experience in using both the FAR and AMS, the FAR may
appear inflexible and cumbersome to inexperienced managers, but those
who are familiar with it can navigate it effectively.
AMS provides some discipline through its various phases, activities,
and decision points for acquiring major ATC systems; however, it does
not ensure the use of a knowledge-based approach found in the best
practices for managing commercial product developments and DOD
acquisitions[Footnote 3] that we have identified in numerous past
reports. Commercial best practices call for specific knowledge to be
captured and used by corporate-level decision-makers to determine
whether a product has reached a level of development (product maturity)
sufficient to demonstrate its readiness to move forward in the
acquisition process. The capture of such knowledge and its use by
executives helps to avoid cost overruns, schedule slips, and
performance shortfalls that can occur if decision-makers commit to a
system design before acquiring critical technology, design, or
manufacturing knowledge. AMS has some good features, which indicate a
process that has some elements of discipline. For example, like the
best practices model, AMS identifies critical junctures that it terms
"decision points," the first three of which call for the preparation of
detailed technical and programmatic information that FAA's corporate
executive-level body, the Joint Resources Council,[Footnote 4] can use
to assess whether or not FAA should initiate an acquisition program.
However, AMS departs from recognized best practices primarily by (1)
not requiring the attainment of specific knowledge satisfying explicit
written criteria for decision-makers to use at each key decision point
and (2) not requiring corporate executive-level oversight at all key
decisions. For example, AMS allows the Joint Resources Council to
delegate two key decisions--the decision to begin production and the
decision to place a system in service. FAA maintains that this approach
gives program managers flexibility, expedites decision-making, and
allows those executives with the most knowledge about a major
acquisition to make key decisions about its continued development.
FAA's reliance on delegation assumes that managers will inform their
superiors if they are unable to meet the performance schedules and
system requirements approved by the Joint Resources Council. However,
best practices call for more than this, including the use of measurable
criteria at key points in the acquisition process to ensure that
specific knowledge has been captured and the independent review of this
knowledge by corporate executive-level decision-makers before the
acquisition moves forward in its development. These criteria and
reviews are particularly important for acquisitions that require a
large funding commitment, such as those that include the production of
multiple costly units (e.g., radars and controller workstations). In
addition, oversight at the corporate-executive or agencywide level is
needed to ensure consideration of an acquisition's likely impact on
other agency projects or operations. These departures from best
practices put FAA's major ATC acquisitions at risk of cost, schedule,
and performance shortfalls. We are making recommendations to the
Secretary of Transportation to align AMS more closely with commercial
best practices.
According to our review of seven major ATC systems and analysis of
FAA's performance in acquiring major systems, AMS has not resolved
management problems that FAA experienced before it implemented AMS, but
the agency is beginning to focus more on the expected results of its
major acquisitions. (See table 5.) Specifically, our review found that
AMS did not call for requirements that were specific enough to minimize
the development of further requirements (requirements growth) or
unplanned work in five of these systems. This lack of specificity
resulted in the inadequate development or definition of requirements,
requirements growth, unplanned work, or a reduction in performance for
five of these systems. In addition, for three of these systems, FAA
underestimated the difficulty of modifying available software to
fulfill its mission needs. Consequently, FAA encountered unexpected
software development needs, higher costs, and schedule delays. Because
AMS guidance was not sufficient to account for the risks associated
with modifying available software, the two systems we reviewed that
were initiated after AMS's implementation--though currently meeting
cost and schedule milestones--are nevertheless showing symptoms of
FAA's past problems with developing requirements and managing software.
It is too soon to tell if these two systems will remain within their
cost, schedule, and performance parameters. In addition, our work on
FAA's major acquisitions, along with that of the DOTIG and others, has
shown that many of the problems FAA experienced in acquiring major
systems before 1996 persist under AMS and that effective acquisition
management, rather than the use of a specific contracting process
(e.g., the FAR or AMS) is the key to successful acquisitions. To its
credit, FAA is beginning to focus more on results, largely through its
new Air Traffic Organization, which has been charged with taking a more
performance-based approach to managing the agency's major acquisitions.
This approach includes implementing a training framework for FAA's
acquisition workforce. While FAA has taken some steps to develop an
evaluation program with criteria for measuring the extent to which this
framework is achieving organizational goals by improving the knowledge
base of FAA's acquisition workforce, at the time of our audit FAA had
no plans to conduct a comprehensive evaluation. We are making
recommendations to the Secretary of Transportation to improve FAA's
development of requirements and management of complex software, and to
comprehensively evaluate FAA's implementation of the training framework
to ensure that it is having the intended effect of improving the
knowledge base of FAA's acquisition workforce. In commenting on a draft
of this report, FAA said that it generally agreed with the report's
contents and said that our recommendations would be helpful to them as
they continue to refine AMS.
Background:
Maintaining that federal procurement requirements contributed to some
of its cost, schedule, and performance problems in the 1980s and early
1990s, FAA sought a statutory exemption from the federal acquisition
system,[Footnote 5] including the FAR, and those parts of title 5 of
the United States Code, parts II and III, that govern federal civilian
personnel management. According to FAA, exemptions from these
requirements would enable it to streamline its acquisition processes,
be more responsive to the airline industry's needs, and increase the
efficiency of ATC operations while maintaining safety. Congress enacted
legislation in November 1995 that exempted FAA from key federal
procurement statutes and the FAR, and directed FAA to develop a new
acquisition management system. In response to these legislative
initiatives, FAA implemented a new, streamlined acquisition process--
the Acquisition Management System (AMS)--on April 1, 1996.
We developed a knowledge-based model of commercial best practices based
on our findings about how leading private firms manage costly and
complex acquisitions effectively--that is, within cost, schedule, and
performance targets. The use of this knowledge-based model has been
found to reduce the risks associated with developing products and
increase the likelihood of successful outcomes. The model divides the
product development cycle into four phases and related activities.
Table 1 presents these phases and activities and explains what takes
place during each.
Table 1: Structure of Best Practices Model for Major Product
Developments:
Phase/Activity: 1.Concept and technology development;
What occurs during this phase or activity: Leading companies work to
understand their mission needs and confirm that the technologies to be
used are mature; that is, the technologies needed to meet essential
product requirements have been demonstrated to work in their intended
environment.
Phase/Activity: 2. Product development: * Integration;
What occurs during this phase or activity: Components and subsystems
are integrated into the product to stabilize the overall system design
and show that the design can meet the product requirements.
Phase/Activity: 2. Product development: * Demonstration;
What occurs during this phase or activity: Tests show that the product
will work as required and can be manufactured within targets.
Phase/Activity: 3. Production;
What occurs during this phase or activity: Operational test articles
are built.
Phase/Activity: 4. Operations and support;
What occurs during this phase or activity: Our best practices model
does not explicitly cover operations and support activities; however,
this phase focuses on maintenance of the system through its retirement.
Source: GAO.
[End of table]
AMS provides guidance for selecting and overseeing investments over
their life cycle. Like our best practices model, it is divided into
phases and activities, although the divisions sometimes occur at
different points. Table 2 summarizes AMS's phases and activities.
Table 2: Structure of AMS:
Phase/Activity: Needs and solution identification: * Mission analysis;
What occurs during this phase or activity: FAA identifies a capability
shortfall and determines that it needs an investment to better carry
out its mission. Recently, FAA began analyzing its mission needs within
the context of its overall goals for the National Airspace System.
Phase/Activity: Needs and solution identification: * Investment
analysis;
What occurs during this phase or activity: FAA, using an investment
analysis team, evaluates alternatives, selects practical and affordable
solutions, and develops a baseline of cost, schedule, and performance
requirements. This document is called the acquisition program baseline.
Phase/Activity: Solution implementation: * System integration;
What occurs during this phase or activity: Both hardware and software
components and subsystems are integrated into a product. Also, intra-
and intersystem compatibility are tested and analyzed.
Phase/Activity: Solution implementation: * System demonstration;
What occurs during this phase or activity: Tests show that the product
can work as required and be manufactured within targets.
Phase/Activity: Solution implementation: * System production;
What occurs during this phase or activity: All activities are carried
out to produce needed quantities. Each end item is tested before it
leaves the factory to verify that it conforms to specifications and is
free from manufacturing defects.
Phase/Activity: In-service management;
What occurs during this phase or activity: All required activities are
carried out, including directly operating, providing maintenance
functions (both scheduled and unscheduled), and furnishing technical
and logistics support for the maintenance of FAA systems, subsystems,
services, or equipment.
Source: FAA.
[End of table]
To implement the new, performance-based organization for managing ATC
modernization and operations, as the President and Congress directed in
2000, FAA appointed a chief operating officer in August 2003 and
formally established the Air Traffic Organization (ATO) in February
2004. ATO, under the direction of a six-member executive council, is
now responsible for further implementing acquisition reforms for major
ATC systems.
AMS Is Broader and Less Prescriptive Than the FAR:
AMS establishes an acquisition life-cycle management system that
encompasses both contracting and program management, whereas the FAR is
primarily a contracting system that focuses on contract formation and
contract administration. As a result, AMS is broader in scope than the
FAR. See figure 1. In addition, AMS takes the form of guidance. This
guidance is expressed in documentation of FAA policy, handbooks,
templates, flowcharts, forms, and standard contract language. It is not
regulatory. By contrast, the FAR is a set of published regulations--a
legal foundation that has the force and effect of law for the federal
agencies that are required to follow it.[Footnote 6] Furthermore, the
FAR is more detailed and prescriptive in establishing contracting
requirements and can require more administrative involvement. This
fundamental difference between AMS and the FAR may suggest to some that
AMS is more flexible. FAA personnel can choose how to apply AMS's
provisions to a major acquisition. Nonetheless, procurement officials
under the FAR also have flexibility because the FAR encourages
innovation consistent with its direction (and other applicable legal
requirements), provides a wide selection of contracting solutions, and
permits contracting officials to choose the methods that they consider
most suitable for a given situation.
Figure 1: Scope of AMS and the FAR:
[See PDF for image]
Note: AMS provides policy for the four phases of life-cycle management,
as well as 14 functional areas, (e.g., test and evaluation, human
factors, procurement, real estate, security, and systems engineering).
[A] The NAS in-service decision is a key program milestone that
authorizes the deployment of a system into the National Airspace System
after thoroughly testing the system to verify its operational
readiness.
[End of figure]
AMS Addresses Both Procurement and Project Management, Whereas the FAR
Focuses Primarily and in Far Greater Detail on Procurement:
AMS comprises six policy sections and five appendixes.[Footnote 7] The
procurement policy section of AMS covers a range of topics, including
contract funding and administration, contracting with small and
disadvantaged businesses, and compliance with labor laws. According to
this section, competition is FAA's preferred method of contracting, but
single-source contracting is permitted when appropriate to fulfill the
agency's mission. This policy section also describes the procurement of
commercially available or nondevelopmental items.
Other sections of AMS cover project management tools that the FAR does
not address, such as investment analysis, configuration
management,[Footnote 8] and integrated logistics support.[Footnote 9]
AMS also addresses areas that fall outside project management and
procurement, including real property management--an area that becomes
important when FAA must lease or purchase real property so that it can
install ATC systems such as radars or antennas on property that it does
not currently own. FAA's policy directs FAA staff to "conduct this
business in a fair and equitable manner following best practices."
Although the FAR includes requirements that address procurement
planning[Footnote 10] and major systems acquisition,[Footnote 11] it
does so only in the context of government procurement policy and
procedure. Agencies subject to the FAR find the broader program
planning requirements, which appear in AMS but not in the FAR, in
documents such as the Office of Management and Budget's Circular A-109
and in their own planning guidance. For example, DOD has issued a
series of directives and instructions on this subject.[Footnote 12]
The contracting procedures set forth in section 3 of AMS do not
prescribe detailed contracting procedures for various categories of
procurements, as do those detailed under the FAR. Instead, AMS provides
two basic contracting models for obtaining products and services
through FAA's contracting process. The first model is called "Complex
and Noncommercial Source Selection" and is used for complex, large-
dollar, developmental, noncommercial items and services. This is the
model that typically would be used for investments approved by the
Joint Resources Council. The second model is called the "Commercial and
Simplified Purchase Method" and is typically used for commercial items
that are less complex and less costly. Procurements of such products or
services may be routine in nature and are generally purchased on a
fixed-price basis. Generally, source selection under AMS follows a
screening process, with the awardee being selected on a "best value"
basis from among those who remain in consideration when the selection
is made.
AMS Provides Broad Guidance While the FAR Establishes Detailed
Requirements, but Managers Have Flexibility under Both:
AMS sets out a nonregulatory FAA policy that is binding on FAA
personnel as FAA employees. AMS also sets out other guidelines that FAA
states should be followed unless there is a rational basis for doing
otherwise. AMS is subject to such internal controls and enforcement as
the Administrator decides and to general overarching legal
requirements, such as the Government Performance and Results Act of
1993 (GPRA).[Footnote 13]FAA has also deemed certain acquisition laws
applicable to its procurements (sometimes with modifications), such as
the Service Contract Act.[Footnote 14] There is also a legal
requirement, created by the 1995 legislation exempting FAA from the
FAR, that small and socially or economically disadvantaged firms be
given all reasonable opportunities to receive contract awards. FAA has
adopted a dispute resolution process with some legal
underpinnings.[Footnote 15] Otherwise, as the preface to AMS states,
"nothing in this document creates or conveys any substantive [legal]
rights." In short, although FAA is subject to the general legal
requirement that government decisions cannot be arbitrary or
capricious, AMS does not establish regulatory requirements for the
conduct of procurements and does not create or convey substantive legal
rights.
In contrast to AMS, the FAR is a set of published regulatory
requirements. It has the force and effect of law, and agencies that are
subject to it are bound to follow it. The FAR's requirements provide
for a range of procurement strategies and approaches. In addition to
negotiated procurement methods, it allows two-step sealed-bid and two-
phase design-build methods,[Footnote 16] among others. It includes
streamlined procedures for soliciting and evaluating offers to furnish
commercial items, as well as permits the use of simplified acquisition
procedures in a broad range of procurements. Furthermore, the FAR
supports a diverse selection of available contract types, product-
testing tools, and other tools that an agency's contracting personnel
may select when conducting an acquisition to meet the agency's needs.
Although contracting personnel in agencies subject to the FAR are
required to comply with it, they enjoy broad discretion in their
management of procurements. For example, the FAR allows wide latitude
in drafting requirements statements, from performance-based statements
of work to design specifications as necessary. It allows broad
discretion in framing solicitations and in conducting procurements,
including scoring proposals, determining how negotiations will be
conducted, eliminating firms whose proposals are not competitive, and
selecting the awardees whose proposals afford the government the best
value when evaluated against the selection criteria established in the
solicitations.
Because AMS consists of broad guidance while the FAR comprises detailed
and prescriptive regulatory requirements, FAA managers view AMS as
giving them more flexibility than they would have under the FAR,
particularly in two areas--competition and oversight. Whereas the FAR
generally requires full and open competition, AMS calls for providing
"reasonable access to" competition to firms interested in obtaining
contracts--a less rigorous standard than full and open competition. AMS
further states that the "preferred" method of selecting sources is to
compete requirements among two or more sources. By contrast, full and
open competition requires that all responsible sources be permitted to
compete.[Footnote 17] Under AMS, there is no policy that firms that
want to participate actually get a chance to do so. Rather, FAA told us
that its system is beneficial because the agency can use screening
requests to preselect competing firms, eliminating those firms that FAA
believes are not likely to receive an award. The following example
illustrates the differences between AMS and the FAR in their respective
requirements on exceptions to competition. FAA may contract with a
single source when this approach is determined to be in the best
interest of FAA.[Footnote 18] The FAR, however, allows exceptions to
full and open competition only for certain specified conditions (such
as unusual and compelling urgency or the availability of only one
source). The FAR describes in detail the circumstances of these
conditions and the requirements for using them as justification for not
providing for full and open competition. The FAR also requires the
contracting officer to prepare a justification document that must
generally be approved by higher-level agency procurement officials (up
to the agency's senior procurement executive) depending on the
estimated dollar value of the procurement. The content of this
justification is prescribed by the FAR. When not providing for full and
open competition, the contracting officer is required under the FAR to
solicit offers from as many potential sources as is practicable under
the circumstances. The FAR prohibits contracting if the justification
for less than full and open competition results from a lack of advanced
planning. For a more detailed comparison of AMS and the FAR, see
appendix III.
Although some of the FAA personnel we interviewed see AMS as more
efficient and flexible than the FAR, other current and former FAA
procurement officials we interviewed who have experience using both the
FAR and AMS did not agree that AMS is more flexible than the FAR.
According to these officials, the FAR may appear inflexible and
cumbersome to persons who lack experience with it, but those who are
familiar with it are able to navigate its complexities effectively. The
FAR requires full and open competition, but as experienced procurement
personnel know, the system does not break down when emergencies
necessitate quick and decisive action. For example, we recently
reported that agencies generally complied with applicable FAR
requirements in awarding new contracts for work in Iraq using other
than full and open competition.[Footnote 19]In some circumstances, the
government's legitimate need for prompt action was sufficient to
justify selecting a contractor on an expedited basis from among the
firms that appeared able to meet the government's emergency need. In
other cases, the agencies reasonably determined that only one source
could meet their requirements.
AMS Provides Some Discipline but Does Not Ensure a Knowledge-Based
Approach to Acquisition:
AMS provides some discipline through its various phases, activities,
and decision points for acquiring major ATC systems; however, it does
not ensure the use of a knowledge-based approach found in the best
practices for managing commercial product developments and DOD
acquisitions that we have identified in numerous past reports.[Footnote
20]Commercial best practices call for specific knowledge to be captured
and used by corporate-level decision-makers to determine whether a
product has reached a level of development (product maturity)
sufficient to demonstrate its readiness to move forward in the
acquisition process. The capture of such knowledge and its use by
executives helps to avoid cost overruns, schedule slips, and
performance shortfalls that can occur if decision-makers commit to a
system design before acquiring critical technology, design, or
manufacturing knowledge. The absence of these key best practices under
AMS puts FAA's major ATC acquisitions at risk of cost overruns,
schedule slips, and performance shortfalls.
Best Practices for Managing Acquisitions Call for a Knowledge-Based
Approach, Including Criteria for Knowledge Needed and Oversight at the
Corporate Executive Level:
Commercial best practices call for managing acquisitions using a
knowledge-based approach, including (1) using established criteria to
attain specific knowledge at three critical junctures in the
acquisition cycle, which we call knowledge points, and (2) requiring
oversight at the corporate executive level for each of these knowledge
points. For example, at each knowledge point, successful product
developers apply specific indicators, or criteria, to determine whether
they have attained the knowledge they need to move to the next phase or
activity in the acquisition process. Such developers also conduct
corporate executive-level reviews to ensure that they obtain the
insights and perspectives of stakeholders throughout their
organization. If the knowledge attained does not meet the criteria for
advancement or if the executive reviewers determine that further
development is inconsistent with their priorities, the acquisition does
not move forward. Table 3 summarizes the knowledge points, criteria,
oversight reviews, and timing of oversight reviews included in our
model of best practices for major acquisitions.
Table 3: Knowledge-Based Approach Called for in Our Best Practices
Model:
Knowledge point: 1. Resources and needs matched;
Criteria:
* Match customers' needs with available resources--technology, design,
time, and funding;
* Demonstrate that technologies needed to meet essential product
requirements can work in intended environment;
* Complete a preliminary product design using systems engineering to
balance customers' desires and available resources;
Oversight review: Executive-level review required to initiate the
program;
Timing of oversight review: Knowledge point 1 should precede the
commitment to begin product development.
Knowledge point: 2. Product design stable;
Criteria:
* Complete 90 percent of design drawings by critical design review;
* Obtain stakeholders' concurrence that drawings are complete and
producible;
* Review subsystem and system designs;
* Demonstrate with prototype that design meets users' requirements;
* Identify critical manufacturing processes;
Oversight review: Executive-level review required to move to
demonstration;
Timing of oversight review: Knowledge point 2 should precede the
commitment to build prototypes to demonstrate the design.
Knowledge point: 3. Production processes mature;
Criteria:
* Demonstrate manufacturing processes;
* Build and test production prototypes;
* Test production-representative prototypes to achieve reliability
goals;
* Test production-representative prototypes to demonstrate product
performance in operational environment;
* Collect statistical process control data;
Oversight review: Executive-level review required to move to
production;
Timing of oversight review: Knowledge point 3 should precede the
commitment to begin production.
Source: GAO.
[End of table]
Experience with commercial best practices has shown that to the extent
that the level of knowledge called for at each knowledge point is not
attained, organizations take on risks in the form of unknowns that will
persist into the later stages of development, where they will take more
time and money to resolve if they become problems. Such problems lead
to cost increases and schedule delays.
AMS Has Some Good Features but Does Not Ensure That High Levels of
Knowledge Are Attained Before Major Commitments Are Made:
AMS has some good features, including phases and key decision points
indicative of an acquisition process that has some elements of
discipline; however, AMS does not ensure that high levels of knowledge
are attained and that corporate executive-level reviews occur before
major commitments of agency resources are made. For example, like the
best practices model, AMS identifies critical junctures, which it terms
"decision points." Three of these decision points occur during the
initial acquisition phase (mission need, initial investment, and the
final investment decision). A fourth decision point occurs before
production, and a fifth decision point occurs before the start of the
final acquisition phase (in-service management). AMS also calls for
detailed technical and programmatic information that decision-makers
can use at the first three decision points to assess whether or not FAA
should initiate an acquisition program. This information includes a
final requirements document, a final acquisition program baseline, a
final investment analysis report, an acquisition strategy paper, and an
integrated program plan. Finally, AMS, like our best practices model,
calls for senior executives to review the information and determine
whether the acquisition is ready to move forward. The FAA executives
who make the decisions at these points include associate and assistant
administrators, acquisition executives, the chief financial officer,
the chief information officer, and legal counsel; they form the Joint
Resources Council (JRC), FAA's senior decision-making body for major
ATC acquisitions. Table 4 summarizes this information.
Table 4: AMS's Decision Points, Information Sources, and Oversight
Reviews:
Decision point by phase/activity:
Phase: Needs and solution identification;
* Activity: Mission analysis;
Decision point: Mission need decision;
Information sources and oversight reviews: Information sources: Input
from users in the field and mission need statement;
Oversight review: JRC review called for to move from mission analysis
to investment analysis.
Decision point by phase/activity:
Phase: Needs and solution identification;
* Activity: Investment analysis;
Decision Point: Initial investment decision;
Information sources and oversight reviews: Information sources: Initial
investment analysis report, initial life-cycle program baseline for the
most viable alternative, updated initial requirements document and
action plan for final investment analysis;
Oversight review: JRC review called for to select a preferred solution.
Decision point: Final investment decision;
Information sources and oversight reviews: Information sources: Final
requirements document, final acquisition program baseline, final
investment analysis report, acquisition strategy paper, integrated
program plan;
Oversight review: JRC review called for to move from investment
analysis to solution implementation.
Decision point by phase/activity:
Phase: Solution implementation;
* Activity: System integration;
* Activity: System demonstration;
Decision point: Production decision;
Information sources and oversight reviews: Information sources:
Determined by JRC;
Oversight review: JRC may retain or delegate decision making authority.
Decision point by phase/activity:
* Activity: System production;
Decision point: In-service decision;
Information sources and oversight reviews: Information Sources:
Determined by JRC;
Oversight Review: JRC review called for to move from solution
implementation to in-service management; however, the JRC may retain or
delegate decision making authority.
Decision point by phase/activity:
Phase: In-service management;
Source: GAO analysis of FAA data.
Note: In this report, we place FAA's "mission analysis" and "investment
analyses" activities in the "Needs and Solution Identification" phase
to facilitate comparison with the "concept and technology development"
phase in our best practices model. Similarly, we place "system
integration" and "system demonstration" in the solution implementation
phase for comparative purposes.
[End of table]
AMS departs from the best practices model in two key ways--it does not
call for high levels of knowledge to be attained at three critical
junctures (knowledge points), and does not call for corporate
executive-level oversight at one of five junctures. Specifically, AMS
does not establish explicit, written criteria for (1) the information
needed to determine technology maturity at solution implementation, (2)
releasable drawings at critical design review and production process
controls at production. Our best practices model calls for attaining
specific knowledge and setting out criteria for what information should
be available to help organizations minimize risks in the form of
unknowns. Risks associated with such unknowns can persist into the
later stages of development, where they can take more time and money to
resolve if they become problems, potentially leading to cost increases
and schedule delays.
In addition, AMS does not provide for corporate executive-level
oversight reviews at two of the three key junctures where our best
practices model calls for such reviews. Although AMS calls for three
Joint Resources Council reviews during the initial acquisition phase--
while our model calls for a single corporate executive-level review--
AMS allows the council to delegate its oversight responsibility later
in the acquisition process to the program managers within the service
organization responsible for an acquisition. By contrast, our model
calls for two corporate executive-level reviews later in the
acquisition process.
According to FAA, its approach gives program managers flexibility,
expedites decision-making, and allows the executives with the most
knowledge about a major acquisition to make key decisions about its
continued development. FAA's reliance on this approach assumes that the
program managers will inform higher-level managers if they are unable
to meet the performance schedules and systems requirements approved by
the Joint Resources Council. However, although program managers may
have the most knowledge about their particular acquisition, they may
not have the agencywide perspective of the Joint Resources Council
members. Having an agencywide perspective, including a broad
understanding of an acquisition's potential impact on other agency
projects and operations, is especially critical when an acquisition
includes the production of multiple units and requires a substantial
commitment of agency resources, as do FAA's primarily multimillion-
dollar acquisitions, such as controller workstations and radars.
Because decisions about moving a major acquisition forward require both
a program manager's specific knowledge of the acquisition itself and a
senior executive's understanding of the acquisition's potential impact
on other agency projects and operations, our best practices model calls
for both measurable criteria at key points in the acquisition process
to ensure that specific knowledge has been captured and corporate
executive-level reviews to ensure that senior decision-makers have the
opportunity to independently consider this knowledge. Without higher-
level reviews such as our best practices model recommends and the Joint
Resources Council could provide later as well as early in the
acquisition process, FAA cannot ensure that it has fully considered the
impact of advancing an acquisition on other agency projects and
operations. This opportunity for full consideration is a central
advantage of managing acquisitions as a portfolio, as we concluded in
our August 2004 report on FAA's information technology investment
management process.[Footnote 21]
Figure 2 contrasts FAA's process for reviewing an acquisition's
progress under AMS with the process that we found leads to successful
commercial acquisitions.
Figure 2: Review Process under Our Best Practices Model and under AMS:
[See PDF for image]
[A] To facilitate the comparison of AMS with out best practices model
in this report, we have done the following: (1) placed FAA's "Mission
Analysis" and "Investment Analyses" activities in the "Needs and
Solution Identification" phase to make it comparable with the "concept
and technology development" phase in our best practices model; (2)
depicted only the final investment decision point, recognizing that the
investment analysis phase includes an initial investment decision; and
(3) placed "system integration" and "system demonstration" in the
solution implementation phase.
[B] AMS does not explicitly call for a design review decision point,
which would fall between system integration and system demonstration.
[C] The in-service decision is a key program milestone. It authorizes
the deployment of a system into the National Airspace System. At times,
the JRC delegates its decision authority for the production and in-
service decisions to service organizations.
[End of figure]
To its credit, FAA continues to improve its AMS process. For example,
the agency is currently modifying its mission needs activity to make
the selection of major ATC acquisitions more consistent with the
overall goals of modernizing the National Airspace System. In addition,
the Air Traffic Organization has established an executive council to
review major acquisitions before they are sent to the Joint Resources
Council. This review is designed to screen acquisitions to determine
which ones are important enough to warrant higher-level review by the
Council. Finally, FAA is currently revising AMS to bring it in line
with the Office of Management and Budget's guidance. Specifically, the
agency is incorporating OMB Exhibit 300, which provides the investment
justifications and management plans required for major ATC
acquisitions.
As Implemented, AMS Has Not Resolved Long-standing Acquisition
Problems, but FAA Is Beginning to Focus More on Results:
According to our review of seven major ATC systems and analysis of
FAA's performance in acquiring major systems, AMS has not resolved the
long-standing problems that FAA experienced before implementing AMS,
but the agency is beginning to focus more on the expected results of
its major acquisitions. (See table 5.) Specifically, our review found
that AMS guidance did not call for requirements that were specific
enough to minimize requirements growth or unplanned work for five of
these systems. This lack of specificity resulted in the inadequate
development or definition of requirements, growth in requirements,
unplanned work, or a reduction in performance for five of these
systems. In addition, for three of these systems, FAA underestimated
the difficulty of modifying available software to fulfill its mission
needs. Because AMS guidance was not sufficient to account for the risks
associated with modifying available software, FAA encountered
unexpected software development needs, higher costs, and schedule
delays. The two systems we reviewed that were initiated after AMS was
implemented are currently meeting cost and schedule milestones;
however, both systems are showing symptoms of FAA's past problems with
developing requirements and managing software, and it is too soon to
tell if these programs will remain within their cost, schedule, and
performance parameters. In addition, our work on FAA's major
acquisitions, along with that of the DOTIG and others has shown that
the problems FAA experienced before 1996 in acquiring major systems
persist under AMS and that effective acquisition management, rather
than the use of a specific contracting process (e.g., the FAR or AMS)
is key to successful acquisitions. To its credit, FAA is beginning to
focus more on results, largely through its new Air Traffic
Organization, which has been charged with taking a more performance-
based approach to managing the agency's acquisitions.
Table 5: Description and Status of Seven Selected ATC Acquisitions:
Dollars in millions;
Project and description: STARS--new controller and maintenance
workstations to replace the legacy system at terminal air traffic
control facilities[A];
Original cost: $940.0;
Current cost: $1,460.0;
Original schedule: 1998;
Current schedule: 2003;
Acquisition issues and status: STARS is a joint FAA and DoD program.
STARS delays and cost increases resulted from poor requirements
definition and schedule estimates. STARS is fully operational at 25 FAA
terminal radar facilities and 17 DoD facilities. Only 50 of the planned
172 systems are being deployed. STARS had difficulties in achieving
many human factor requirements for improving system efficiency and
safety.
Project and description: ASR-11--digital radar for terminal
environments;
Original cost: $743;
Current cost: $891.7;
Original schedule: 1997;
Current schedule: 2013;
Acquisition issues and status: ASR-11 was approved for its in-service
decision in September 2003 and is being deployed at 108 sites. These
systems are being deployed at a slower pace than originally planned
because of budget cuts and deferrals.
Project and description: ITWS--computer processors and displays to
automate weather data near the airport;
Original cost: $276.1;
Current cost: $288.3;
Original schedule: September 2001;
Current schedule: 2002;
Acquisition issues and status: Currently, six ITWS systems are
operational. In May 2004, the ATO Executive Council rebaselined the
program to include a new weather-forecasting capability into the
production baseline. FAA proposes to defer 12 of the 34 systems it
planned to procure.
Project and description: LAAS--a precision approach and landing system
that augments the Global Positioning System;
Original cost: $530.1;
Current cost: $696.1;
Original schedule: 2002;
Current schedule: Deferred at least until 2009;
Acquisition issues and status: LAAS has been adversely affected by poor
requirements development, a lack of understanding of its technical
complexity, incomplete software development, and an unrealistic
development schedule. Unresolved radio interference precludes the safe
operation of LAAS. As a result, FAA has delayed national deployment to
continue further research on this issue.
Project and description: NEXCOM--digital radios to improve air traffic
communications;
Original cost: $318.4;
Current cost: $318.4;
October 2002;
Current schedule: 2004;
Acquisition issues and status: NEXCOM program delays were due to
misunderstanding of a program requirement and testing procedures.
NEXCOM was recently approved for its in-service decision in July 2004.
Project and description: ATOP--new workstations and processing
capability to control ocean air traffic;
Original cost: $548.2;
Current cost: $548.2;
Original schedule: June 2004;
Current schedule: 2004;
Acquisition issues and status: ATOP achieved its acquisition program
baseline objectives; however, this baseline does not reflect program
delays and cost increases resulting from poor requirements development,
unrealistic schedule estimates, and inadequate evaluation of software
complexity.
Project and description: ERAM--upgrades the existing en route system
with improved hardware and software;
Original cost: $3,649.0;
Current cost: $3,649.0;
Original schedule: December 2009;
Current schedule: December 2009;
Acquisition issues and status: To date, ERAM has not breached any cost
and schedule parameters. However, it remains a high-risk program
because of the large amount of software that must be developed. The
ERAM contractor is experiencing software engineering difficulties as a
result of lower-than-expected productivity and software code growth.
Source: GAO analysis of FAA data.
[A] Terminal air traffic control facilities, known as Terminal Radar
Approach Control (TRACON) facilities, direct aircraft in the airspace
that extends from the point where the tower's control ends to about 50
nautical miles from the airport. A TRACON can be located at or outside
an airport.
[End of table]
Our Reviews of Seven Major Systems Show That Problems with Requirements
and Software Management Persist under AMS:
Our reviews of seven of FAA's costliest ATC system acquisitions found
that the problems FAA experienced with requirements and software
management and their related impact on cost, schedule, and performance
goals persist today under AMS.[Footnote 22]Figure 3 identifies these
seven acquisitions and their milestones, which are expressed in terms
of AMS decisions even when the acquisitions were initiated before AMS
was implemented. (See app. V for a description and the status of each
of these projects.) Specifically, for 6 of these 7 major ATC
acquisitions, FAA did not consistently (1) clearly define system
requirements at the investment decision point or (2) adequately assess
software complexity. Moreover, as FAA has acknowledged, it has never
managed its major acquisitions by focusing on how each would improve
the efficiency of ATC operations while maintaining or improving safety.
Although FAA has made progress in improving its acquisition of major
ATC systems--by, for example, improving the maturity of its processes
for acquiring software, using a "build a little, test a little"
approach to acquisitions as it did for Free Flight Phase 1,[Footnote
23] and restructuring its organization to minimize stovepipes--long-
standing problems persist in these areas. In addition, the two systems
we reviewed that were initiated after AMS's implementation are
currently operating within cost and schedule goals; however, they are
showing symptoms of past problems with developing requirements and
managing software complexity. Moreover, our work for more than two
decades--before and after AMS's implementation--has cited these types
of weaknesses as central reasons for the agency's long history of cost,
schedule, and performance shortfalls. This work has also found that the
effectiveness of an agency's acquisition management has had a greater
impact on the success of its major acquisitions than the contracting
process used (e.g., the FAR or AMS).
Figure 3: Key Milestones for Selected ATC Acquisitions Initiated before
and after AMS:
[See PDF for image]
[End of figure]
Inadequate Development or Definition of Requirements Led to
Requirements Growth or Unplanned Work for Five Acquisitions:
For five of the seven acquisitions we reviewed, AMS guidance did not
call for requirements that were specific enough to minimize
requirements growth or unplanned work. For four of these five
acquisitions--STARS, LAAS, NEXCOM, and ATOP--incomplete and poorly
defined requirements in the final requirements documents, used at the
investment decision point to assess an acquisition's readiness to enter
the development phase, led to requirements growth, unplanned
development work, or a reduction in system performance.[Footnote 24]For
the fifth acquisition--ASR-11--FAA misjudged the extent to which the
high-level requirements that were used to support the commercial-off-
the-shelf/nondevelopmental item (COTS/NDI) procurement by the
Department of Defense could result in a product capable of meeting
FAA's mission or user needs. As a result, unplanned software changes
were required.
* FAA's cost estimate for STARS has grown from its original estimate of
$0.94 billion in 1996 to $1.46 billion in 2004 and will deploy only 50
of the 172 STARS initially planned. Much of the cost growth has been
due to FAA requirements creep. As a result, the STARS program has
experienced delays of more than five years from its original plan, in
part due to added requirements to the commercial-off-the-shelf Initial
System Configuration (ISC). However, the STARS ISC was satisfactory for
use by the Department of Defense as deployed.
* A final requirements document was approved, and the development of
LAAS was scheduled to begin in 1999. However, poorly established
requirements resulted in the addition of 113 new requirements to the
initial specification, entailing significant software and hardware
changes. Furthermore, LAAS may not achieve its promised capabilities
because FAA has been unable to develop technologies necessary to warn
pilots of a disruption in the LAAS signal. Until this technology is
developed, LAAS cannot be operated safely. As a result, FAA recently
cut the fiscal year 2005 funding for LAAS, and the program will revert
to a research and development effort.
* FAA developed a final requirements document for the NEXCOM system,
but the requirements lacked the specificity needed to assess the
development risk. According to a NEXCOM contractor program official,
this led to miscommunication about the program requirement relating to
signal interference. This official stated that they misunderstood this
requirement and had not planned on the additional development work for
the NDI solution to meet such program objectives and delayed the
program 21 months. Another program requirement involved the NEXCOM
radios meeting or exceeding the operational coverage area of the
existing voice system. The existing radios had power output levels of
50 watts but the NEXCOM contractor could only achieve 34 watts of power
to meet the coverage requirement. A program official stated that the
contractor and FAA had not agreed on the testing procedures to assess
the power levels. This posed an "unacceptable consequence" and, as a
result, FAA performed additional testing or flight checks of the
reduced radio performance (50 watts versus 34 watts) and determined
that the performance reduction should not affect NEXCOM's mission or
its coverage requirement.
* FAA did not follow the AMS guidelines that call for completing a
final requirements document before proceeding to the development phase
for ATOP. The Joint Resources Council approved a delay in developing
the final requirements until after contract award. This decision
resulted in schedule delays and additional unplanned software
development. The ATOP program office asserted that the requirements
remained very stable and that the program is within cost and schedule
objectives established by the Council. However, FAA's internal
documents revealed that the requirements were not adequately defined.
For example, the ATOP Investment Analysis Study reported to the Joint
Resources Council prior to contract award that the lack of more
detailed ATOP requirements at this stage of acquisition added risk and
was of concern to the investment analysis team. Under AMS, this team is
responsible for, among other things, conducting risk analyses for the
various acquisitions. Furthermore, an ATOP Assessment Team conducted a
study in March 2003 and determined that at the ATOP contract award,
"requirements were written at a high level and not mutually understood
by FAA and the contractor." However, FAA management allowed the ATOP
program to proceed to solution implementation without the final
requirements document and, according to the contractor, this resulted
in schedule delays and growth in the amount of software needing
development.
* The high-level requirements for ASR-11, jointly generated by FAA and
the Department of Defense, to support a COTS/NDI acquisition, resulted
in a product that did not initially meet the FAA mission or user needs.
The software changes that were required to meet FAA's target detection
needs, as well as significant hardware design changes, parts
obsolescence, and production issues, added approximately two years to
system qualification and acceptance.
FAA Underestimated Software Complexity for Three Systems:
For three of the seven major ATC acquisitions we reviewed--ITWS, LAAS,
and ATOP--FAA's AMS guidance was not sufficient to address the risks
associated with modifying available software[Footnote 25] to fulfill
FAA's mission needs. In all three cases, FAA officials underestimated
the difficulty of modifying available software. Our work has shown that
underestimates are likely to result in unexpected software development,
higher costs, and schedule delays.
* ITWS experienced delays from the beginning because of the complexity
of its software development. Although the program appeared to be
progressing according to its baseline, immediately after the critical
design review in September 1998, the contractor revealed that it had
exceeded the target cost by $4 million. In addition, the contractor
claimed that the program did not recognize that the computer processor
originally planned for the program was becoming outdated, that the
manufacturer planned to discontinue its production because the market
was demanding a processor with greater processing and storage
capability, and that as a result, the original computer processor would
not be available to the program. Consequently, ITWS experienced cost
increases, schedule delays, and performance shortfalls. According to
the contractor and the original acquisition plan, all systems were
scheduled for delivery by December 2001, but that date has now
stretched to after 2009.
* LAAS's technology maturity was not adequately assessed, and further
development was needed. Specifically, the potential for radio
interference through the atmosphere was not understood and could limit
LAAS's operations. FAA has now placed all LAAS activities in research
and development. FAA did not adequately assess LAAS's software
development. At the time of the contract award, the contractor and FAA
estimated that 80 percent of the software that LAAS required had been
developed. FAA later determined that only 20 percent had been
developed. FAA and the contractor attribute this discrepancy to a lack
of communication on the steps necessary to satisfy the program's
requirements. FAA agrees that it should have conducted a software audit
and a software capabilities assessment, but pressures to keep LAAS on
schedule resulted in an inadequate assessment.
* The ATOP contractor underestimated by about half the extent to which
legacy nondevelopmental item software, which is the core of the ATOP
system, met the program's 1,036 requirements. As a result, a
significant amount of unanticipated new software code development and
other modifications were required.[Footnote 26]
ATC Systems Have Required Multiple Rebaselining Decisions to Address
Delays and Cost Growth:
As figure 3 illustrates, FAA initiated at least one rebaselining
decision for three of the five acquisitions that were begun before AMS
was implemented and were later transitioned to AMS. These rebaselining
decisions responded to delays and cost growth--problems that arise when
requirements are not stable, a program's design is not fixed, or
software code growth is not controlled. For example, FAA rebaselined
STARS two times--first in 1999 and again in 2002. Similarly, 2 years
after the investment decision for ITWS, FAA rebaselined the program
twice, in 1997 and again in 2001. Given the frequency of these past
rebaselining decisions for major ATC systems and the number of years
that elapsed before or between the rebaselining decisions (3 to 4
years), it is too soon to tell whether the two systems that were
initiated under AMS--ATOP and ERAM--will require similar rebaselinings
and ultimately meet their cost, schedule, and performance goals.
Although both programs are currently operating within their cost and
schedule goals and have not yet been rebaselined, FAA has had problems
with managing its major acquisitions in the past and is currently
having difficulties developing requirements and managing software
complexity. Furthermore, as we reported in May 2004, FAA's budget
increased from $9 billion in 1998 to $14 billion in 2004 but will be
constrained for the foreseeable future. In such a constrained budget
environment, cost growth and schedule problems can have serious
negative consequences for ongoing modernization efforts--postponed
benefits, costly interim systems, delays in funding other systems, or
reductions in the number of units purchased.
Internal and External Reviews Have Found That FAA Has Made Some
Progress but Continues to Experience Problems in Acquiring Major ATC
System under AMS:
Reviews of FAA's acquisition process, conducted by FAA, GAO, the DOTIG,
and others have shown that FAA has improved its management of major ATC
acquisitions in recent years but continues to experience cost overruns,
schedule slips, and performance shortfalls under AMS. Table 7
summarizes the results of 22 internal and external reviews of FAA's
major ATC acquisitions. According to these reviews, issued from 1997
through 2004, the same problems have persisted over many years, despite
various initiatives to address them, and FAA needs to strengthen its
management controls. For example, a key FAA review of eight major ATC
acquisitions, published in 1999, 3 years after AMS was implemented,
found that these acquisitions, though on track to meet their
performance goals, were not meeting their cost and schedule baselines.
FAA attributed these cost and schedule issues to new or poorly
understood requirements, underestimates of the acquisitions' technical
complexity, and funding shortfalls.
In addition, our reviews of major FAA acquisitions--initiated before
and after AMS was implemented--have found for more than two decades
that FAA's failure to meet schedule, cost, and performance baselines
for major ATC acquisitions has been due to shortfalls in planning, weak
management controls, and a lack of systematic processes for acquiring
new systems, including inadequate requirements management, cost-
accounting data, and estimates of technical difficulty. As we reported
in August 2004, judged against the criteria of GAO's framework for
information technology (IT) investment management, which measures the
maturity of an organization's investment management processes, FAA has
established about 80 percent of the basic selection and control
practices that it needs to manage its mission-critical investments for
the National Airspace System.[Footnote 27] For example, FAA's business
units actively monitor projects throughout their life cycles.[Footnote
28]However, the agency's senior IT investment board does not regularly
review investments that are in the "in-service management," or
operational phase, and this creates a weakness in FAA's ability to
oversee more than $1 billion of its IT investments. In addition, the
agency has not yet established the practices that would enable it to
effectively manage its annual IT budget of about $2.5 billion, and
agency executives lack assurance that they are selecting and managing
the mix of investments that best meets the agency's needs and
priorities. DOT has responded to our recommendations to FAA to
strengthen its IT investment management capability.
Moreover, other reviews, such as those by Booz-Allen & Hamilton and
MITRE, have identified other shortfalls, which reflect a lack of proper
management controls and planning. For example, in 1997, Booz-Allen &
Hamilton found, among other things, that FAA had not clearly defined
organizational roles and responsibilities within the various phases of
AMS and that greater guidance and training under AMS were warranted. In
1999, Booz-Allen & Hamilton reported that FAA had not demonstrated
improvement in adhering to planned costs and schedules under AMS and
that the agency needed to better manage its development of requirements
and address persistent funding shortfalls. Moreover, in 2001, a MITRE
report on selected major acquisitions found inadequate management
controls and deficiencies in both contractors' performance and in FAA's
measurement of acquisition performance. See table 7 for a chronological
listing of the reviews.
Table 6: Internal and External Reviews of FAA's Use of AMS for
Acquiring Major ATC Systems:
Review: GAO, Air Traffic Control: Improved Cost Information Needed to
Make Billion-Dollar Modernization Investment Decisions, GAO/AIMD-97-
20, (Washington, D.C.: Jan. 22, 1997);
Selected findings: FAA's cost- estimation practices do not satisfy
recognized estimating requisites, increasing the likelihood of poor
acquisition selection decisions;
Contributing factors: FAA's cost-accounting practices do not provide
for the proper accumulation of actual project costs.
Review: GAO, Air Traffic Control: Complete and Enforced Architecture
Needed for FAA Systems Modernization, GAO/AIMD-97-30, (Washington,
D.C.: Feb. 3, 1997);
Selected findings: Incompatibilities exist between current and planned
ATC acquisitions, resulting in high costs and reduced performance;
Contributing factors: FAA lacks a complete systems architecture or
overall "blueprint" to guide and constrain the development and
maintenance of ATC acquisitions.
Review: GAO, Air Traffic Control: Immature Software Acquisition
Processes Increase FAA System Acquisition Risks, GAO/AIMD-97-47,
(Washington, D.C.: Mar. 21, 1997);
Selected findings: Planned acquisitions frequently are not delivered on
time and within budget;
Contributing factors: Weaknesses in some key process areas, such as
planning, requirements development, and management, limit FAA's ability
to consistently acquire software-intensive ATC systems on time and
within budget.
Review: FAA, Evaluation of FAA Acquisition Reform--The First Year:
April 1996 - March 1997, (Washington, D.C.: May 1997);
Selected findings: AMS addresses 15 of the 17 problems facing
acquisitions;
Contributing factors: Inadequate management has not enabled FAA to meet
its goals of reducing acquisition deployment time by 50 percent and
cost by 20 percent.
Review: FAA, Evaluation of FAA Acquisition Reform--The First Two Years:
April 1996 - March 1998, Report #1998-02, (Washington, D.C.: May 29,
1998);
Selected findings: Further improvements are necessary if acquisition
reform is going to allow FAA to meet its cost and schedule goals;
Contributing factors: Procedural weaknesses limit FAA's ability to
achieve cost and schedule goals.
Review: GAO, Air Traffic Control: Observations on FAA's Air Traffic
Control Modernization Program, GAO/T-RCED/AIMD-99-137, (Washington,
D.C.: Mar. 25, 1999);
Selected findings: From the inception of its modernization efforts, FAA
has not consistently followed a disciplined management approach for new
acquisitions;
Contributing factors: Weaknesses persist in key areas, such as how FAA
monitors the status of its acquisitions throughout their life cycles.
Review: GAO, Air Traffic Control: FAA's Modernization Investment
Management Approach Could Be Strengthened, GAO/RCED/AIMD-99-88,
(Washington, D.C.: Apr. 30, 1999);
Selected findings: AMS contained weaknesses in the selection of
acquisitions and in the review of acquisitions' performance during the
postimplementation phase;
Contributing factors: FAA lacked adequate cost data for making
selection decisions; adequate management controls, and a defined,
documented process for conducting reviews during the in-service
management phase.
Review: FAA, Evaluation of FAA Acquisition Reform--The First Three
Years: April 1996 - March 1999, Report #1999-04,; (Washington, D.C.:
May 28, 1999);
Selected findings: FAA's cost and schedule plans were not on track, but
performance plans were met;
Contributing factors: Requirements changed or were misunderstood;
technical difficulties were underestimated; and funding fell short.
Review: Booz-Allen & Hamilton, Independent Assessment of the Federal
Aviation Administration's Acquisition Management System, (McLean, VA:
July 6, 1999);
Selected findings: FAA has yet to implement a seamless life-cycle
approach to acquisitions management;
Contributing factors: AMS is not being consistently implemented across
all life-cycle phases.
Review: GAO, National Airspace System: Persistent Problems in FAA's New
Navigation System Highlight Need for Periodic Reevaluation, GAO/RCED/
AIMD-00-130, (Washington, D.C.: June 12, 2000);
Selected findings: FAA experienced delays and cost increases in
developing its global positioning navigation system; as a result, it is
unclear whether the benefits of the system will outweigh the cost;
Contributing factors: FAA lacks a comprehensive plan with checkpoints
for reviewing the contractor's approach to meeting the system's
performance requirements.
Review: GAO, National Airspace System: Problems Plaguing the Wide Area
Augmentation System and FAA's Actions to Address Them, GAO/T-RCED-00-
229, (Washington, D.C.: June 29, 2000);
Selected findings: FAA experienced cost and schedule problems in
developing this navigational system because of unplanned software
development needs and a requirement to warn pilots of any system
failure that would provide misleading information;
Contributing factors: FAA underestimated the complexity of developing
the acquisition.
Review: GAO, National Airspace System: Free Flight Tools Show Promise,
but Implementation Challenges Remain, GAO-01-932, (Washington, D.C.:
Aug. 31, 2001);
Selected findings: Three acquisitions that are components of FAA's
planned new approach for air traffic management have uncertain
potential benefits and may not be worth FAA's investment;
Contributing factors: FAA needs better data collection and analysis
processes to ensure that benefits are realized.
Review: GAO, National Airspace System: Better Cost Data Could Improve
FAA's Management of the Standard Terminal Automation Replacement
System, GAO-03-343, (Washington, D.C.: Jan. 31, 2003);
Selected findings: The reliability of the life-cycle cost estimate for
STARS is uncertain because cost data obtained from the contractor do
not reflect the current status of the contract;
Contributing factors: The development cost estimate is based on the
contractor's projections, which FAA has not yet independently analyzed,
as called for under AMS.
Review: GAO, National Airspace System: Current Efforts and Proposed
Changes to Improve Performance of FAA's Air Traffic Control System,
GAO-03-542, (Washington, D.C.: May 30, 2003);
Selected findings: FAA was unable to hire a chief operating officer to
head the ATO;
Contributing factors: Uncertainties about the position's
responsibilities, reporting relationships, and performance measurement
criteria hampered the hiring.
Review: DOT/OIG, Status of FAA's Major Acquisitions, AV-2003-045,
(Washington, D.C.: June 26, 2003);
Selected findings: Cost growth, schedule delays, and performance
problems continue with FAA's major acquisitions;
Contributing factors: Cost and schedule baselines are not reliable, and
decisions are being made with unclear data.
Review: GAO, Air Traffic Control: FAA's Modernization Efforts--Past,
Present, and Future, GAO-04-227T, (Washington, D.C.: Oct. 30, 2003);
Selected findings: Systemic management issues, including inadequate
management controls and human capital issues, have contributed to major
ATC acquisitions' persistent cost overruns, schedule delays, and
performance shortfalls;
Contributing factors: FAA lacked the information technology and
financial management systems that would have helped it reliably
determine the acquisitions' technical requirements and estimate and
control their costs and schedules; and the agency's organizational
culture discouraged collaboration among technical experts and users.
Review: GAO, Information Technology: FAA Has Many Investment Management
Capabilities in Place, but More Oversight of Operational Systems Is
Needed, GAO-04-822, (Washington, D.C.: Aug. 20, 2004);
Selected findings: Although weaknesses remain, FAA has established
about 80 percent of the basic practices needed to manage its mission-
critical acquisitions so that it can be assured that it is selecting
and managing the mix of investments that best meets its needs and
priorities;
Contributing factors: Remaining weaknesses include inadequate
management controls and the lack of a defined, documented process for
conducting reviews during the in-service management phase.
Review: GAO, Air Traffic Control: System Management Capabilities
Improved, but More Can Be Done to Institutionalize Improvements, GAO-
04-901, (Washington, D.C.: Aug. 20, 2004);
Selected findings: FAA made progress in improving its system management
capabilities, but can do more to institutionalize process improvement
initiatives;
Contributing factors: Process improvement efforts have not been
institutionalized.
Source: GAO analysis.
[End of table]
FAA's ATO Is Taking Steps to Improve Major ATC Acquisitions:
FAA's recent reorganization, which brought ATC acquisitions and
operations together in the ATO,[Footnote 29] is expected to help the
agency address many of the concerns we have identified for more than
two decades, including those identified in this report. For example,
the ATO is continuing to develop and refine specific guidance for
critical areas, such as requirements management, software development,
and cost estimation. In addition, as the overseer of both ATC
acquisitions and operations, the ATO is in a position to facilitate
more effective management of major ATC acquisitions than has occurred
in the past. The ATO is attempting, for example, to link acquisition
decisions directly with expected improvements in operational efficiency
without compromising safety. This is important, given that FAA has
spent about $2.5 billion on ATC modernization per year since 1996 while
operating costs have continued to rise--from $4.6 billion to $7.5
billion over the past decade. FAA had not completed its reorganization
or implemented all of its initiatives at the time of our audit.
Improvements to Requirements Development:
With the establishment of the ATO, FAA consolidated requirements
development from two organizations (the organization sponsoring an
acquisition and the former agencywide acquisition organization) into a
single new organization--the Air Traffic System Requirements
Service.[Footnote 30] In addition, the ATO developed guidance to better
manage requirements during the middle phase of AMS (solution
implementation). According to FAA officials, some more complex
development efforts may need to develop systems requirements and a more
detailed requirements document than AMS currently calls for in the
final requirements document. More important, in January 2003, FAA
issued guidance on requirements management, Roles in Requirements
Management During Solution Implementation Phase, which provides for
integrated requirements teams that maintain responsibility for
requirements management throughout an acquisition's life cycle.
According to this guidance, when the final requirements document is
accepted by the Joint Resources Council at the investment decision
point, a requirements baseline is established and any proposed changes
to the requirements must be assessed for their impact on the program
and shown to be operationally suitable, affordable, executable, and
justifiable. An FAA official on an integrated requirements team stated
that any changes that may affect an acquisition's cost and the schedule
require approval by the Executive Committee. The FAA official also
stated that this guidance has already helped to stabilize NEXCOM's
requirements during the solution implementation phase. Other FAA
officials representing the Joint Resources Council acknowledged that
the guidance should ensure greater control over program requirements
growth, but said that not all program offices have consistently applied
it.
Improvements to Managing Software and System Acquisition and
Development:
To better manage software programs for ATC modernization acquisitions,
FAA established a centralized process improvement office that reports
to the Chief Information Officer (CIO).[Footnote 31] This office
developed an FAA integrated capability maturity model (i-CMM), a
software development and management model that is similar to a model
developed by Carnegie Mellon University called the Capability Maturity
Model Integration (CMMI®), which is used to appraise the maturity of an
organization's processes for acquiring software. However, FAA's i-CMM
goes beyond Carnegie Mellon's model to reflect international standards.
The CMMI® appraisal methodology calls for assessing process areas--such
as project planning, requirements management, and quality assurance--by
determining whether key practices are implemented and overarching goals
are satisfied. Both the i-CMM model and CMMI® appraisal methodologies
provide a logical framework for measuring and improving key processes
needed for achieving quality software and systems.
However, as we reported in August 2004,[Footnote 32] FAA projects are
not required to use the capability maturity model for process
improvement, and individual projects that use the i-CMM model are
allowed to choose which process areas they seek to improve and to
determine when they are ready for an appraisal of their progress. To
date, fewer than half of FAA's major ATC projects have used this model.
The recurring weaknesses we identified in our project-specific
evaluations are due in part to the flexibility these projects were
given in deciding whether and how to adopt this process improvement
initiative. Furthermore, after combining its ATC organizations into a
single performance-based organization (the ATO), FAA is reconsidering
prior policies, and it is not yet clear whether process improvement
will remain a priority. Without a strong senior-level commitment to
process improvement and a consistent, institutionalized approach to
implementing and evaluating it, FAA cannot ensure that key projects
will continue to improve systems acquisition and development
capabilities. As a result, FAA will continue to risk the project
management problems--including cost overruns, schedule delays, and
performance shortfalls--that have plagued past acquisitions. To address
these shortcomings, we recommended that the Secretary of Transportation
address specific weaknesses and institutionalize FAA's process
improvement initiatives by establishing a policy and plans for
implementing and overseeing process improvement initiatives.
Improvements to Estimating Costs:
FAA has taken steps to improve its cost estimation for major ATC
projects by issuing guidance on how to develop and use pricing under
AMS. For example, AMS policy calls for audit trails to record and
explain the values that are used as inputs to cost models. In addition,
it calls for agency officials, when reporting to executive oversight
agencies and Congress, to disclose the level of uncertainty and
imprecision that are inherent in cost estimates for major ATC systems.
According to AMS policy, estimators record the procedures, ground rules
and assumptions, data, environment, and events that underlie their
development or update of a cost estimate. This information supports the
credibility of the cost estimate, aids in the analysis of changes in
program costs, enables reviewers to assess the cost estimate
effectively, and contributes to the population of FAA databases that
can be used for estimating the cost of future programs. Finally,
despite a delay of many years, FAA officials told us that they are in
the final stages of completing the agency's cost-accounting system and
plan to have it in place across the agency by the end of this calendar
year, which will bring FAA into compliance with the Federal Managers'
Financial Integrity Act of 1982. This measure will help reduce the
likelihood of cost overruns or improper payments for unallowable costs
and provide decision-makers with critical information. As we have
reported in the past,[Footnote 33] a cost-accounting system is critical
to managing major ATC acquisitions, because without it, FAA lacks the
information it needs to reliably estimate operating costs over an
acquisition's life cycle.
Other Improvement Efforts:
In May 2004, the FAA Administrator testified to Congress that, to date,
in attempting to improve the efficiency of ATC operations while
maintaining safety, FAA had not managed its major ATC acquisitions to
be aware of their cost implications for its operations. The
Administrator said, however, that the agency was taking its first steps
to fundamentally change how it makes acquisition decisions by adopting
a more results-oriented approach. Under this approach, the agency plans
to link its decisions to fund major acquisitions directly with their
expected contribution to improving operational efficiency and
controlling escalating operating costs. Whereas, in the past, FAA
measured results in terms of its progress in completing and deploying a
major ATC system, it was now going to focus on how a given system
improved operational efficiency. Such an approach holds promise for
helping FAA more effectively manage its portfolio of major ATC
acquisitions by providing a sound basis for choosing among competing
priorities. However, because FAA has only recently begun to incorporate
this type of analysis of acquisitions' costs and operational efficiency
into its decision-making and management processes, it is still too
early to assess the results.
In addition, to its credit, FAA has created a training framework for
its acquisition workforce, which we found mirrors human capital best
practices that we have identified. In January 2003, we reported on
FAA's efforts to define and train its workforce to meet the
requirements of the Clinger-Cohen Act of 1996.[Footnote 34] This act
required FAA and other civilian agencies to establish education,
training, and experience requirements for their acquisition workforce.
Our work on public and private best practices has identified six
elements of training as critical to acquisition. These elements include
(1) prioritizing the acquisition initiatives most important to the
agency, (2) securing top-level commitment and resources, (3)
identifying those who need training on specific initiatives, (4)
tailoring training to meet the needs of the workforce, (5) tracking
training to ensure it reaches the right people, and (6) measuring the
effectiveness of training. These six elements are crucial for
successfully implementing acquisition initiatives and reforms.
Agencies that do not focus their attention on these critical elements
risk having an acquisition workforce that is ill equipped to implement
new processes. The probability of success is higher if training is
well planned rather than left to chance. In 2003, we found that FAA's
model for training its acquisition workforce largely mirrored public
and private-sector best practices and that the agency had highly
developed processes for four of these six elements. See figure 4.
Figure 4: Our Analysis of FAA's Progress as of 2003 in Implementing Key
Elements of Training for Its Acquisition Workforce:
[See PDF for image]
[End of figure]
Since 2003, FAA has taken some steps to measure the effectiveness of
its training. For example, the agency collects and reviews
participants' assessments of the knowledge they have gained, the extent
that learning objectives were achieved and the applicability and
usefulness of the training. In addition, members of FAA's Intellectual
Capital Investment Plan Council[Footnote 35] have attempted to make
qualitative judgments about the impact of the training on the
effectiveness or efficiency of their organizations. However, FAA is
still developing an evaluation program with metrics to measure the
extent to which organizational goals are achieved when individual
training objectives are met. Industry and government experts believe
training and human capital investments are prerequisites for
successfully introducing and implementing effective acquisition best
practices. FAA's acquisition workforce plays a critical role in
addressing long-standing weaknesses that we and others have identified
with FAA's acquisition of major ATC systems. Given the importance of
training for acquisition workforces, it will be important for the ATO
to put mechanisms in place to comprehensively evaluate the
effectiveness of the training it provides to improve the knowledge base
of FAA's acquisition workforce.
To improve its investment management decision-making and oversight of
major ATC acquisitions, the ATO also initiated the following
procedures:
* Integrate AMS and the Office of Management and Budget's Capital
Planning and Investment Control Process[Footnote 36]to develop a
process for analyzing, tracking, and evaluating the risks and results
of all major capital investments made by FAA.
* Conduct Executive Council reviews of project breaches of 5 percent in
cost, schedule, and performance to better manage cost growth;
* Issue monthly variance reports to upper management to keep them
apprised of cost and schedule trends.
* Monitor progress in meeting the goals identified in FAA's Flight
Plan, the agency's blueprint for action through 2008. The Executive
Council tracks this progress monthly and reports to the Administrator,
using a color-coded system to keep her apprised of how well FAA is
meeting its goals. Green denotes that a goal will be met, yellow
denotes that some of the activities leading to a main goal may be in
jeopardy but the overall goal can be achieved, and red denotes serious
concerns about reaching a goal without major intervention. A formal
progress report is issued quarterly and made publicly available on the
agency's Web site; and:
* Increase the use of cost monitoring or earned value management
systems[Footnote 37] to improve oversight of programs.
Despite FAA's current and planned efforts to improve its acquisition of
major ATC systems under the ATO, given the newness of these efforts and
the agency's poor track record in this area for more than two decades,
it is critical for FAA to (1) modify AMS to more fully reflect the best
practices followed by high-performing acquisition organizations, (2)
follow through on planned improvement initiatives, and (3) adopt a
continuous improvement approach to acquiring new ATC systems.
Conclusions:
In the early 1990s, FAA contended that it needed relief from the FAR to
remedy long-standing problems with cost, schedule, and performance
shortfalls in its major ATC acquisitions; however, our work for more
than two decades in this area has found that acquiring major ATC
systems successfully depends more on managing an acquisition process
well than on using a specific contracting process (e.g., the FAR or
AMS). While our recent work has shown some improvement in FAA's
management of major ATC system acquisitions, some key problems that
existed before 1996 persist under AMS--including difficulty with
clearly defining system requirements at the investment milestone and
adequately assessing complex software requirements. These problems
continue to make these acquisitions vulnerable to cost, schedule, and
performance shortfalls. Without further measures to improve the
development and management of requirements and to better estimate the
complexity of the software development needed for major ATC systems,
such shortfalls are likely to persist.
Although AMS provides some discipline for acquiring major ATC systems
through its various phases, activities, and decision points, it does
not require that (1) specific knowledge be attained using explicit
written criteria and (2) corporate executive-level oversight be
provided to determine--independently from the program offices--whether
a system has reached a level of development (product maturity)
sufficient to move forward in the acquisition process. Commercial best
practices call for such knowledge-based decision-making at the
corporate executive-level to help ensure that acquisitions are not
moved into the development phase prematurely, to obtain greater
predictability in ATC system program costs and schedules, to improve
the quality of the ATC systems that are deployed, and to deliver new
capability to the National Airspace System faster. A knowledge-based
approach is also important because it provides assurance that agency
decision-makers have critical information about an acquisition's
ability to meet a mission need and FAA's readiness to move forward in
the acquisition process before making large commitments of agency
resources. Absent such an approach, FAA lacks assurance that it has
obtained the critical technological, design, or manufacturing knowledge
that best practices call for to avoid cost overruns, schedule slips,
and performance shortfalls. As a result, FAA is not doing all that it
can to systematically address persistent shortcomings in its management
of major ATC acquisitions. Moreover, although FAA has established a
framework for training its acquisition workforce under the ATO, it has
not yet developed comprehensive performance criteria to evaluate how
effectively it has implemented this framework. As a result, the agency
lacks assurance that its use of this framework is having the intended
effect of improving the knowledge base of this workforce.
Recommendations for Executive Action:
We are making five recommendations to the Secretary of Transportation.
To reduce the risk of persistent cost and schedule shortfalls in major
ATC system acquisition programs, to improve the quality of the ATC
systems that are deployed, and to deliver new capability to the
National Airspace System faster, we recommend that the Secretary of
Transportation advise the FAA Administrator to take the four following
actions:
* Modify AMS to specify that requirements be more clearly defined for
major ATC systems, including providing more detailed guidance on
setting clear, objective, and measurable requirements that reflect
customers' needs, before making large investments of agency resources.
* Establish a strategy for identifying and measuring all additional
development needed for complex software (e.g., commercial-off-the-
shelf or nondevelopmental items) used for major ATC systems.
* Develop explicit written criteria for the key decision points called
for under best practices, including the capture of specific design and
manufacturing knowledge.
* Require corporate executive-level decisions at these key decision
points (before an acquisition moves from integration to demonstration
and, again, before it moves to production).
In addition, to assure FAA that the training framework it has adopted
for the ATO's acquisition workforce is improving the knowledge base of
this workforce as intended, we recommend that the Secretary advise the
Administrator to develop performance criteria to comprehensively
evaluate the framework's effectiveness.
Agency Comments:
We provided copies of a draft of this report to DOT for review and
comment and met with Department and FAA officials, including the ATO's
Vice President for Acquisition and Business Services, to obtain their
comments. FAA officials told us that they have made great strides in
improving their acquisition of major ATC systems under AMS; however,
they recognize that there is room for improvement and are firmly
committed to implementing best practices for acquisitions. These
officials generally agreed with the report's findings and conclusions
and said that our recommendations would be useful to them as they
continue to refine their acquisition management system, including
training their acquisition workforce. The agency provided us with oral
comments, primarily technical clarifications, which we have
incorporated as appropriate.
As agreed with your office, unless you publicly announce the contents
of this report earlier, we plan no further distribution until 30 days
from the report date. At that time, we will send copies of this report
to interested congressional committees, the Secretary of
Transportation, and the Administrator, FAA. We will also make copies
available to others upon request. In addition, the report will be
available at no charge on the GAO Web site at
[Hyperlink, http://www.gao.gov].
Please call me at (202) 512-2834 if you or your staff have any
questions concerning this report. Key contributors to this report are
listed in appendix VI.
Signed by:
JayEtta Z. Hecker:
Director, Physical Infrastructure Team:
[End of section]
Appendixes:
Appendix I: FAA Has Begun Analyzing Spending Trends to Take a More
Strategic Approach to Procurement:
Our review of the Federal Aviation Administration's (FAA) general
procurement of goods and services focused on the Air Traffic
Organization (ATO) and its predecessor offices. According to FAA
officials, the ATO has recently begun to consider ways to better
leverage its buying power by taking a more strategic approach to
procurement. While FAA uses the Acquisition Management System (AMS) for
all FAA acquisitions, including the procurement of such goods and
services as office supplies, computers, telephone services, and
engineering and technical support services, these procurement
activities take place in a decentralized environment of independent,
transaction-oriented buying processes. Each FAA unit determines its
need for goods and services and procures them as necessary, leaving
headquarters with limited oversight of the agency's total procurement
spending. For example, in 2003, FAA units carried out over 346,000
procurement actions for goods and services and purchase
cardholders[Footnote 38] made an additional 335,000 transactions. This
fragmented environment does not permit the agency to leverage its
buying power through lower-cost, consolidated contracts, at the local,
regional, or national level and to rationalize the number of suppliers
best suited to meet the agency's needs. At the same time, as part of a
strategic procurement effort, FAA can use spend analysis to monitor
trends in small and disadvantaged business participation so that it can
balance the goals of lower-cost contract consolidation and promoting
small business contracting opportunities.
Spend analysis, a tool used in a strategic approach to procurement,
provides knowledge about how much is being spent for what goods and
services, who the buyers are, who the suppliers are, and where the
opportunities are to leverage buying power. Our past work[Footnote 39]
shows that private companies are using spend analysis as a foundation
for employing a strategic approach to procurement. The analysis
identifies where numerous suppliers are providing similar goods and
services--often at varying prices--and where purchasing costs can be
reduced and performance improved by better leveraging buying power and
reducing the number of suppliers to meet the company's needs. Our
research on commercial best practices has found that spend analysis is
an important driver of strategic planning and execution. As part of an
overall strategic procurement effort, companies use spend analysis to
(1) define the magnitude and the characteristics of their spending, (2)
understand their internal clients and supply chain, (3) create lower-
cost consolidated contracts, and (4) monitor spending with small and
disadvantaged businesses to achieve socioeconomic procurement goals.
We previously reported[Footnote 40]that six agencies, including DOT,
did not take advantage of opportunities to obtain more favorable prices
on purchase card buys with frequently used vendors--vendors where an
agency spends more than $1 million annually. In these six agencies,
which accounted for over 85 percent of federal government purchase card
spending, frequently used vendors accounted for purchases totaling
nearly $3 billion in 2002. We recommended several actions--including
conducting spend analysis using available data and gathering additional
information where feasible--that could ultimately help these agencies
achieve $300 million annually in potential savings.
In fiscal year 2003, FAA procured nearly $4 billion in goods and
services and spent an additional $132 million using purchase cards.
According to senior FAA officials, the agency has just begun to
implement a strategic approach to general procurements. Other federal
agencies are beginning to use strategic tools such as spend analysis to
improve their spending for goods and services, and some have
initiatives under way to obtain more favorable prices on purchase card
buys. According to a senior FAA acquisition official, FAA has to
balance the need of its units to independently make purchases that
pertain solely to unit requirements with the agency's need to aggregate
purchases of goods and services that are used by more than one unit.
FAA has hired a consultant to help begin the use of spend analysis.
This effort could reduce the agencywide costs for mobile wireless
services by 40 percent--an effort expected to save the agency $8 to $10
million annually. FAA intends to expand its use of spend analysis to
target other procurement category savings opportunities, including
information technologies, training, facilities, and professional
services, as its accounting systems improve.
FAA has taken some preliminary steps to set up a spend analysis
program; however, progress has been challenging for FAA because of
deficiencies in its accounting systems. For example, because the
agency's accounting system did not identify all of the mobile wireless
services for which it was being billed, the contractor implementing the
spend analysis had to obtain this information from the wireless
providers. FAA will need to expedite its efforts in this area to fully
realize potential savings. Our prior research has shown that setting up
a spend analysis program can be challenging. Companies have had
problems accumulating sufficient data from internal financial systems
that do not capture information on all of what a company buys or is
using in different, unconnected parts of the company. Despite these
challenges, companies that have developed formal, centralized spend
analysis programs have been able to track their costs and identify
areas for strategic sourcing and savings opportunities.
In our recent report on spend analysis,[Footnote 41] we found that DOT,
at the time of our review, had not yet begun to collect the data needed
for a strategic approach to procurement; however, the department is
engaged in ongoing efforts to improve procurements, and its top
leadership is committed to using spend analysis to change the way goods
and services are purchased. One obstacle to using spend analysis that
the department cited during our review was a lack of comprehensive and
reliable spending data. However, since we completed our review, the
department reports stepping up efforts to use currently available data
and evaluate business intelligence software to overcome those
obstacles. In commenting on our report, Transportation's senior
procurement executive told us that the department is expanding its
spend analysis efforts. For example, his office recently reviewed
purchase card spending data to identify volume discount opportunities
and is now using the results to negotiate new discount agreements with
several office product vendors. In addition, he told us that to
facilitate future agencywide purchase card spend analyses, DOT awarded
a task order in June 2004 to one bank card company that will provide
purchase-card audit software and enhanced data-mining capabilities. He
also indicated that the department's leadership supports fiscal year
2005 funding to enhance spend analysis capabilities and that software
options for the new agencywide spend analysis system are now being
evaluated as part of an ongoing financial and procurement review.
[End of section]
Appendix II: Objectives, Scope, and Methodology:
To compare FAA's Acquisition Management System (AMS) with the Federal
Acquisition Regulation (FAR), we reviewed AMS and changes in it over
time. We also compared FAA's acquisition authority under the FAR and
under AMS. In addition, we identified relevant recommendations from
reports that we, the Department of Transportation's Inspector General
(DOTIG), and others have issued to determine which recommendations have
been implemented, rejected, or left open, and to evaluate how those
recommendations have modified FAA's acquisition policies and practices.
We also collected and summarized published reports and analyzed
available life-cycle management data on the current status of major and
nonmajor acquisitions being carried out under AMS.
To determine the ways in which FAA's acquisition policies compare with
our best practices model, we used information from several of our
products that examine how commercial best practices can improve
outcomes for acquisition programs. This model consists of four phases:
(1) concept and technology development; (2) product development, which
includes both integration and demonstration activities; (3) production;
and (4) operations and support. In between these four phases are three
key knowledge points at which commercial firms must have sufficient
knowledge to make large investment decisions. We also reviewed and
analyzed AMS, accessible at h [Hyperlink, http://fast.faa.gov] ttp: //
fast.faa.gov. Furthermore, to clarify the content of FAA's acquisition
process, we met with various FAA vice-presidents and officials from
FAA's Acquisition Planning and Policy Division. Next, we compared and
contrasted FAA's acquisition policies with the best practices for
commercial acquisitions identified in our past reports. Our analysis
focused on whether FAA's policies contained the measurable criteria and
management controls necessary to achieve FAA's intent of minimizing
cost, schedule, and performance risks. We also interviewed current and
former FAA procurement officials that have experience using both the
FAR and AMS.
To determine if FAA has effectively implemented its new acquisition
authority and improved its acquisition outcomes, we reviewed seven of
FAA's most expensive major ATC acquisitions, including the Airport
Surveillance Radar 11 (ASR-11), Standard Terminal Automation
Replacement System (STARS), Integrated Terminal Weather System (ITWS),
Local Area Augmentation System (LAAS), Next Generation Air/Ground
Communications System (NEXCOM), Advanced Technologies and Oceanic
Procedures (ATOP), and En Route Automation Modernization (ERAM). See
table 7 for specific program costs.
Table 7: Program Costs for the Seven Systems We Reviewed:
Dollars in millions.
Program: STARS;
Total program cost as of 9/30/04: $1,460.0.
Program: ASR-11;
Total program cost as of 9/30/04: $891.7.
Program: ITWS;
Total program cost as of 9/30/04: $288.3.
Program: LAAS;
Total program cost as of 9/30/04: $696.1.
Program: NEXCOM;
Total program cost as of 9/30/04: $318.4.
Program: ATOP;
Total program cost as of 9/30/04: $548.2.
Program: ERAM;
Total program cost as of 9/30/04: $2,154.6.
Total;
Total program cost as of 9/30/04: $6,357.3.
Source: GAO analysis.
Note: These amounts are for facilities and equipment only (not
operations and maintenance).
[End of table]
We also selected these seven acquisitions because we considered them to
fall into two basic categories-pre-AMS and post-AMS. Five of the
acquisitions were initiated before AMS was implemented in April 1996
and were transitioned into AMS at various times before their
completion. The two remaining acquisitions--ATOP and ERAM--were
initiated and have remained completely under AMS. We then reviewed
program documents and reports and interviewed program and agency
officials responsible for developing these acquisitions, as well as
other acquisitions experts in the private sector. For some
acquisitions, we discussed programmatic issues with representatives of
the primary contractor for the specific acquisition to obtain
information on the practices and procedures used for the acquisition.
In addition, we interviewed some current and former FAA procurement
officials with experience using both the FAR and AMS to obtain their
views on the use of each contracting process and how the two compare.
Furthermore, to see how FAA has progressed in addressing problems with
its acquisitions, we reviewed our work on acquisitions over the last 20
years, as well as reports by the DOTIG, FAA, Booz-Allen & Hamilton, and
MITRE. Because the data in this report on cost, schedule and
performance are used as background information or to otherwise provide
a description of acquisitions, we did not assess their reliability.
The effect of the current budget process on FAA's ability to
successfully modernize the National Airspace System, including
acquiring major ATC systems is not within the scope of this review.
[End of section]
Appendix III: Comparison of the Scope and Flexibility of FAA's
Acquisition Management System and the Federal Acquisition Regulation
Process:
Background:
FAA's business processes, including its acquisition of major systems,
differ significantly from the business processes followed by most other
federal agencies. FAA relies on its Acquisition Management System
(AMS), which establishes FAA internal acquisition policy. AMS resulted
from the adoption of language in the Department of Transportation and
Related Agencies Appropriations Act,[Footnote 42] which directed the
FAA Administrator to develop and implement an acquisition management
system for FAA. The adoption of this language (section 348) followed
FAA's assertions that the requirement that it conduct procurements in
accordance with the Federal Acquisition Regulation (FAR) was at least a
contributing factor in its repeated failure to complete air traffic
control (ATC) and other modernization programs on schedule. The
Administrator was directed to put in place a system that would address
the "unique needs of the agency" that FAA contended prevented its
acquisitions from being timely and cost-effective.
Section 348 distinguished FAA from other federal agencies by removing
FAA from the federal acquisition system. Under section 348, FAA was no
longer subject to title III of the Federal Property and Administrative
Services Act of 1949,[Footnote 43] which among other things requires
that the government procure supplies and services competitively. It
removed FAA as an agency subject to the Office of Federal Procurement
Policy Act[Footnote 44] and eliminated the requirement that FAA comply
with the FAR. While mandating that FAA conduct its acquisitions so that
"all reasonable opportunities to be awarded contracts shall be provided
to small business concerns and small business concerns owned and
controlled by socially and economically disadvantaged individuals,"
section 348 eliminated the requirement that FAA comply with the Small
Business Act.[Footnote 45] Furthermore, it made the procurement protest
system of the U.S. Government Accountability Office inapplicable to
FAA, although disappointed offerors can still file protests with FAA's
Office of Dispute Resolution for Acquisition.[Footnote 46]
AMS Defines an Investment/Life-Cycle Project Management System:
Much of AMS guidance concerns project, financial, and property life-
cycle management issues. In fact, FAA's policy describes AMS as
applying to all investment programs regardless of cost or the
appropriation funding them. It recognizes that a single investment
program may span multiple procurements and projects. It applies,
according to its terms, to the activities associated with needs
analysis, determination of requirements, analysis of investment
alternatives, establishment of investment programs, allocation and
expenditure of resources, procurement and deployment of needed products
and services, in-service management of fielded capability, and eventual
disposal of obsolete products.
AMS focuses on the following key program milestones:
* Mission Analysis--encompasses those key corporate and service-level
processes that define, coordinate, and integrate the work of service
organizations,[Footnote 47] thereby providing strategic direction to
keep FAA responsive to the service needs of its customers. Mission
analysis is used to update a mission need statement, which in turn may
identify capability shortfalls or technological opportunities, that is,
unmet needs. Unmet needs are presented to the Joint Resources Council
(JRC) for a mission need decision. To be approved, the unmet need
should be supported by the updated mission need statement and the
initial requirements document, including a concept of use, and the
initial investment plan.
* Investment Analysis--builds on the results of the mission need
decision by developing detailed plans and final requirements for each
proposed investment program and by defining an acquisition program
baseline that establishes cost, schedule, performance, benefit, and
risk-management boundaries for the program. AMS calls for planning the
entire solution--an effort that may use market survey data but is based
in large measure on FAA's assumptions and data. The service
organization produces a final implementation and life-cycle support
strategy. A detailed program plan and an acquisition program baseline
are also produced. The results are presented to the JRC for a "final
investment decision."[Footnote 48]
* Solution Implementation--encompasses acquiring, accepting,
deploying, installing and preparing for the operational use of an
approved investment. Approval of the investment carries with it
authorization for the service organization to conduct all acquisitions
needed to execute the investment decision, subject to any constraints
established in the final investment decision.
* In-Service Decision--is an FAA system qualification milestone, which
is achieved when an otherwise operational investment is satisfactorily
tested to demonstrate its operational effectiveness and suitability
before it is placed in service in the National Airspace System. The JRC
designates the decision maker.
* In-Service Management--covers activities throughout a system's life
cycle, starting at the time that an investment becomes operational. In-
service product improvements may eliminate latent defects, fix systemic
problems, and enhance the utility of the investment. These changes may
be made within the approved acquisition program baseline without
corporate-level approval. In-service management also includes
planning, programming, and developing supporting budget input;
monitoring and assessing performance, cost of ownership, and support
trends; and planning for service-life investment decisions.
* Service Life Extension--seeks a new investment decision by the JRC
when a current capability is unable to satisfy demand or when another
solution may be more effective. The JRC can decide to revalidate the
mission need satisfied by the solution by upgrading or refurbishing
fielded capability or by replacing that capability with another
equivalent or new superior solution. The JRC may also decide that the
capability should be retired.
Only a Portion of AMS Deals Directly with the Procurement Process:
Although the FAR includes requirements addressing procurement planning
and major system acquisition, AMS as just outlined differs
significantly from the FAR in its focus and scope. The FAR addresses
planning[Footnote 49] and major system acquisition[Footnote 50] in the
context of government procurement policy and procedure. Agencies other
than FAA find the broader program planning and management issues
addressed in AMS outside of the FAR, in documents such as the Office of
Management and Budget's (OMB) Circular A-109, in their own planning
guidance, such as the Department of Defense's (DOD) 5000
series,[Footnote 51]and in established knowledge-based best practices.
As indicated earlier, much of AMS focuses on just such issues. Only AMS
section 3 addresses procurement policy and procedure.[Footnote 52]
AMS States a Nonregulatory FAA Policy:
A further significant foundational difference between AMS and the FAR
is that AMS sets out a nonregulatory FAA policy, whereas the FAR was
adopted and is maintained as a set of published governmentwide
regulatory requirements, which form a legal basis for federal agencies'
contract decision-making. AMS is binding on FAA personnel as FAA
employees and establishes other guidelines that FAA states should be
followed unless there is a rational basis for doing otherwise. AMS is
subject to such internal controls as the Administrator chooses to
enforce and general overarching legal requirements, such as the
Government Performance and Results Act of 1993 (GPRA).[Footnote 53]
There is a legal requirement, created by section 348, that small and
socially or economically disadvantaged firms be given all reasonable
opportunities to receive contract awards. FAA in its Office of Dispute
Resolution for Acquisition has adopted a dispute resolution process
with some legal underpinnings.[Footnote 54] Otherwise, as the preface
to AMS states, "nothing in this document creates or conveys any
substantive [legal] rights." In short, FAA has assumed no legal
obligation to follow AMS other than to ensure that its actions are not
arbitrary and capricious or contrary to law. By contrast, the FAR has
the force and effect of law, and agencies that are subject to the FAR
are bound to follow it.
AMS Chapter 3 Parallels a Subset of the FAR:
When FAA personnel apply the procurement methodology in AMS chapter 3,
they are applying guidance that closely parallels some of the
procedures set out in the FAR. The AMS Chapter 3 acquisition process
parallels a subset of the varied selection of procurement methods
available under the FAR, requiring that all competitive FAA contracts
be negotiated with the awardee being selected on a "best value" basis.
The FAR also provides a much more detailed set of information and
guidance than does AMS. A comparison of high-level differences and
similarities between AMS and the FAR is presented in table 8.
Table 8: Comparison of AMS and the FAR:
Best value source selection;
AMS: Yes, following screening;
FAR: Yes, although other methods are also available for use when
appropriate.
Public announcement of requirement;
AMS: Public announcement through Internet or other means when value of
contract is anticipated to exceed $100,000;
FAR: Yes, for proposed contract actions expected to exceed $25,000.
Competition;
AMS: FAA's policy is to provide reasonable access to competition for
firms interested in obtaining contracts. In selecting sources, the
preferred method of procurement is to compete requirements among two or
more sources;
FAR: Full and open competition--all responsible sources are permitted
to compete.
Sole-source procurement;
AMS: Yes, when deemed to be in FAA's "best interest" as determined by
the service organization on the basis of "adequate objective supporting
data.";
FAR: Yes, full and open competition need not be obtained under certain
specified conditions based upon a written justification from the
contracting officer that is approved at an appropriate level of
authority.
Prequalification;
AMS: Yes, qualification information screens for those vendors that meet
FAA's stated minimum capabilities or requirements for providing a given
product or service;
FAR: Yes, for products or manufacturers when justified in writing and
conducted in a manner that meets requirements justifying the use of
qualifications requirements.
Basic methodology in negotiated procurement;
AMS: FAA issues one or more "screening requests," which may include
requests for binding offers from competing firms;
FAR: Agency issues a solicitation, usually a request for proposals.
Methodology for negotiation;
AMS: FAA encourages one-on-one communications throughout the process
provided that no offeror is given an "unfair advantage";
FAR: Clarification and discussions are permitted;
one offeror cannot be favored over another.
Evaluation and award selection;
AMS: Selection is based on evaluation in accord with criteria
identified in the screening request. The selection decision is a
judgmental decision made by the source selection official;
FAR: Selection is based on evaluation in accord with criteria
identified in the request for proposals. The selection decision is a
judgmental decision made by the source selection official.
Use of simplified acquisition methods;
AMS: Commercial and simplified purchases are used for commercial items
or for products or services that have been sold at established catalog
or market prices and are generally purchased on a fixed-price basis;
FAR: Generally required for purchases up to $100,000, for noncommercial
items, or on a test basis, up to $5,000,000 for commercial items
competition is to be obtained to the maximum practicable extent.
Use of credit card purchases;
AMS: Permitted;
FAR: Permitted.
Procurement methodology;
AMS: AMS does not include the level of detail found in the FAR. It does
not prescribe many of the procurement methods and techniques permitted
under the FAR, but encourages use of "any method of procurement deemed
appropriate.";
FAR: Provides a broad selection of procurement methods and techniques
suitable for use in most circumstances.
Responsibility;
AMS: Awards to responsible offerors only;
FAR: Awards to responsible offerors only.
Source: GAO analysis.
[End of table]
AMS Includes a Less Rigorous Competition Requirement Than Does the FAR:
As table 8 indicates, AMS incorporates a less rigorous competition
standard than the FAR imposes on the rest of the government. AMS states
that it is FAA's policy to provide reasonable access to competition for
firms interested in obtaining contracts. According to AMS, in selecting
sources, the preferred method of procurement is to compete requirements
among two or more sources. However, there is no requirement to ensure
that firms that want to participate actually get a chance to do so.
Instead FAA may limit competition for further consideration in its
screening process to firms with known capabilities or past performance.
The FAR Gives Procurement Professionals Tighter Control over
Procurement Decisions:
AMS states that authority is delegated to appropriate levels. Once the
final investment decision is made, and subject only to any constraints
imposed by that decision, the service-level organization is responsible
for conducting required acquisitions. Contracting personnel as well as
other specialists are then assigned to teams that are responsible to a
program manager within the service-level organization. FAA states that
this approach increases the pace of doing business. By comparison, the
FAR gives contracting professionals clear control over contracting
decisions by requiring that procurement decisions be made by
procurement professionals--typically contracting officers or their
superiors.
Although FAA Project Managers View AMS as More Efficient and Flexible
Than the FAR, Some Procurement Officials We Interviewed Do Not Agree:
As part of our work, we interviewed project management personnel within
FAA as well as current and former FAA procurement officials that have
experience using both the FAR and AMS. Generally, FAA personnel see AMS
as more efficient and flexible than the FAR, although 9 years after
AMS's adoption, many FAA officials have only limited knowledge of and
experience with the FAR. The FAA project managers we interviewed see
AMS as more efficient and flexible than the FAR,[Footnote 55] but some
procurement officials with experience in applying both AMS and the FAR
did not agree with the view that the FAR was unduly rigid. According to
these officials, the FAR may appear inflexible and cumbersome to
persons who are inexperienced with it, but those who are familiar with
it are able to navigate its complexities effectively. For example, even
though the FAR generally requires full and open competition--a process
that can take time to give all interested firms an opportunity to
participate--contracting officers may be able to expedite the
procurement process by using authorized streamlined procedures or, if
circumstances warrant, by justifying sole-source or limited
competition.
[End of section]
Appendix IV: How FAA's Acquisition Policy Adapted Key Recommendations
Made by GAO and DOT (1996-2003):
FAA Refined AMS in Response to Recommendations:
Since FAA developed and implemented AMS in 1996, GAO and the DOTIG have
made recommendations to improve FAA's acquisition processes. FAA has
adopted many of these recommendations and incorporated them into AMS
(see table 9). These implemented recommendations address four main
themes:
* Developing a strategy for culture change that relies on successfully
integrating the various elements of acquisition, including specific
responsibilities and performance measures for all stakeholders, and
providing the incentives needed to promote the desired changes.
* Establishing an effective management structure for developing,
maintaining, and enforcing the ATC systems architecture to provide an
overall plan for the National Airspace System (NAS). This management
structure should assign the responsibility and accountability to
develop, maintain, and enforce a complete and unified ATC system by
ensuring that every project conforms to the overall plan.
* Improving cost and schedule tracking to provide data for estimating
the costs and schedules of programs. To estimate the costs and time
needed for projects, a historical database that includes cost and
schedule estimates, revisions, reasons for revisions, actual cost and
schedule information, and relevant contextual information is needed.
* Improving the management of modernization projects, including the use
of project reviews, milestones, and baselines, and cost-accounting
information to ensure that programs can be adjusted as needed.
The reports identified in table 10 provide recommendations to address
problems we and the DOTIG have identified under these four themes.
Table 9: Key Recommendations Made to Improve FAA's Acquisition
Processes:
Key recommendation: Aviation Acquisition: A Comprehensive Strategy Is
Needed for Cultural Change at FAA; August 22, 1996, (GAO/RCED-96-159);
FAA should develop a comprehensive strategy for cultural change. This
strategy should include specific responsibilities and performance
measures for all stakeholders throughout FAA and provide the incentives
needed to promote the desired behaviors and to achieve agencywide
cultural change;
Evidence of policy change: FAA issued an organizational culture
framework in 1997 and is working to implement it;
Rationale for change: Over the past 15 years, FAA's ATC modernization
projects have experienced substantial cost overruns, lengthy schedule
delays, and significant performance shortfalls. We found that FAA's
organizational culture has been an underlying cause of the agency's
acquisition problems. Its acquisitions were impaired because employees
acted in ways that did not reflect a strong commitment to mission
focus, accountability, coordination, and adaptability.
Key recommendation: Air Traffic Control: Improved Cost Information
Needed to Make Billion Dollar Modernization Investment Decisions;
January 22, 1997, (GAO/AIMD-97-20); Because the success of FAA's
investment analysis and decision-making process depends in large
measure on the reliability of ATC project cost information, FAA should
institutionalize defined processes for estimating ATC projects' costs.
At a minimum, these processes should include the following six
institutional process requisites, developed for organizations that are
building or acquiring software-intensive systems by Carnegie Mellon
University's Software Engineering Institute (SEI), an institution
recognized for its expertise in software processes. Each of these
requisites is described in more detail in this report:
* a corporate memory, or historical database(s), which includes cost
and schedule estimates, revisions, reasons for revisions, actual cost
and schedule information, and relevant descriptive information;
* structured approaches for estimating software size and the amount and
complexity of existing software that can be reused;
* cost models calibrated/ tuned to reflect demonstrated accomplishments
on past projects;
* audit trails that record and explain all values used as cost model
inputs;
* processes for dealing with externally imposed cost or schedule
constraints in order to ensure the integrity of the estimating process;
* data collection and feedback processes that foster capturing and
correctly interpreting data from work performed; [Empty];
Evidence of policy change: Chapter 19 of FAA's Pricing Handbook
embodies SEI's philosophy, which maintains that developing credible
software estimates is a function of how thorough and disciplined an
organization's estimating processes are. SEI's six institutional
process requisites are designed to ensure that organizations
consistently produce reliable cost estimates for software-intensive
systems. These requisites are as follows:
* a corporate memory, or historical database(s), for cataloging cost
estimates, revisions, reasons for revisions, actual cost and schedule
information, and other descriptive information, such as any constraints
or trends that affect the project;
* structured processes for estimating software size and the amount and
complexity of existing software that can be reused;
* cost models calibrated/tuned to reflect demonstrated accomplishments
on similar past projects;
* audit trails that record and explain the values used as cost model
inputs;
* processes for dealing with externally imposed cost or schedule
constraints to ensure the integrity of the estimating process;
* data collection and feedback processes that foster capturing and
correctly interpreting data from work performed;
Rationale for change: We found that FAA's ATC modernization program's
cost estimating processes do not satisfy recognized estimating
requisites, and its cost-accounting practices do not provide for proper
accumulation of actual costs. The result is an absence of reliable
project cost and financial information that the Congress has
legislatively specified and that leading public-sector and private-
sector organizations point to as essential to making fully informed
investment decisions among competing ATC projects. Not having this
information, increases the likelihood of poor ATC investment decisions,
not only when a project is initiated but also throughout its life
cycle. It also means that Congress does not have reliable cost
information to use in making funding decisions about FAA. Such a
situation is unacceptable when making small investments, but is
especially egregious when making multimillion or billion-dollar
investments in mission-critical ATC systems.
Key recommendation: FAA should immediately begin disclosing the
inherent uncertainty and range of imprecision in all ATC projects'
official cost estimates presented to executive oversight agencies or
Congress;
Evidence of policy change: Chapter 19 of FAA's Pricing Handbook
incorporates our recommendation and refers explicitly to GAO/AIMD-97-20
and the work of other experts. The handbook suggests where to
incorporate audit trails, constraint processes, and the inherent
uncertainty and range of imprecision in all ATC cost estimates. The
handbook advocates that staff qualify early project estimates by
disclosing the level of uncertainty associated with them and refining
the estimates as the project is completed and the uncertainty
eliminated.
Key recommendation: FAA should acquire or develop and implement a
managerial cost-accounting capability that will satisfy the
requirements of Statement of Federal Financial Accounting Standards no.
4 (SFFAS 4) Managerial Cost Accounting Concepts and Standards for the
Federal Government. This system capability should provide the cost-
accounting and financial management information needed by FAA
management and those who make investment decisions. Such information
should include full life-cycle costs, which include the costs of
resources consumed by a project that directly or indirectly contribute
to the output and the costs of identifiable supporting services
provided by other organizations within the reporting entity;
Evidence of policy change: The Department of Transportation is in the
process of meeting key objectives of the Federal Managers' Financial
Integrity Act (FMFIA) of 1982. A key material weakness was FAA's
oversight of cost reimbursable contracts. FAA made significant progress
in the closeout of past cost reimbursable contracts. To resolve this
material weakness, FAA needs to complete the close out of old contracts
and increase the use of cost incurred audits. Additionally, FAA needs
to ensure that appropriate audits are obtained for all active
contracts. These steps will help reduce the likelihood of cost overruns
or improper payments for unallowable costs.
Key recommendation: FAA should report its lack of a cost-accounting
capability for its ATC modernization as a material internal control
weakness in the Department's fiscal year 1996 Federal Managers'
Financial Integrity Act (FMFIA) report and in subsequent annual FMFIA
reports until the problem is corrected.
Key recommendation: FAA should report to the Secretary of
Transportation and FAA's authorizing and appropriation committees on
its progress in implementing these recommendations as part of its
fiscal year 1999 budget submission.
Key recommendation: Air Traffic Control: Complete and Enforced
Architecture Needed for FAA Systems Modernization; February 3,1997,
(GAO/AIMD-97-30); FAA should ensure that a complete ATC systems
architecture is developed and enforced expeditiously before deciding on
the architectural characteristics of a replacement for the Host
Computer System. FAA should also take the following steps to establish
an effective management structure for developing, maintaining, and
enforcing the complete ATC systems architecture:
* Assign the responsibility and accountability needed to develop,
maintain, and enforce a complete ATC systems architecture to a single
FAA organizational entity;
* Provide this single entity with the resources, expertise, and
budgetary and/or organizational authority needed to fulfill its
architectural responsibilities;
* Direct this single entity to ensure that every ATC project conforms
to the architecture unless careful, thorough, and documented analysis
supports an exception. Given the importance and the magnitude of the IT
initiative at FAA, a management structure similar to the department-
level chief information officer (CIO) structure prescribed in the
Clinger-Cohen Act should be established for FAA;
Evidence of policy change: AMS states the National Air Space (NAS)
Configuration Control Board shall approve changes to NAS technical
documentation, and shall ensure the traceability of requirements from
the NAS level to the system and subsystem level. This responsibility
begins with the approval of the technical architecture by the Joint
Resources Council at the investment decision and continues throughout
the life of the program; AMS states that the Joint Resources Council
approves FAA budget submissions for Research, Engineering and
Development (RE&D) and Facilities and Equipment (F&E) appropriations,
participates in the development of FAA budget submissions for the
operations appropriation, and approves the NAS architecture baseline;
AMS states that a configuration control board with an approved charter
and operating procedures shall be the official FAA-wide forum used to
establish configuration management baselines and to approve or
disapprove subsequent changes to those baselines;
Rationale for change: FAA lacks a complete system architecture, or
overall blueprint, to guide and constrain the development and
maintenance of the many interrelated systems that make up its ATC
infrastructure. To its credit, FAA is developing one of the two
principal components of a complete systems architecture, namely, the
"logical" description of FAA's current and future concept of ATC
operations as well as descriptions of the ATC business functions to be
performed, the associated systems to be used, and the information flows
among systems. However, FAA is not developing, nor does it have plans
to develop, the second essential component--the ATC-wide "technical"
descriptions that define all required information technology (IT) and
telecommunications standards and critical ATC systems' technical
characteristics; We also found that an architecture is the centerpiece
of sound systems development and maintenance; FAA is developing a
logical architecture component for ATC modernization and evolution; FAA
lacks a technical architectural component to guide and constrain ATC
modernization and evolution; without a technical ATC architecture,
costly system incompatibilities have resulted and; will continue; and
FAA lacks an effective management structure for developing and
enforcing an ATC systems architecture.
Key recommendation: Air Traffic Control: Immature Software Acquisition
Processes Increase FAA System Acquisition Risks; March 21, 1997, (GAO/
AIMD-97-47); Given the importance and the magnitude of IT at FAA, this
report reiterates our earlier recommendation calling for the
establishment at FAA of a CIO management structure similar to the
department-level CIO structure prescribed in the Clinger-Cohen Act of
1996. To improve its ability to acquire software for its ATC
modernization, FAA should;
* assign responsibility for software acquisition process improvement to
the agency's CIO;
* provide the CIO with the authority needed to implement and enforce
ATC modernization software acquisition process improvement;
* require the CIO to develop and implement a formal plan for ATC
modernization software acquisition process improvement that is based on
the software capability evaluation results contained in this report and
specifies measurable goals and time frames, prioritizes initiatives,
estimates resource requirements, and assigns roles and
responsibilities;
* allocate adequate resources to ensure that planned initiatives are
implemented and enforced; and;
* require that, before being approved, every; ATC modernization
acquisition project have software acquisition processes that satisfy at
least Software Acquisition Capability Maturity Model (SA-CMM) level 2;
Evidence of policy change: FAA states that the CIO:
* serves as the principal adviser to the Administrator, Deputy
Administrator, and FAA offices on information management and technology
across the agency. As the agency's senior management official, serves
as the spokesperson on IT matters before Congress, other agencies, and
the public;
* leads and directs agencywide strategic planning for IT;
* oversees IT investments to ensure optimization across all agency
groups and the full range of cost trade-offs;
* creates and maintains an IT strategy to guide research, development,
maintenance, and sharing of information systems, applications, data,
and other resources across the lines of business and throughout the
agency;
* leads the establishment of world-class software and information
systems engineering methodologies including Capability Maturity
Models, and applies them to agency systems, operations, and processes
to provide continuous improvement of IT performance; and;
* leads and directs agencywide efforts on; information systems
security, ensuring that standards and policies are in place to provide
security for the critical information architecture of the agency;
Rationale for change: To accommodate forecasted growth in air traffic
and replace aging equipment, FAA embarked on an ambitious ATC
modernization program in 1981. FAA estimated that it would spend about
$20 billion to replace and modernize software-intensive ATC systems
between 1982 and 2003. Our work over the years has chronicled many FAA
failures in meeting ATC projects' cost, schedule, and performance
goals, largely because of software-related problems. As a result of
these failures as well as the tremendous cost, complexity, and mission
criticality of FAA's ATC modernization program, we designated the
program as a high-risk IT initiative in our 1995 and 1997 report series
on high-risk programs; Software quality is governed largely by the
quality of the processes involved in developing or acquiring, and
maintaining it. SEI has developed models and methods that define and
determine organizations' software process maturity. Together, they
provide a logical framework for baselining an organization's current
process capabilities (i.e., strengths and weaknesses) and providing a
structured plan for incremental rocess improvement. We found that;
* FAA's ATC modernization software acquisitions processes are immature
and;
* FAA's approach for improving AT; modernization software acquisition
processes is not effective.
Key recommendation: Air Traffic Control: FAA's Modernization Investment
Management Approach Could Be Strengthened, April 30, 1999, (GAO/RCED/
AIMD-99-88); FAA should implement a comprehensive investment management
approach through AMS that includes the following actions:
* Establish a complete portfolio of investments--including existing
systems funded by the operations budget account as well as projects
funded by the facilities and equipment account--and require the Joint
Resources Council to periodically review the baseline status and merits
of each of these investments throughout their entire life cycle. As
part of this portfolio, cost baselines for operating and maintaining
all projects should be developed, and this information should be
included in the agency's financial plan for its investments and in its
annual budget request to Congress;
Evidence of policy change: FAA's AMS states that five decisions are
always made at the corporate level by the Joint Resources Council: the
mission need decision, the investment decision, the decision to approve
a change to an acquisition program baseline, approval of the RE&D and
F&E budget submissions, and approval of the NAS Architecture baseline.
The selection of a solution to satisfy a mission need, the investment
of resources into a fully funded program, and the possible need to
cancel other programs to accommodate a new program make the investment
decision the most important in the life-cycle management process;
Rationale for change: Over the past 17 years, FAA's modernization
projects have experienced substantial cost overruns, lengthy delays,
and significant performance shortfalls. Because of FAA's contention
that some of its modernization problems were caused by federal
acquisition regulations, the Congress enacted legislation in November
1995 that exempted the agency from most federal procurement laws and
regulations and directed FAA to develop a new acquisition management
system. In response, FAA implemented AMS on April 1, 1996. AMS provides
high-level acquisition policy and guidance for selecting and
controlling investments throughout all phases of the acquisition life
cycle. GAO found that:
* FAA's AMS is designed to provide a discipline, structured process for
selecting and controlling investments;
* Lack of oversight of the operations portion of projects prevents FAA
from managing investments as a complete portfolio;
* Weaknesses in selection, control, and evaluation phases limit FAA's
effectiveness in managing its portfolio.
Key recommendation:
* Improve the selection process by (1) establishing clearly defined
procedures for validating each project's cost, schedule, benefit,
performance, and risk information and (2) requiring documentation of
the results of the validation procedures applied to each project;
Evidence of policy change: FAA's AMS states that the investment
analysis team develops an initial acquisition program baseline (i.e.,
performance, cost, schedule, benefits, and risk) for each alternative
solution offering superior value and benefit to FAA and its customers.
Service organization members of the investment analysis team lead the
development of cost and schedule baselines using FAA's work breakdown
structure and other applicable standards.
Key recommendation:
* Strengthen control over investments by (1) revising the acquisition
program baseline requirements to include project risks and to add
milestones for project reviews during the operations phase and (2)
ensuring that project officials fully track and document estimated
versus actual results for all the elements (i.e., cost, schedule,
benefit, performance, and risk) contained in the baseline
documentation;
Evidence of policy change: AMS states that the acquisition program
baseline should include cost, schedule, performance, benefits, and risk
information. It also should include all events that are key to
satisfying mission need, providing intended operational capability, and
accruing benefits, as well as events crucial to interrelated programs
or NAS systems. Once an estimate has been completed and a project
started, FAA establishes reporting and performance measures to compare
estimated and actual costs, schedules, and performance.
Key recommendation:
* Initiate post implementation evaluations for projects within 3 to 12
months of deployment or cancellation to compare the completed projects'
cost, schedule, performance, and mission improvement outcomes with the
original estimates;
* Incorporate key information from the selection process (e.g., mission
need statements, cost-benefit analyses, and risk assessments) into
FAA's management information system for investments;
Evidence of policy change: FAA published a methodology for conducting
such evaluations entitled An Approach for Developing a Standard Method
for Conducting Post-Implementation Reviews, Report #2001-13, June 6,
2001.
Key recommendation: Major Management Challenges and Program Risks,
Department of Transportation; January 2001, (GAO-01-253); FAA should
develop a comprehensive plan that would include established checkpoints
at which the agency would determine, among other things, whether users'
needs have changed and whether other technologies have matured and
could better meet users' needs and the agency's requirements for
satellite navigation. FAA should also have an external organization
evaluate its progress at established checkpoints and include the
results of this evaluation in its request for future funding of the
navigation system;
Evidence of policy change: FAA has appointed an independent board--
consisting of external experts in satellite navigation, safety
certification, and radio spectrum--that reports directly to the FAA
Administrator. The board is tasked with reviewing the soundness of the
panel's recommendations and with revalidating the future path for WAAS.
However, given the past problems in developing this system and the
long-term effort that is still required, we believe that continued
oversight by an independent group of experts is warranted. It is not
clear whether the current independent board will fulfill this role. We
will continue to evaluate FAA's progress on this and other system
acquisition efforts;
Rationale for change: DOT's management of its major acquisitions and
assets needs improvement in several areas. FAA and the U.S. Coast Guard
are undertaking costly, long-term programs to modernize and replace
aging equipment. Over the past 19 years, FAA's multibillion-dollar ATC
modernization program has experienced cost overruns, delays, and
performance shortfalls of large proportions. FAA is making progress in
addressing some of our recommendations, but its reform efforts are not
complete, and major projects continue to face cost, schedule, and
performance problems. Because of its size, complexity, cost, and
problem-plagued past, we designated FAA's IT program as a high-risk IT
initiative in 1995.
Key recommendation: Status of FAA's Major Acquisitions; DOT/OIG, AV-
2003-045, June 26, 2003; Update the cost, schedule, and performance
baselines for many of FAA's major acquisition, including STARS, ITWS,
LAAS, and WAAS at a minimum. Develop--and use--performance goals for
assessing progress with its major acquisitions. This should involve
holding staff and contractors accountable for keeping projects within
cost and schedule, as appropriate;
Evidence of policy change: FAA officials generally agreed with the
analysis and recommendations in this report. FAA is implementing this
recommendation. It updated the baseline of STARS in April 2004 and
updated the baselines of ITWS and WAAS in May 2004. The LAAS program
was deferred because of budget cuts;
Rationale for change: FAA has made progress with a number of
acquisitions, including Free Flight Phase 1 and new information
exchange systems that link FAA and airline operations centers. However,
other modernization programs have experienced cost, schedule, and
performance problems. Problems with acquisition efforts have serious
consequences because they result in costly interim systems, reduce the
number of units procured, postpone benefits, or "crowd out" other
modernization projects.
Key recommendation: Status Report on FAA's Operational Evolution Plan;
DOT/OIG, AV-2003-048, July 23, 2003; Develop realistic cost estimates,
and link the Operational Evolution Plan (OEP) with FAA's budget in
order to set priorities for what can be accomplished in the short term;
Determine--in concert with the aviation community--how to move forward
(and at what pace) with systems that require airspace users to purchase
and install new technologies; Determine and maximize the benefits
associated with airspace design changes, new procedures, and
capabilities currently onboard aircraft to enhance system capacity;
Evidence of policy change: FAA officials generally agreed with the
analysis and recommendations in this report. FAA is currently updating
the OEP, which includes design changes to the National Airspace to, for
example, enhance capacity;
Rationale for change: The OEP is an important effort because it will
shape FAA and industry investments over the next decade. However, much
has changed since the OEP was introduced. The demand for air travel has
declined, major network carriers are in financial distress, and
Aviation Trust Fund revenues have declined sharply. The Inspector
General found that fundamental assumptions about the OEP, such as the
cost, schedule, and benefits of key efforts as well as the ability of
airspace users to pay for and equip with new technologies in the near
term, are no longer valid and need to be revised.
Source: GAO analysis.
[End of table]
[End of section]
Appendix V: Status of the Seven ATC Modernization Acquisitions That GAO
Reviewed:
Standard Terminal Automation Replacement System (STARS):
Purpose and Status:
STARS is a joint FAA and Department of Defense (DOD) program. It will
replace aging legacy terminal FAA and DOD automation systems with
terminal ATC systems. Civil and military air traffic controllers across
the nation are using STARS to direct aircraft near major airports.
In June 2003, FAA commissioned STARS for use at the Philadelphia
International Airport in Pennsylvania. Currently, STARS is fully
operational at 24 FAA terminal radar control facilities and 17 DOD
facilities. Under the ATO's new business model of breaking large and
complex programs into smaller phases to control cost and schedule,
STARS is a candidate for further deployment to about 120 FAA and DOD
operational facilities. In May 2004, FAA changed STARS's cost and
schedule estimates for the third time and estimates that it will cost
$1.46 billion to deploy STARS at 50 operational facilities.
Contractor: Raytheon.
[See PDF for image]
Photo caption: STARS Display Monitor.
Source: FAA
[End of figure]
Baseline Changes to STARS Scope, Schedule and Cost:
Date: February 1996;
Number of FAA facilities receiving STARS: 172;
Projected date for first deployment: 1998;
Projected date for last deployment: 2005;
Estimated cost (F&E)[A]: $0.94 billion.
Date: October 1999;
Number of FAA facilities receiving STARS: 188;
Projected date for first deployment: 2002;
Projected date for last deployment: 2008;
Estimated cost (F&E)[A]: $1.4 billion.
Date: March 2002;
Number of FAA facilities receiving STARS: 73;
Projected date for first deployment: 2002;
Projected date for last deployment: 2005;
Estimated cost (F&E)[A]: $1.33 billion.
Date: April 2004;
Number of FAA facilities receiving STARS: 50;
Projected date for first deployment: 2003;
Projected date for last deployment: 2008;
Estimated cost (F&E)[A]: $1.46 billion.
Total change;
Number of FAA facilities receiving STARS: -122;
Projected date for first deployment: + 3 years;
Projected date for last deployment: + 3 years;
Estimated cost (F&E)[A]: +$0.52 billion.
Source: GAO's presentation of FAA data;
[A]FAA's Facilities and Equipment (F&E) account funds capital projects.
[End of table]
Risks and Challenges:
Certification issues - FAA also experienced problems in certifying
STARS, in part because of aggressive scheduling. FAA's approach to
certifying STARS was oriented to rapid deployment to meet critical
needs. To meet these needs, FAA compressed its original 32-month
development and testing schedule into 25 months. This compressed
schedule left only limited time for human factor evaluations and not
enough time for computer human; interface issues and involvement of
controllers and maintenance technicians.
Airport Surveillance Radar Model-11 (ASR-11):
Purpose and Status:
ASR-11 will provide high-quality digital data to terminal controllers
in terminal environments. It will also provide a more reliable
replacement for aging analog radars like ASR-7 and ASR-8; it will also
provide digitized radar data for the new automation systems such as
STARS. In addition, ASR-11 will provide six levels of weather
information, a significant improvement over the current two levels. The
ASR-11 program is a joint program with DOD-that is, DOD is managing the
program to joint specifications, and FAA will provide DOD with the
funds to procure 112 units. ASR-11 is a nondevelopmental item.
The in-service decision was made in 2003, and the radar is being
deployed to 108 sites. The ASR-11 program is scheduled to be
rebaselined for cost and schedule in fiscal year 2005.
Contractor: Raytheon.
ASR-11 Scope, Schedule, and Cost:
Date: March 2002;
Number of facilities receiving ASR-11: 112;
Projected date for first deployment: 2000;
Projected date for last deployment: 2005;
Estimated cost (F&E): $743.3 million.
Date: July 2004;
Number of facilities receiving ASR-11: 112;
Projected date for first deployment: 2003;
Projected date for last deployment: 2013;
Estimated cost (F&E): $891.7 million.
Total change;
Number of facilities receiving ASR-11: 0;
Projected date for first deployment: 3 years;
Projected date for last deployment: 8 years;
Estimated cost (F&E): $148.4 million.
[End of table]
Source: GAO's presentation of FAA data.
Risks and Challenges:
The Capital Investment Plan does not support the service as required in
the current Acquisition Program Baseline, which could put the program
in jeopardy.
Integrated Terminal Weather System (ITWS):
Purpose and Status:
ITWS provides automated weather information for use by air traffic
controllers and supervisors in airport terminal airspace (60 miles
around the airport). It provides products that require no
meteorological interpretation to air traffic controllers, air traffic
managers, pilots, and airlines. ITWS provides a comprehensive current
weather situation and highly accurate forecasts of expected weather
conditions for the next 30 minutes.
Current FAA plans call for the installation of 34 systems that will
service various airports. Six systems are operational, and feedback
from users is satisfactory. In May 2004, the ATO Executive Council
rebaselined the program to include a weather-forecasting capability in
the production baseline, a new requirement to provide operational
support for the New York prototype, and change the operations and
maintenance cost baseline for the program. However, the council did not
include additional funding, and therefore, in order to stay within the
capital improvement program's (CIP) funding levels, the program has
proposed to defer 12 of the planned 34 systems installations.
Contractor: Raytheon.
Baseline Changes to ITWS Scope, Schedule, and Cost:
Date: June 1997;
Number of facilities receiving ITWS: 34;
Projected date for first deployment: Sep 01-Mar 02;
Projected date for last deployment: Jan 03-Jul 03;
Estimated cost (F&E): $276.1 million.
Date: August 2001;
Number of facilities receiving ITWS: 34;
Projected date for first deployment: December 2002;
Projected date for last deployment: May 2004;
Estimated cost (F&E): $282.2 million.
Date: May 2004;
Number of facilities receiving ITWS: 34;
Projected date for first deployment: December 2002;
Projected date for last deployment: 2009+;
Estimated cost (F&E): $288.3 million.
Total change;
Number of facilities receiving ITWS: 0;
Projected date for first deployment: 1+ years;
Projected date for last deployment: 6+ years;
Estimated cost (F&E): $12.2 million.
Source: GAO's presentation of FAA data:
[End of table]
Risks and Challenges:
Funding issues -The program requested and obtained approval to
rebaseline. The baseline is being modified to incorporate the Terminal
Convective Weather Forecasting (TCWF) capability into the production
baseline. As directed by the ATO Executive Council, responsibility for
funding operational support for the New York prototype system is also
being added to the baseline. The ATO Executive Council also directed
that the cost of the program remain at the current CIP funding levels
for fiscal years 2005, 2006, and 2007. In order to stay within the CIP
funding levels, the program proposed to defer 12 of the planned 34
systems installations.
Schedule issues - Because of constrained funding, 12 airports will not
receive ITWS capabilities until after 2009.
Local Area Augmentation System (LAAS):
Purpose and Status:
LAAS is a precision approach and landing system that will augment the
Global Positioning System (GPS) to broadcast highly accurate
information to aircraft on the final phases of a flight. LAAS consists
of both ground and avionics components. Ground components include GPS
reference receivers, which monitor and track GPS signals; very-high-
frequency transmitters for broadcasting the LAAS signal to aircraft;
and ground station equipment, which generates precision approach data
and is housed at or near an airport. Aircraft will be equipped with
avionics to receive LAAS signals.
FAA's fiscal year 2005 budget eliminated funding for LAAS, and
remaining fiscal year 2004 funds will continue to validate LAAS
requirements and address radio frequency interference issues. FAA
officials will reconsider national deployment when more research
results are completed.
Baseline Changes to LAAS Schedule and Cost:
Date: January 1998;
Projected date for first deployment: 2002;
Projected date for last deployment: TBD;
Estimated cost (F&E): $530.1 million.
Date: September 1999;
Projected date for first deployment: 2003;
Projected date for last deployment: TBD;
Estimated cost (F&E): $696.1 million.
Total change;
Projected date for first deployment: 1 year;
Estimated cost (F&E): +$166 million.
Source: GAO presentation of FAA data.
[End of table]
Risks and Challenges:
Cost issues - LAAS cost estimates are not reliable, reflecting
inadequate requirements development in the early stages of the program,
a lack of understanding of a mission degradation issue, incomplete
software development, and an unrealistic development schedule.
Schedule issues -The LAAS schedule was not realistic. Specifically, FAA
lacked an understanding of the integrity requirement and software
development, which were the two biggest technological maturity issues
facing the LAAS program.
Performance Issues - FAA has not resolved the integrity requirement
that ensures pilots are alerted in a timely manner when the LAAS signal
is not reliable. FAA has not been able to prove that the system is safe
during solar storms. An analysis of the effects of solar storms on the
LAAS signal's integrity is under way, but an atmospheric monitoring
device that could address this issue may not be available until fiscal
year 2009.
Next Generation Air/Ground Communications (NEXCOM):
Purpose and Status:
The Next Generation Air/Ground Communications (NEXCOM) project is to
replace the existing analog ATC communications system with a new
digital system that would have greater capabilities. The initial
development, of a multimodal digital radio (MDR), is to be followed by
the development of aircraft avionics and ground systems. NEXCOM is
expected to increase the number of available communications channels,
provide simultaneous voice and data transmission between controllers
and pilots, and require a digital form of authentication, designed to
prevent "phantom controllers" from gaining access to the communications
system. FAA plans to deploy 6,000 MDR pairs (a radio pair is one
receiver and one transmitter) during the first phase, which will
provide voice channels to aircraft in the en route environment.
NEXCOM completed Independent Operational Test and Evaluation assessment
of the radio component at the Santa Barbara, California, Remote Center
Air/Ground Communications facility, and radios were approved for in-
service and national deployment in July 2004. The avionics component's
development is scheduled to be completed by 2006. However, proposed
funding cuts to FAA's fiscal year 2005 budget required the termination
of the ground station development, which would enable communications in
the more efficient digital mode.
Contractor: ITT for MDR.
Baseline Changes to NEXCOM Scope, Schedule, and Cost:
Date: May 2000;
Number of radio pairs deployed: 6,000;
Date first site Initial Operating Capability: July 2002;
Date of In-Service Decision: October 2002;
Estimated cost (F&E): $318.4 million[A].
Date: February 2004;
Number of radio pairs deployed: 6,000;
Date first site Initial Operating Capability: March 2004;
Date of In-Service Decision: July 2004;
Estimated cost (F&E): $318.4 million.
Total change;
Number of radio pairs deployed: 0;
Date first site Initial Operating Capability: 20 months;
Date of In-Service Decision: 21 months;
Estimated cost (F&E): $0.
Source: GAO presentation of FAA data.
[A] Estimated cost is only for the NEXCOM MDR. The NEXCOM ground
station contract was canceled in March 2004 and is being terminated.
Risks and Challenges:
Schedule issues-FAA planned to base the MDR on a nondevelopmental item
(NDI), and the initial schedule allowed only limited development.
However, FAA's requirement that communications channels be free of
signal interference ("quiet channels") was more demanding than the NDI
solution was capable of achieving. As a result, further development was
necessary, delaying the initial operational capability and in-service
decision by 21 months.
Performance-The NEXCOM radio meets its operational requirement for
coverage. However, to achieve this requirement FAA determined that the
NEXCOM radios would have to achieve the same power output level (50
watts) that the existing radios produced. The contractor is delivering
radios that put out no more than 34 watts per channel. This posed an
"unacceptable consequence" and FAA performed additional tests or
flights checks and determined that the reduced power would not
adversely affect operations and has approved the use of the lower-
output radios.
Advanced Technologies and Oceanic Procedures (ATOP):
Purpose and Status:
The Advanced Technologies and Oceanic Procedures (ATOP) program
introduces new controller workstations, data-processing equipment, and
software designed to enhance the control and flow of oceanic air
traffic to and from the United States. ATOP processes aircraft position
updates automatically, whereas currently, oceanic traffic control
operations are performed manually and updated via paper flight strips.
ATOP is designed to present flight data "electronically" in a format
similar to these paper strips ATOP completed operational testing at its
first site, the Oakland Air Route Traffic Control Center (ARTCC) and
achieved initial operational capability (IOC) on June 30, 2004.
Currently, ATOP is in limited use for 4 hours a day 5 days a week in
one of nine sectors under Oakland's control. Plans to fully transition
ATOP to all nine sectors depend upon feedback from the initial trials
and sector-by-sector capabilities. Other operational considerations
still to be resolved are additional staff needs, ATOP's training
schedule, and coordination with North American Aerospace Defense
Command on an interface device. FAA is currently in the early phases of
installing ATOP at the New York ARTCC and is scheduled to achieve IOC
in March 2005. Additional software that will incorporate radar data
into ATOP is under development and scheduled to be completed by
November 2004. This software is expected to be operational at the final
site, the Anchorage ARTCC, in March 2006.
Contractor: Lockheed Martin Transportation and Security Solutions.
Baseline Changes ATOP Scope, Schedule, and Cost:
Date: May 2001 (baseline);
Number of facilities receiving ATOP: 3;
Projected date for first deployment: June 2004 Oakland ARTCC;
Projected date for last deployment: March 2006;
Estimated cost (F&E): $548.2 million.
Date: July 2004;
Number of facilities receiving ATOP: 3;
Projected date for first deployment: June 2004;
Projected date for last deployment: March 2006;
Estimated cost (F&E): $548.2 million.
Total change;
Number of facilities receiving ATOP: 0;
Projected date for first deployment: None;
Projected date for last deployment: None;
Estimated cost (F&E): $0.
Source: GAO presentation of FAA data:
[End of table]
Risks and Challenges:
Cost issues-Although the contractor's costs to develop ATOP have grown
by approximately $20 million, FAA is not responsible for these cost
increases because it has a fixed-price contract arrangement with the
contractor.
Schedule issues-ATOP achieved its initial operational capability
milestone of June 2004 but a more aggressive development schedule was
agreed to with the ATOP contractor to achieve this milestone by April
2003 or 14 months earlier. An ATOP Assessment Team determined that the
contractor could not achieve this earlier date due poor requirements
development, unrealistic schedule estimates, and inadequate evaluation
by the contractor of the software complexity. The development delay has
exacerbated the scheduled transition from the current oceanic system to
the ATOP and would cost an additional $4 million a year to operate and
maintain the old system until ATOP is fully operational. Program
officials told us they were not certain when the transition could be
achieved because several operational issues needed to be resolved
including ATOP operational trials sector by sector, training schedule,
and filling new controller positions, and budgetary allocations to
support these activities.
En Route Automation Modernization (ERAM):
Purpose and Status:
The En Route Automation Modernization (ERAM) program will enable air
traffic controllers to provide ATC services to users of en route
airspace (generally, high-altitude airspace at 10,000 feet or above)
Services provided to users include separation, routing, and advisory
services needed to meet FAA's mission of providing safe, efficient, and
reliable air traffic management. Specifically, ERAM is to replace the
hardware and software in the current en route Host computer system, the
direct-access radar channel, and associated infrastructure. This
replacement will result in the installation of new system en route
automation architecture at each air route traffic control center
(ARTCC). In concert with other en route programs, ERAM will modernize
the en route infrastructure to provide a supportable, open-standards-
based system that will be the basis for future capabilities and
enhancements. ERAM is to be deployed at 20 ARTCCs in the continental
United States.
FAA awarded a letter contract to Lockheed Martin in December 2002. To
date, ERAM has not breached any JRC cost or schedule parameters.
However, the ERAM program is highly software intensive, requiring the
writing of over 1 million lines of software code. In addition, Lockheed
Martin is behind schedule because of software design and production
control issues that Lockheed expects to resolve. Lockheed Martin
officials stated that it does not expect any downstream impact from the
current negative schedule variance of about $1 million.
Contractor: Lockheed Martin Transportation and Security Solutions.
Baseline Changes to ERAM Scope, Schedule, and Cost:
Date: June 2003;
Number of facilities receiving ERAM: 20;
Projected date for first deployment: December 2009;
Projected date for last deployment: December 2010;
Estimated cost (F&E and O&M): $3.649 billion.
Total change; None.
Source: GAO's presentation of FAA data:
[Need another set of data to determine any change.]
[End of table]
Risks and Challenges:
Software Issues - Software development is one of ERAM's major risk
items. The ERAM program is a high-risk effort because of its size and
the amount of software code - over 1 million lines of software code
expected. Lockheed Martin is experiencing cost variances because of
software engineering difficulties. According to its cost performance
report, software engineering costs are being hampered by lower
productivity than originally planned and by software code growth across
the program. However, according to FAA officials, these additional
software development costs can be easily absorbed within the
contractor's management reserve that is currently on the contract.
[End of section]
Appendix VI: Key Contributors:
GAO Contacts:
JayEtta Z. Hecker, (202) 512-2834;
Beverly L. Norwood, (202) 512-2834:
Staff Acknowledgments:
In addition to the individuals named above, Tamera Dorland, Elizabeth
Eisenstadt, Brandon Haller, Bert Japikse, Carolyn Kirby, Steve
Martinez, Richard Scott, Adam Vodraska, and Dale Yuge made key
contributions to this report.
(540072):
FOOTNOTES
[1] GAO, Major Management Challenges and Program Risks: Department of
Transportation, GAO-03-108, (Washington, D.C.: Jan. 2003).
[2] For example, Best Practices: Capturing Design and Manufacturing
Knowledge Early Improves Acquisition Outcomes, GAO-02-701,
(Washington, D.C.: July 15, 2002) and Best Practices: Better Matching
of Needs and Resources Will Lead to Better Weapon System Outcomes, GAO-
01-288, (Washington, D.C.: Mar. 8, 2001).
[3] In this report, we refer to both commercial product developments
and federal agency acquisitions as acquisitions.
[4] The Joint Resources Council is an executive body consisting of
associate and assistant administrators, acquisition executives, the
chief financial officer, the chief information officer, and legal
counsel. The council makes corporate-level decisions, including those
that determine whether an acquisition meets a mission need and should
proceed. The council also approves changes to a program's baseline,
budget submissions, and the National Airspace System's architecture
baseline.
[5] The term "federal acquisition system" is used to refer to the
various statutes and regulations that govern procurement practices by
federal government agencies--the controlling regulation is the FAR.
[6] Currently, the FAR applies to all federal executive agencies except
FAA and the Transportation Security Administration.
[7] The six sections provide an overview and address life-cycle
acquisition management, procurement, configuration management, real
property, and integrated logistic support. AMS also includes
implementing guidance, flow charts, handbooks, clauses, forms, and
other information that expands, illustrates, or supplements policy.
[8] A management process for establishing and maintaining the
consistency of a product's performance and physical attributes with its
requirements, design, and operational information throughout its life.
[9] Integrated logistics support (ILS) is a critical functional
discipline that establishes and maintains a support system for all FAA
products and services. Elements of ILS include spare parts, training,
supply support, manuals and documentation, maintenance, and repair.
[10] 48 C.F.R. pt. 7.
[11] 48 C.F.R. pt. 34.
[12] DOD's 5000 series consists of DOD Directive 5000.1, the Defense
Acquisition System, and DOD Instruction 5000.2, Operation of the
Defense Acquisition System.
[13] P. L. 103-62, 107 Stat. 285.
[14] P.L. 89-286, 79 Stat. 1034.
[15] 14 C.F.R. pt. 17.
[16] In two-step sealed bid procurements, the acquisition process is
divided into two parts. In the first step, proposals are solicited and
evaluated to determine their acceptability without evaluating price. In
the second step, offerors who submitted acceptable step-one proposals
compete for award on the basis of price. Two-phase design-build
selection procedures are a selection method in which a limited number
of offerors is selected during the first phase (design) to submit
detailed proposals for the second phase (construction).
[17] 48 C.F.R. § 2.101 (definition of "full and open competition").
[18] A rational basis for such action may be based on emergencies,
standardization, or that a source is the only source available to
satisfy the requirement within the time required, which are necessary
and important to support FAA's mission. The decision to contract with a
single source may be made as part of the overall program planning. The
rational basis must be documented and approved as a part of the
acquisition strategy paper, a procurement plan, or as a separate
document. The AMS states that if an acquisition strategy paper is not
required, and the service organization determines that a procurement
plan is unnecessary, an independent single-source justification should
be documented and endorsed by the service organization and approved by
the contracting officer.
[19] GAO, Rebuilding Iraq: Fiscal Year 2003 Contract Award Procedures
and Management Challenges, GAO-04-605, (Washington, D.C.: June 1,
2004).
[20] For example, GAO, Best Practices: Capturing Design and
Manufacturing Knowledge Early Improves Acquisition Outcomes, GAO-02-
701, (Washington, D.C.: July 15, 2002) and Best Practices: Better
Matching of Needs and Resources Will Lead to Better Weapon System
Outcomes, GAO-01-288, (Washington, D.C.: Mar. 8, 2001).
[21] GAO, Information Technology: FAA Has Many Investment Management
Capabilities in Place, but More Oversight of Operational Systems Is
Needed, GAO-04-822, (Washington, D.C.: Aug. 20, 2004).
[22] Performance deficiencies in relation to the final requirements or
system specifications are used to assess whether the agency's goals
have been met. Such deficiencies may not degrade the mission standards
needed to ensure the safety and efficiency of the National Airspace
System.
[23] Free Flight Phase 1, completed in 2002, provided new information-
exchange systems and automated controller tools.
[24] We reported in August 2004 that FAA had implemented sound
requirements development and management practices on four other
systems, but noted that process improvement initiatives such as these
were not institutionalized across the agency. See GAO, Air Traffic
Control: System Management Capabilities Improved, but More Can Be Done
to Institutionalize Improvements, GAO-04-901, (Washington, D.C.: Aug.
20, 2004).
[25] Available software refers to commercial-off-the-shelf (COTS) and/
or nondevelopmental items (NDI). AMS defines COTS as a product or
service that has been developed for sale, lease, or license to the
general public. The product is currently available at a fair market
value. AMS defines an NDI as an item that was previously developed for
use by a government (federal, state, local, or foreign) and requires
limited further development. For example, the Army's SINCGARS radio is
the core of FAA's NEXCOM radio, and the software FAA selected for ATOP
was NDI software from the New Zealand air system.
[26] ATOP is under a fixed-price contract, but the contractor has
experienced over $20 million in cost overruns during the development
phase. Also, FAA renegotiated the terms of the contract to ensure that
the initial software development phase, known as build 1, would meet
its June 2004 Initial Operating Capability milestone.
[27] AMS does not call for critical design reviews, but they can be
done at the program's discretion.
[28] GAO, Information Technology: FAA Has Many Investment Management
Capabilities in Place, but More Oversight of Operational Systems Is
Needed , GAO-04-822, (Washington, D.C.: Aug. 20, 2004).
[29] FAA is organized into five business units that include: Airports;
Regulation and Certification; Commercial Space Transportation; the
Office of Security and Hazardous Materials; and the Air Traffic
Organization.
[30] Merging the former Air Traffic Services and the Research and
Acquisitions organizations formed the ATO; individual organizations
within FAA sponsor specific acquisitions to meet identified needs
(e.g., controller workstations and radars).
[31] The CIO is not part of the ATO; however, the CIO's efforts to
improve FAA's acquisition and management of software for major ATC
systems are directly related to the ATO's efforts to improve the
agency's acquisition of major ATC systems.
[32] GAO, Air Traffic Control: System Management Capabilities Improved,
but More Can Be Done to Institutionalize Improvements, GAO-04-901,
(Washington, D.C.: Aug. 20, 2004).
[33] GAO, Air Traffic Control: FAA's Modernization Investment
Management Approach Could be Strengthened, GAO/RCED/AIMD-99-88,
(Washington, D.C.: April 30, 1999).
[34] GAO, Acquisition Management: Agencies Can Improve Training on New
Initiatives, GAO-03-281, (Washington, D.C.: Jan. 15, 2003).
[35] In October 1997, FAA created the Intellectual Capital Investment
Plan Council to address the development needs of staff in its research
and acquisition organization. The council is made up of directors and
deputy directors from the agency's acquisition and research programs.
[36] Capital Planning and Investment Control is a disciplined process
that links planning to budgeting to procurement to operations,
maintenance, and management.
[37] Earned value management compares the actual work performed at
certain stages of a job to its actual costs--rather than comparing
budgeted and actual costs, the traditional management approach to
assessing progress. By measuring the value of the work that has been
completed at certain stages in a job, earned value management can alert
program managers, contractors, and administrators to potential cost
overruns and schedule delays before they occur and to problems that
need correcting before they worsen.
[38] Through the purchase card program, agency personnel can acquire
the routine goods and services they need directly from vendors as long
as the purchase is $2,500 or less.
[39] GAO, Best Practices: Using Spend Analysis to Help Agencies Take a
More Strategic Approach to Procurement, GAO-04-870, (Washington, D.C.:
Sept. 16, 2004); Best Practices: Improved Knowledge of DOD Service
Contracts Could Reveal Significant Savings, GAO-03-661, (Washington,
D.C.: June 9, 2003); and Best Practices: Taking a Strategic Approach
Could Improve DOD's Acquisition of Services, GAO-02-230, (Washington,
D.C.: Jan. 18, 2002).
[40] GAO, Contract Management: Agencies Can Achieve Significant Savings
On Purchase Card Buys, GAO-04-430, (Washington, D.C.: Mar. 12, 2004).
[41] GAO, Best Practices: Using Spend Analysis to Help Agencies Take a
More Strategic Approach to Procurement, GAO-04-870, (Washington, D.C.:
Sept. 16, 2004).
[42] P. L. 104-50, § 348, 109 Stat 436 (1995). Included in the United
States Code as 49 U.S.C. § 40110.
[43] 41 U.S.C. Ch. 4.
[44] 41 U.S.C. Ch. 7.
[45] 15 U.S.C. Ch. 14A.
[46] 14 C.F.R. pt. 17.
[47] AMS views FAA as consisting of numerous service-level
organizations, which in turn are organizational subunits that deliver
services within FAA, to industry or to the public, including technical
as well as nontechnical service providers.
[48] Investment analysis also includes identifying and analyzing
alternatives; developing life-cycle cost estimates; assessing net
present value, return on investment, and benefits; assessing
affordability; analyzing risk; evaluating the impact of an alternative
on enterprise architecture; and planning for deployment and
implementation.
[49] 48 C.F.R. pt. 7.
[50] 48 C.F.R. pt. 34.
[51] DOD's 5000 series consists of DOD Directive 5000.1, The Defense
Acquisition System and DOD Instruction 5000.2, Operation of the Defense
Acquisition System.
[52] Section 5 of AMS focuses on the acquisition of real property, a
subject that is also not covered by FAR.
[53] P. L. 103-62; 107 Stat. 285.
[54] 14 C.F.R. pt. 17.
[55] And up to $10,000,000 under limited special circumstances.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
