Information Technology
FAA Has Many Investment Management Capabilities in Place, but More Oversight of Operational Systems Is Needed Gao ID: GAO-04-822 August 20, 2004The Federal Aviation Administration's (FAA) mission is to promote the safe, orderly, and expeditious flow of air traffic in the United States airspace system, commonly referred to as the National Airspace System (NAS). To maintain its ability to effectively carry out this mission FAA embarked, in 1981,on a multi-billion dollar effort to modernize its aging air traffic control (ATC) system, the principle technology component of the NAS. Yet the NAS modernization has continued to be plagued by cost increases, schedule delays, and performance shortfalls. To gain insight into how FAA is meeting its management challenges, congressional requesters asked GAO to evaluate FAA's processes for making IT investment management decisions. The objectives of this review included (1) evaluating FAA's capabilities for managing its IT investments and (2) determining what plans, if any, the agency might have for improving these capabilities.
Judged against the criteria of GAO's framework for information technology investment management (ITIM), which measures the maturity of an organization's investment management processes, FAA has established about 80 percent of the basic selection and control practices that it needs to manage its mission-critical investments. For example, business lines actively monitor projects throughout their life cycles. However, the agency's senior IT investment board does not regularly review investments that are in the "in-service management," or operational, phase of their life cycles, and this creates a weakness in FAA's ability to oversee more than $1 billion of its IT investments. In addition, the agency has not yet established the key practices that would allow it to manage all of its investments as one portfolio--an integrated set of competing options. Until FAA has established the practices that would enable it to effectively manage its annual IT budget of about $2.5 billion, agency executives lack assurance that they are selecting and managing the mix of investments that best meets the agency's needs and priorities. The agency has initiated efforts to improve its investment management processes, but it has not yet developed and implemented a comprehensive plan--supported by management--to guide all of its improvement efforts. Such a plan is crucial in helping FAA to coordinate and prioritize its improvement efforts and sustain its commitment to the efforts it already has under way. Without such a plan--and controls for implementing it--FAA will be unlikely to develop a mature investment management capability.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-04-822, Information Technology: FAA Has Many Investment Management Capabilities in Place, but More Oversight of Operational Systems Is Needed
This is the accessible text file for GAO report number GAO-04-822
entitled 'Information Technology: FAA Has Many Investment Management
Capabilities in Place, but More Oversight of Operational Systems Is
Needed' which was released on September 20, 2004.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
GAO Highlights:
Highlights of GAO-04-822, a report to congressional requesters
Report to Congressional Requesters:
August 2004:
INFORMATION TECHNOLOGY:
FAA Has Many Investment Management Capabilities in Place, but More
Oversight of Operational Systems Is Needed:
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-822]:
Why GAO Did This Study:
The Federal Aviation Administration‘s (FAA) mission is to promote the
safe, orderly, and expeditious flow of air traffic in the United States
airspace system, commonly referred to as the National Airspace System
(NAS). To maintain its ability to effectively carry out this mission
FAA embarked, in 1981,on a multi-billion dollar effort to modernize
its aging air traffic control (ATC) system, the principle technology
component of the NAS. Yet the NAS modernization has continued to be
plagued by cost increases, schedule delays, and performance shortfalls.
To gain insight into how FAA is meeting its management challenges,
congressional requesters asked GAO to evaluate FAA‘s processes for
making IT investment management decisions. The objectives of this
review included (1) evaluating FAA‘s capabilities for managing its IT
investments and (2) determining what plans, if any, the agency might
have for improving these capabilities.
What GAO Found:
Judged against the criteria of GAO‘s framework for information
technology investment management (ITIM), which measures the maturity
of an organization‘s investment management processes, FAA has
established about 80 percent of the basic selection and control
practices that it needs to manage its mission-critical investments
(see table below). For example, business lines actively monitor
projects throughout their life cycles. However, the agency‘s senior IT
investment board does not regularly review investments that are in the
’in-service management,“ or operational, phase of their life cycles,
and this creates a weakness in FAA‘s ability to oversee more than $1
billion of its IT investments. In addition, the agency has not yet
established the key practices that would allow it to manage all of its
investments as one portfolio”an integrated set of competing options.
Until FAA has established the practices that would enable it to
effectively manage its annual IT budget of about $2.5 billion, agency
executives lack assurance that they are selecting and managing the mix
of investments that best meets the agency‘s needs and priorities.
The agency has initiated efforts to improve its investment management
processes, but it has not yet developed and implemented a
comprehensive plan”supported by management”to guide all of its
improvement efforts. Such a plan is crucial in helping FAA to
coordinate and prioritize its improvement efforts and sustain its
commitment to the efforts it already has under way. Without such a
plan”and controls for implementing it”FAA will be unlikely to develop
a mature investment management capability.
Summary of Results for Foundational Critical Processes and Key
Practices:
[See PDF for table]
Source: GAO.
[End of table]
What GAO Recommends:
To strengthen FAA‘s investment management capability, GAO recommends
that FAA develop and implement a plan to address the weaknesses
identified in this report. In commenting on a draft of this report,
the Department of Transportation commented that the report was
balanced and fair, showing where FAA has many capabilities in place
and identifying areas that need improvement.
www.gao.gov/cgi-bin/getrpt?GAO-04-822.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact David Powner, 202-512-
9286, pownerd@gao.gov or Lester Diamond, 202-512-7957, diamondl@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
FAA Has Established an IT Management Structure to Manage Its NAS
Investments:
FAA Has Initiated Efforts to Improve Its Investment Management Process:
DOT is Taking Steps to Integrate Oversight of FAA's IT Investments:
Conclusions:
Recommendations for Executive Action:
Agency Comments:
Appendixes:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Investment Management Process Used by Some Organizational
Units to Manage Non-NAS Investments:
Tables:
Table 1: NAS and Non-NAS IT Investments:
Table 2: Stage 2 Critical Processes--Building the Investment
Foundation:
Table 3: Summary of Results for Stage 2 Critical Processes and Key
Practices for NAS Investments:
Table 4: Instituting the Investment Board:
Table 5: Meeting Business Needs:
Table 6: Selecting an Investment:
Table 7: Providing Investment Oversight:
Table 8: Capturing Investment Information:
Table 9: Stage 3 Critical Processes--Developing a Complete Investment
Portfolio:
Table 10: Status of Stage 3 Critical Processes:
Figures:
Figure 1: FAA's Life Cycle Management Process:
Figure 2: Detailed Breakdown of FAA's Life Cycle Management Process:
Figure 3: The Five ITIM Stages of Maturity with Critical Processes:
Abbreviations:
ABA: Financial Services unit:
AHR: Human Resource Management unit:
AIO: Information Services unit:
AMS: Acquisition Management System:
APB: acquisition program baseline:
ARA: Research and Acquisition unit:
ARC: Region and Center Operations unit:
ATC: air traffic control:
ATO: Air Traffic Organization:
ATS: Air Traffic Services unit:
AVR: Regulation and Certification unit:
CIO: Chief Information Officer:
DOT: Department of Transportation:
ECG: En Route Communications Gateway:
F&E: facilities and equipment (predeployment stage of system life
cycle):
FAA: Federal Aviation Administration:
FTI: FAA Telecommunications Infrastructure:
IAT: Investment Analysis Team:
IT: information technology:
ITEB: Information Technology Executive Board:
ITIM: Information Technology Investment Management framework:
ITIPS: Information Technology Investment Portfolio System:
JRC: Joint Resources Council:
NAS: National Airspace System:
NSIP: NAS Support Integration Process:
OMB: Office of Management and Budget:
OPS: operations (postdeployment stage of system life cycle):
PIR: postimplementation review:
VCSU: VSCS Control Subsystem Upgrade:
VSCS: Voice Switching and Control System:
Letter August 20, 2004:
The Honorable Tom Davis:
Chairman:
Committee on Government Reform:
U.S. House of Representatives:
The Honorable Adam H. Putnam:
Chairman:
Subcommittee on Technology, Information Policy, Intergovernmental
Relations and the Census:
Committee on Government Reform:
U.S. House of Representatives:
The Federal Aviation Administration's (FAA) mission is to promote the
safe, orderly, and expeditious flow of air traffic in the United States
airspace system, commonly referred to as the National Airspace System
(NAS). To maintain its ability to effectively carry out this mission
FAA embarked, in 1981, on a multibillion dollar effort to modernize its
aging air traffic control (ATC) system, the principle technology
component of the NAS. Over the past 2 decades, individual FAA
modernization projects have experienced cost overruns, schedule delays,
and performance shortfalls of large proportions. Because of the size,
complexity, cost, and problem-plagued past of FAA's modernization
program, we have designated it a high-risk information technology
investment since 1995.[Footnote 1]
This report is one in a series of reports responding to your request to
evaluate FAA's efforts to address the information technology (IT)
management challenges it faces as it continues to modernize the ATC
system. It focuses on FAA's processes for making IT investment
management decisions and uses our Information Technology Investment
Management (ITIM) framework,[Footnote 2] which was released at a
hearing of the subcommittee on March 3, 2004. The framework provides a
method for evaluating and assessing how well an agency is selecting and
managing its IT resources. As agreed, our objectives were to
(1) evaluate FAA's capabilities for managing its IT investments,
(2) determine what plans the agency might have for improving these
capabilities, and (3) describe the Department of Transportation's (DOT)
oversight of FAA's investments and investment management process. To
address these objectives we analyzed documents and interviewed agency
officials to (1) validate and update FAA's self-assessments of the key
practices in the framework, (2) evaluate FAA's plans for improving its
capabilities, and (3) describe the department's oversight role. We
performed our work from October 2003, through July 2004, in accordance
with generally accepted government auditing standards. Appendix I
contains further details on our objectives, scope, and methodology.
Results in Brief:
FAA has established most--about 80 percent--of the basic practices
needed to manage its mission critical investments, including many of
the foundational practices for selecting and controlling IT
investments. These key practices provide additional assurance that the
investments selected will meet organizational needs and will be
completed on time and within budget. The practices also will enable the
agency to manage its IT investments as a portfolio, or integrated set
of competing options.
Even with these many capabilities in place, weaknesses remain in
several areas. Specifically, FAA:
* does not involve its senior IT investment board in regular reviews of
investments that have entered the in-service management phase, that is,
those systems that have completed development and become operational;
* does not have standard practices for managing its mission-support and
administrative investments;
* has not developed a process where the senior IT investment board
regularly reviews the full portfolio of investments; and:
* has not implemented postimplementation reviews of its major
investments to validate that they are providing the expected benefits
after they become operational.
FAA has begun to act to resolve the weaknesses described above, but
until FAA establishes the practices it needs to effectively manage its
IT investments, executives cannot be assured that they are selecting
and managing the mix of investments that best meets the agency's needs
and priorities. Establishing the capabilities needed to effectively
manage investments requires the development and implementation of a
plan, supported by management, that defines and prioritizes
improvements to the investment process. While FAA has initiated a
series of efforts to improve its investment management processes, it
does not have such a plan. Without this plan--and controls for
implementing it--it is unlikely that the agency will effectively
establish mature investment management capabilities.
The Department of Transportation has recently initiated several efforts
that can serve to provide better departmental oversight of FAA
investments. For example, DOT is finalizing capital investment guidance
for all of its operating administrations to follow in implementing
their investment management processes, and it has initiated a process
for reviewing the fiscal year 2006 budget justifications for major
programs, including those of FAA. The department has also identified
about a dozen programs it plans to monitor on a regular basis and has
asked FAA to report cost, schedule, and performance data on some of its
programs quarterly.
To further strengthen FAA's investment management capability, we are
recommending that the agency develop and implement a plan aimed at
addressing the weaknesses identified in this report.
In commenting on a draft of this report, the Department of
Transportation's Director of Audit Relations stated that GAO did a good
job of keeping the report balanced and fair, showing where FAA has many
capabilities in place and identifying areas that need improvement. The
agency also provided a technical comment, which we have incorporated
into the report.
Background:
FAA's Mission and Organizational Structure:
As an agency of the Department of Transportation, FAA's mission is to
promote the safe, orderly, and expeditious flow of air traffic in the
national airspace. To fulfill its mission requires the extensive use of
technology. The achievement of the agency's mission is also dependent
in large part on the skills and expertise of its workforce. Its
workforce of nearly 50,000 people provides aviation services that
include air traffic control; maintenance of air traffic control
equipment; and certification of aircraft, airline operations, and
pilots.
FAA is organized into several staff support offices (examples include
the Office of Information Services and the Office of Human Resource
Management) and five lines of business, which include Airports,
Regulation and Certification, Commercial Space Transportation, the
Office of Security and Hazardous Materials, and the newly formed Air
Traffic Organization (ATO). The ATO was formed on February 8, 2004, to
better provide safe, secure, and cost-effective air traffic services
now and into the future. The Air Traffic Services and the Research and
Acquisitions units, which had been primarily responsible for managing
air traffic services within FAA, were combined into one performance-
based organization to create ATO. ATO is led by FAA's Chief Operating
Officer and consists of 10 service units.[Footnote 3]
FAA's Use of IT:
FAA relies extensively on information technology to carry out its NAS
operations. It constantly depends on the adequacy and reliability of
the nation's ATC system, which comprises a vast network of radars;
automated data processing, navigation, and communications equipment;
and ATC facilities.[Footnote 4] Through this system, FAA provides
services such as controlling takeoffs and landings and managing the
flow of traffic between airports. For example, the Integrated Terminal
Weather System is employed to allow maximum use of airport runways in
all kinds of weather through a variety of weather sensors. The Wide
Area Augmentation System is used to provide vertically guided landing
to aircraft at thousands of airports and airstrips where there is
currently no vertically guided landing capability.
FAA also relies on IT to carry out its mission-support and
administrative operations (non-NAS operations). For example, FAA uses
IT to support accident and incident investigations, security
inspections, and personnel and payroll functions.
With an IT budget of about $2.5 billion for fiscal year 2004, FAA
accounts for over 90 percent of the Department of Transportation's IT
budget. The amount of investments in both NAS and non-NAS IT is shown
in the table 1 below.
Table 1: NAS and Non-NAS IT Investments:
Type of investment: NAS;
Funding of IT investments: Facilities and Equipment (F&E) (development
through 2 years of operations);
Total IT investment in fiscal year 2004 in billions of dollars: $1.464.
Type of investment: NAS;
Funding of IT investments: Operations (OPS) (through the rest of the
life cycle);
Total IT investment in fiscal year 2004 in billions of dollars: $0.834.
Type of investment: Non-NAS;
Funding of IT investments: Operations;
Total IT investment in fiscal year 2004 in billions of dollars:
$0.350-0.500[A].
Source: FAA.
[A] According to FAA, these numbers will be verified via baselining by
the end of Fiscal Year 2004 to reflect non-NAS IT assets and their
costs.
[End of table]
Prior Reviews Identified Weaknesses in the Agency's IT Investment
Management Process:
In 1995, we designated FAA's modernization of its air traffic control
system, the principle technology component of the NAS, as a high-risk
area because of the size and complexity of the program and FAA's many
failures in meeting projects' cost, schedule, and performance goals. In
our latest High-Risk Series, issued in January 2003,[Footnote 5] we
addressed the critical need for FAA to continue to improve its
investment management practices--the management processes the agency
uses to select, control and evaluate the benefits realized from its IT
spending--because the agency would be spending nearly $16 billion more
through FY 2007, after having already spent $35 billion since 1981.
Other reports have also noted weaknesses in FAA's IT investment
management processes and have made a number of recommendations to
address this area.[Footnote 6] For instance, last year we reported that
while FAA had improved its processes, several issues remained
unresolved. We noted, for example, that the agency had not yet
implemented processes for evaluating projects after implementing them,
in order to identify lessons learned and improve the investment
management process.
FAA's Current Approach to Investment Management:
FAA's process for managing an IT investment varies depending on the
type of investment--NAS systems in development through the second year
of operation (F&E), NAS systems in operation after the second year
(OPS), and non-NAS systems each follow different processes. NAS
investments are managed through a standardized process, the FAA
Acquisition Management System (AMS), and non-NAS investments are
managed through a number of different processes.
Process for Managing NAS Investments:
In April 1996, FAA implemented its AMS in response to legislation that
directed the agency to develop a new acquisition management
system.[Footnote 7] Because of FAA's contention that some of its
modernization problems were caused by federal acquisition regulations,
the Congress enacted legislation in November 1995 that exempted the
agency from most federal procurement laws and regulations and directed
FAA to develop and implement a new acquisition management system that
would address the unique needs of the agency. AMS was intended to
reduce the time and cost for fielding new products and services by
introducing (1) a new investment management system that spans the
entire life cycle of an acquisition, (2) a new procurement system that
provides flexibility in selecting and managing contractors, and
(3) organizational and human capital reforms that support the new
investment and procurement systems.
AMS provides high-level acquisition policy and guidance for selecting
and controlling FAA's NAS investments through all phases of the
acquisition life cycle, which is organized into a series of phases and
decision points that include (1) mission analysis, (2) investment
analysis, (3) solution implementation, and (4) in-service management.
To select investments, FAA has established two processes--mission
analysis and investment analysis--which together constitute a set of
policies, procedures, and guidance that enhance the agency's ability to
screen projects that are submitted for funding. Also, through these two
processes FAA is to assess and rank each project based on its relative
costs, benefits, risks, and contribution to FAA's mission, and a
senior, corporate-level decision-making group selects projects for
funding. After a project has been selected, FAA officials are required
to formally establish the life cycle cost, schedule, benefits, and
performance baselines that are used to monitor the project's status
throughout the remaining phases of the acquisition management life
cycle. See figure 1 for a graphic depiction of FAA's life cycle
management process.
Figure 1: FAA's Life Cycle Management Process:
[See PDF for image]
Note: During the front end of the life cycle, research and system
analysis activities are undertaken to discover applications of new
technology for FAA's present services, explore new opportunities for
service delivery, solve problems within current operations, and define
requirements.
[End of figure]
Several groups are involved in managing FAA's NAS investments; they
perform functions from analysis of mission needs and alternative
investments through system development, implementation, operation,
and, ultimately, disposal. The roles and responsibilities of each group
are described below:
Joint Resources Council (JRC)--This board makes corporate-level
resource and investment decisions and establishes investment programs.
Members include Associate Administrators representing FAA's lines of
business, the FAA Acquisition Executive,[Footnote 8] the Chief
Financial Officer, the Chief Information Officer (CIO), and the
Assistant Administrators for System Safety, for Policy, Planning and
International Aviation, and for Region and Center Operations. The board
is supported by the JRC Secretariat Team, a group that facilitates the
board's processes by maintaining the meeting calendar and guidance
documents, developing records of decisions, and providing advisory and
liaison support to programs.
Systems Engineering/Operational Analysis Team--This team performs
affordability assessments for newly proposed investments and prepares
recommendations for the reprogramming of funds from lower priority
programs. It also prepares annual budget submissions for approval by
the JRC. This team is composed of representatives from each line of
business and from other functional disciplines and is chaired by the
Director, System Architecture and Investment Analysis.
Investment Analysis Team (IAT)--This team is assembled for a relatively
short period for each specific investment being considered, to conduct
the detailed analysis of alternatives that will lead to selecting and
recommending a preferred acquisition solution. The team draws experts
from the integrated product teams,[Footnote 9] the organizational unit
with the need, the investment analysis staff,[Footnote 10] and other
organizations.
Corporate Mission Analysis Organization--Performs agency-level mission
analysis and coordinates service area analysis, an activity that is
conducted during mission analysis to (1) identify capability shortfalls
for or in conjunction with service organizations,[Footnote 11] (2)
ensure alignment with agency strategic goals, and (3) eliminate
redundant activity, duplicate benefits, service gaps, and service
overlaps. It also develops and maintains standards and tools for
conducting service area analysis, and it assists service organizations
in establishing a service area analysis capability.
In addition to identifying the roles and responsibilities of the groups
involved in the management process, AMS provides guidance on the
documents and decisions that result from each of the life cycle phases.
For example, through the mission analysis phase, FAA identifies
critical needs that the agency must meet for improving the safety,
capacity, efficiency, and effectiveness of the NAS. Approval of a
mission need statement by the JRC signifies that the agency agrees that
the need is critical enough to proceed to the next phase--investment
analysis. During the investment analysis phase, the IAT is to analyze
and recommend a solution that best satisfies FAA's performance goals
and customer service needs. This team is then to rank each proposed
project based on a number of factors, including how well it meets
mission needs compared to other projects and whether it has a favorable
cost-benefit ratio. As part of the JRC selection process, the life
cycle cost, schedule, benefits, and performance baselines are
established in a formal document called the acquisition program
baseline (APB), which is designed to be used by program offices to
monitor a project's status in achieving those baselines throughout the
remaining phases of the acquisition management life cycle.
The solution implementation phase begins when the JRC approves and
funds a project, establishes its acquisition program baseline, and
authorizes the service organizations to implement and manage the
project over its life cycle. After the project has been implemented and
is in operation (FAA's in-service management phase), the service
organizations monitor and assess operational performance. Also during
this phase, the project is monitored to determine whether the current
capability satisfies the demand for services or whether another
solution offers the potential for improving safety or effectiveness or
for significantly lowering costs. If the current capability is lacking,
FAA initiates a process whereby the mission need would be revalidated
and the investment analysis process begun again, possibly leading to a
new investment decision. Figure 2 provides detail on the phases of
FAA's IT investment management process and decision points. The
highlighted decision points represent those for which the JRC must make
an approval decision before a project can move forward.
Senior executives have stated that with the reorganization of the ATO
in February 2004, discussions have been held about realigning the
investment management process to make the heads of the service units
responsible and accountable for managing programs' capital investments
and operating costs from inception to retirement. In the past, the
business units have been organized to manage either capital investments
or operating costs, but not both. These discussions have not yet led to
specific changes in FAA's investment management processes and
responsibilities.
Figure 2: Detailed Breakdown of FAA's Life Cycle Management Process:
[See PDF for image]
[A] Corporate requirements organization takes the lead in planning for
the concept and requirements definition phase of the life cycle
management process.
[B] In a February 2004 memorandum, FAA's Chief Operating Officer
assumed the in-service decision authority and stated that he would
delegate this responsibility to the vice presidents of the service
organizations, unless the JRC retained the in-service decision
authority. If the JRC retains this authority, it determines the in-
service decision authority at the time of the final investment
decision.
[End of figure]
Process for Managing Non-NAS Investments:
While the AMS was intended to apply to all FAA investment programs, it
has not been implemented for non-NAS investments. Each of the agency's
business line and staff offices that manage non-NAS
investments[Footnote 12] has implemented its own processes for managing
these investments. Examples of these various non-NAS investment
processes include the following:
* Regarding an investment management board structure, the Financial
Services staff office has an informal board consisting of the Chief
Financial Officer, Deputy Chief Financial Officer, and heads of offices
within Financial Services. The Financial Services life cycle process
guide directs the board's operations. In the Regulation and
Certification unit, the senior management team makes investment
management decisions with input from the Chief Information management
team. This unit is developing an IT investment management processes
guide, which is expected to be completed by the end of the fiscal year.
* When selecting investments, the Human Resource Management unit uses
its established annual budget formulation process, while the Region and
Center Operations unit is moving toward a new process whereby in order
to be selected investments need to demonstrate, at a minimum, that they
(1) are compliant with FAA's architecture, (2) have a business sponsor,
(3) have a solid business case, and (4) can be funded.
* In controlling investments, Information Services has developed
processes to monitor contract expenditures, and unit managers regularly
perform financial management reviews of the programs under their
purview, but there is no structured process for oversight of projects'
performance against expectations. In the Human Resource Management
unit, division managers hold quarterly reviews to assess projects'
progress in meeting cost and schedule expectations and aligning with
strategic goals.
Descriptions of the processes used by each of the units responsible for
managing non-NAS investments can be found in appendix II.
In January 2004, the FAA Administrator established the Information
Technology Executive Board (ITEB) to "strengthen FAA's ability to use
IT as an agencywide strategic asset" and "guide fundamental changes in
the governance of IT assets." Its charter calls for the ITEB to assume
responsibility for making investment decisions about non-NAS IT
investments. However, the ITEB has not yet implemented this aspect of
its charter. Therefore, at the current time there is no single board or
investment management process for non-NAS investments that would be
analogous to the JRC board and AMS process that are used for NAS
investments.
ITIM Maturity Framework:
The ITIM framework is a maturity model composed of five progressive
stages of maturity that an agency can achieve in its investment
management capabilities.[Footnote 13] It was developed on the basis of
our research into the IT investment management practices of leading
private-and public-sector organizations. The framework identifies
critical processes for making successful IT investments, organized into
the five increasingly mature stages. These maturity stages are
cumulative; that is, in order to attain a higher stage of maturity, the
agency must have institutionalized all of the requirements for all of
the lower stages, in addition to those for the higher stage.
The ITIM can be used both to assess the maturity of an agency's
investment management processes and as a tool for organizational
improvement. The overriding purpose of the framework is to encourage
investment processes that increase business value and mission
performance, reduce risk, and increase accountability and transparency
in the decision process. We have used the framework in several of our
evaluations,[Footnote 14] and a number of agencies have adopted it.
These agencies have used ITIM for purposes ranging from self-assessment
to redesign of their IT investment management processes.
ITIM's five maturity stages represent steps toward achieving stable and
mature processes for managing IT investments. Each stage builds on the
lower stages; the successful attainment of each stage leads to
improvement in the organization's ability to manage its investments.
With the exception of the first stage, each maturity stage is composed
of "critical processes" that must be implemented and institutionalized
in order for the organization to achieve that stage. These critical
processes are further broken down into key practices that describe the
types of activities that an organization should be performing to
successfully implement each critical process. An organization may be
performing key practices from more than one maturity stage at the same
time. This is not unusual, but efforts to improve investment management
capabilities should focus on becoming compliant with lower-stage
practices before addressing higher-stage practices.
Stage 2 of the ITIM framework encompasses building a sound investment
management process by establishing basic capabilities for selecting new
IT projects. It also involves developing the capability to control
projects so that they finish predictably within established cost and
schedule expectations and the capability to identify potential
exposures to risk and put in place strategies to mitigate that risk.
The basic selection processes established in Stage 2 lays the
foundation for more mature selection capabilities in Stage 3.
Stage 3 requires that an organization continually assess both proposed
and ongoing projects as parts of a complete investment portfolio--an
integrated and competing set of investment options. It focuses on
establishing a consistent, well-defined perspective on the IT
investment portfolio and maintaining mature, integrated selection (and
reselection), control, and evaluation processes, which are to be
evaluated during postimplementation reviews (PIR). This portfolio
perspective allows decision makers to consider the interaction among
investments and the contributions to organizational mission goals and
strategies that could be made by alternative portfolio selections,
rather than relying exclusively on the balance between the costs and
benefits of individual investments.
Stages 4 and 5 require the use of evaluation techniques to continuously
improve both the investment portfolio and investment processes in order
to better achieve strategic outcomes. At Stage 4 maturity an
organization has the capacity to conduct IT succession activities and
therefore can plan and implement the deselection of obsolete, high-
risk, or low-value IT investments. An organization with Stage 5
maturity conducts proactive monitoring for breakthrough information
technologies that will enable it to change and improve its business
performance. Organizations implementing Stages 2 and 3 have in place
the selection, control, and evaluation processes that are required by
the Clinger-Cohen Act. Stages 4 and 5 define key attributes that are
associated with the most capable organizations.
Figure 3 shows the five maturity stages and the critical processes
associated with each.
Figure 3: The Five ITIM Stages of Maturity with Critical Processes:
[See PDF for image]
[End of figure]
As defined by the model, each critical process consists of "key
practices" that must be executed to implement the critical process.
FAA Has Established Many Key Practices for Managing NAS Investments but
Lacks Oversight of Operational Systems:
In order to have the capabilities to effectively manage IT investments,
an agency should, at a minimum, (1) build an investment foundation by
putting basic, project-level control and selection practices in place
(Stage 2 capabilities) and (2) manage its projects as a portfolio of
investments, treating them as an integrated package of competing
investment options and pursuing those that best meet the strategic
goals, objectives, and mission of the agency; and it should also
conduct PIRs to maintain mature, integrated selection, control, and
evaluation processes (Stage 3 capabilities). In addition, an agency
would be well served by implementing capabilities for improving its
investment process through performance evaluations of its portfolio and
succession management of current investments (Stage 4 capabilities). In
order to develop the capabilities to effectively manage its
investments, FAA would, at minimum, need to implement Stage 2
capabilities for both its NAS and non-NAS investments and Stage 3
capabilities for its portfolio of investments.
FAA's investment management capabilities vary depending on whether an
investment is considered to be NAS or non-NAS. Specifically:
* For NAS investments, FAA has executed 30 of the 38 Stage 2 key
practices that are required to establish a foundation for investment
management maturity. For these investments, the agency has in place a
strong set of processes to support investment management, although the
JRC does not regularly review investments that have passed into the in-
service management phase (i.e., operational systems).
* For its non-NAS investments, the agency has not yet adequately
implemented a single management line of responsibility and the standard
processes needed to manage in a consistent manner. Although some
structured processes exist within individual business units, this lack
of consistency undermines the agency's maturity.
* In Stage 3, the lack of regular JRC oversight of operational systems
and the absence of a structured approach to managing non-NAS
investments prevent FAA from managing its investments as a portfolio
that includes all major NAS and non-NAS investments. In addition, the
agency is not conducting PIRs on its major investments.
* FAA has not executed any of the Stage 4 key practices for managing
the succession of its information systems, although the agency has
begun to address this weakness by defining procedures for retiring
investments in the AMS.
When FAA implements all of the key practices associated with building
the investment foundation and managing its investments as a portfolio,
the agency will have greater assurance that it has selected the mix of
investments that best supports its strategic goals and that it will be
able to manage the investments to successful completion.
FAA Has Established Much of the Foundation Needed to Manage Its NAS
Investments:
At the ITIM Stage 2 level of maturity, an organization has attained
repeatable, successful IT project-level investment control processes
and basic selection processes. Through these processes, the
organization can identify expectation gaps early and take appropriate
steps to address them. According to ITIM, critical processes at Stage 2
include (1) defining IT investment board[Footnote 15] operations,
(2) identifying the business needs for each IT investment,
(3) developing a basic process for selecting new IT proposals and
reselecting ongoing investments, (4) developing project-level
investment control processes, and (5) collecting information about
existing investments. Table 2 describes the purpose of each of the
Stage 2 critical processes.
Table 2: Stage 2 Critical Processes--Building the Investment
Foundation:
Critical process: Instituting the investment board;
Purpose: To define and establish an appropriate IT investment
management structure and the processes for selecting, controlling, and
evaluating IT investments.
Critical process: Meeting business needs;
Purpose: To ensure that IT projects and systems support the
organization's business needs and meets users' needs.
Critical process: Selecting an investment;
Purpose: To ensure that a well-defined and disciplined process is used
to select new IT proposals and reselect ongoing investments.
Critical process: Providing investment oversight;
Purpose: To review the progress of IT projects and systems, using pre-
defined criteria and checkpoints, in meeting cost, schedule, risk, and
benefit expectations and to take corrective action when these
expectations are not being met.
Critical process: Capturing investment information;
Purpose: To make available to decision makers information to evaluate
the impacts and opportunities created by proposed (or continuing) IT
investments.
Source: GAO.
[End of table]
To its credit, FAA has put in place about 80 percent of the key
practices associated with managing its NAS investments through the
Stage 2 critical processes. The agency has satisfied all of the key
practices associated with capturing investment information and most of
those associated with instituting the investment board, meeting
business needs, selecting an investment, and providing investment
oversight. Most of the weaknesses in these critical processes relate
to NAS investments in the in-service management phase. Table 3
summarizes the status of FAA's critical processes for Stage 2, showing
how many key practices FAA has executed in managing its NAS
investments.
Table 3: Summary of Results for Stage 2 Critical Processes and Key
Practices for NAS Investments:
Critical process: Instituting the investment board;
Key practices executed: 7;
Total required by critical process: 8;
Percentage of key practices executed: 88%.
Critical process: Meeting business needs;
Key practices executed: 6;
Total required by critical process: 7;
Percentage of key practices executed: 86%.
Critical process: Selecting an investment;
Key practices executed: 7;
Total required by critical process: 10;
Percentage of key practices executed: 70%.
Critical process: Providing investment oversight;
Key practices executed: 4;
Total required by critical process: 7;
Percentage of key practices executed: 57%.
Critical process: Capturing investment information;
Key practices executed: 6;
Total required by critical process: 6;
Percentage of key practices executed: 100%.
Critical process: Total;
Key practices executed: 30;
Total required by critical process: 38;
Percentage of key practices executed: 79%.
Source: GAO.
[End of table]
FAA Has Established an IT Management Structure to Manage Its NAS
Investments:
The establishment of decision-making bodies or boards is a key
component of the IT investment management process. At the Stage 2 level
of maturity, organizations define one or more boards, provide resources
to support their operations, and appoint members who have expertise in
both operational and technical aspects of proposed investments. The
boards operate according to a written IT investment process guide that
is tailored to the organization's unique characteristics, thus ensuring
that consistent and effective management practices are implemented
across the organization. Once board members are selected, the
organization ensures that they are knowledgeable about policies and
procedures for managing investments. Organizations at the Stage 2 level
of maturity also take steps to ensure that executives and line managers
support and carry out the decisions of the IT investment board.
According to ITIM, an IT investment management process guide should be
a key authoritative document that the organization uses to initiate and
manage IT investment processes and should provide a comprehensive
foundation for the policies and procedures that are developed for all
of the other related processes. (The complete list of key practices is
provided in table 4.)
FAA has executed 7 of the 8 key practices for this critical process.
For example, in 1996, Congress directed FAA to develop a new
acquisition management system as part of a broad mandate for
acquisition reform at the agency.[Footnote 16] In response, FAA
implemented AMS in April 1996. AMS establishes policy and guidance for
all aspects of the agency's acquisition life cycle and documents the
investment management process used for NAS investments. The agency
established the JRC as its corporate-level investment board for the NAS
investments. The JRC makes select and control decisions, including
corporate decisions on mission needs, acquisition investments, and
acquisition program baseline changes; it also reviews and recommends
approval of the agency's F&E budget submission.
The board is adequately resourced to support its operations. The JRC
Secretariat Team supports the board in such ways as developing and
updating guidance, scheduling meetings, and preparing and executing the
JRC readiness process. In addition, the Mission Analysis Steering
Group[Footnote 17] is responsible for assisting the board in
prioritizing mission needs, while the Systems Engineering/Operational
Analysis Team is to assist in addressing budget issues among
investments. The JRC consists of senior officials from both business
and IT areas, including the Chief Information Officer and the associate
administrators representing FAA lines of business. These members are to
exhibit the core competencies required by FAA in selecting executives
and in assessing executive training needs. In addition, the agency
offers a 3-day AMS overview course for all employees, including JRC
members. Although the board as an entity does not oversee the
development and maintenance of AMS, it is involved through FAA's
Acquisition System Advisory Group, which evaluates all proposed changes
to AMS. To ensure that the board's decisions are carried out, an
acquisition program baseline document is approved at the JRC final
investment decision point; this document identifies the capabilities,
benefits, costs, and schedule for the approved investment, which are
monitored by FAA through its variance reporting process.
Despite these strengths, FAA has not yet clearly defined the
relationship between the JRC and the newly formed ITEB. Although the
ITEB was established by the Administrator to function as the central
authority responsible for assuring that FAA IT investments are based on
sound business practices, FAA has not yet clearly delineated the
specific roles the ITEB is to play and the relationship it will have
with the JRC. This task has been assigned to the ITEB as a longer-range
initiative.
Table 4 shows the rating for each key practice required to implement
the critical process for instituting the investment board at the Stage
2 level of maturity. Each of the "Executed" ratings shown below
represents instances where, based on the evidence provided by FAA
officials, we concluded that the specific key practices were executed
by the organization.
Table 4: Instituting the Investment Board:
Type of practice: Organizational commitments;
Key Practice: 1. An enterprisewide IT investment board composed of
senior executives from IT and business units is responsible for
defining and implementing the organization's IT investment governance
process;
Rating: Executed;
Summary of evidence: The JRC, FAA's corporate-level investment board
for the NAS investments, is responsible for defining and implementing
the agency's IT investment governance process. It consists of the
agency's most senior executives, including the CIO, Chief Financial
Officer, FAA's Acquisition Executive, and Associate Administrators
from its lines of business.
Type of practice: Organizational commitments;
Key Practice: 2. The organization has a documented IT investment
process directing each investment board's operations;
Rating: Executed;
Summary of evidence: FAA's AMS sets forth acquisition policy and
processes for the JRC. Also, the board has established its own
guidance to implement AMS core JRC policy.
Type of practice: Prerequisites;
Key Practice: 1. Adequate resources, including people, funding, and
tools, are provided for supporting the operations of each IT investment
board;
Rating: Executed;
Summary of evidence: Adequate resources are provided to support the
board's operations. The JRC Secretariat Team provides such operations
support as developing and updating guidance, scheduling meetings, and
preparing and distributing records of decisions. Two other groups
support the JRC decision-making process. The Mission Analysis Steering
Group assists the board in ranking mission needs, while the Systems
Engineering/Operational Analysis Team provides assistance by performing
affordability assessments for the JRC when it is considering
alternatives during investment analysis.
Type of practice: Prerequisites;
Key Practice: 2. The board members understand the organization's IT
investment management policies and procedures and the tools and
techniques used in the board's decision-making process;
Rating: Executed;
Summary of evidence: JRC members are senior managers representing all
agency lines of business. They include the CIO, Chief Financial
Officer, and Associate Administrators representing FAA lines of
business such as Air Traffic Services. Core executive competencies
based on FAA's Executive Success Profile are used in selecting
executives and in assessing executive training needs. A 3-day AMS
overview course is also available.
Type of practice: Prerequisites;
Key Practice: 3. Each board's span of authority and responsibility is
defined to minimize overlaps or gaps among the boards;
Rating: Not executed;
Summary of evidence: The JRC is FAA's corporate-level investment board
for making decisions related to NAS investments. In January 2004, the
ITEB was established to oversee the governance of the agency's IT
assets. However, the ITEB has yet to take significant action on the
charge in its charter to clearly delineate the roles it is to play and
its relationship with the JRC.
Type of practice: Activities;
Key Practice: 1. The enterprisewide investment board has oversight
responsibilities for the development and maintenance of the
organization's documented IT investment process;
Rating: Executed;
Summary of evidence: Although the JRC does not directly oversee the
development and maintenance of the FAA's documented investment process,
it is involved in this process through FAA's Acquisition System
Advisory Group, which is a corporate crossfunctional body that
evaluates all proposed changes to AMS. Membership consists of
representatives from each line of business as well as the JRC
Secretariat Team. Policy changes that are endorsed by the Group are
presented, via the FAA Acquisition Executive, who is on the JRC, to
the Administrator for approval.
Type of practice: Activities;
Key Practice: 2. Each investment board operates in accordance with its
assigned authority and responsibility;
Rating: Executed;
Summary of evidence: The JRC is operating in accordance with its
assigned authority and responsibility as FAA's corporate-level
investment board for making decisions related to NAS investments. The
charter for the ITEB specifically indicates the ITEB's
responsibilities, including for making decisions for non-NAS IT
acquisitions.
Type of practice: Activities;
Key Practice: 3. The organization has established management controls
for ensuring that investment boards' decisions are carried out;
Rating: Executed;
Summary of evidence: FAA has controls for ensuring that the JRC's
investment decisions are carried out as approved. At the JRC final
investment decision point, an acquisition program baseline document is
finalized and approved, which represents the mutual agreement between
the JRC, the provider organization, and the user organization
concerning the expected capability, benefits, costs, and schedule for
the investment program. It also establishes performance metrics for
assessing the program's success.
Source: GAO.
[End of table]
FAA Has a Process for Ensuring That its Investments Support Business
Needs and Meet Users' Needs:
Defining business needs for each IT project helps to ensure that
projects and systems support the organization's business needs and meet
users' needs. This critical process ensures that a link exists between
the organization's business objectives and its IT management strategy.
According to ITIM, effectively meeting business needs requires, among
other things, (1) documenting business needs with stated goals and
objectives, (2) identifying specific users and other beneficiaries of
IT projects and systems, (3) providing adequate resources to ensure
that projects and systems support the organization's business needs and
meet users' needs, and (4) periodically evaluating the alignment of IT
projects and systems with the organization's strategic goals and
objectives. (The complete list of key practices is provided in table
5.)
FAA has in place 6 of the 7 key practices for meeting business needs.
The agency's AMS and mission analysis guidance calls for business needs
for both proposed and ongoing IT projects and systems to be identified
in the mission need statement developed during the mission analysis
phase. FAA also has detailed procedures for developing this document
that call for identifying business needs. Resources for ensuring that
IT projects and systems support the organization's business needs and
meet users' needs include service organizations, the Corporate Mission
Analysis Organization, the Mission Analysis Steering Group, and
detailed procedures and associated templates for developing mission
need statements. FAA's specific business mission, with stated goals and
objectives, is defined in the Federal Aviation Administration Flight
Plan for fiscal years 2004 through 2008.
Further, FAA defines and documents business needs for both proposed and
ongoing IT projects and identifies users and other beneficiaries during
its mission analysis activities. In addition, the AMS policy calls for
users to participate in project management throughout the FAA life
cycle management process. For the three projects we reviewed,[Footnote
18] we verified that business needs and specific users and other
beneficiaries were identified and documented in mission needs
statements as well as in other documents. In addition, users are
involved in project management throughout the life cycle of the
projects. For example, according to project officials, En Route
Communications Gateway (ECG) users participate in project meetings,
weekly integrated product team status meetings, and monthly En Route
domain national deployment teleconferences. FAA Telecommunications
Infrastructure's (FTI) end users are heavily involved in the
"operational test" period, which determines whether the equipment can
be safely implemented in NAS. VSCS Control Subsystem Upgrade[Footnote
19] users are involved in the project's life cycle via a Web site
through which they review and comment on project documentation.
Despite these strengths, the JRC has no process for evaluating the
organizational alignment of NAS systems through most of their in-
service management phase (and non-NAS investments, which are described
separately in this report). While the JRC does evaluate the alignment
of projects and systems with organizational goals throughout the
systems' development and 2 years into their operations as part of the
annual budget formulation process, it does not use any consistent
process to review projects and systems after that point in their life
cycles. For NAS systems in the in-service management phase, these
activities are carried out within the business unit that owns the
system, but the JRC does not regularly oversee these processes and may
go for several years without reviewing a system's alignment with
organizational goals. In-service NAS systems only return to the JRC if
they are judged to require additional funds for correction. Until FAA
establishes a process for periodic evaluation of systems throughout the
in-service management phase and takes corrective actions when
misalignment occurs, the agency will not be able to ensure that these
projects, totaling about $1.3 billion per year, are still continuing to
maintain alignment with the FAA's strategic plans and its business
goals and objectives.
Table 5 shows the rating for each key practice required to implement
the critical process for meeting business needs at the Stage 2 level of
maturity and summarizes the evidence that supports these ratings.
Table 5: Meeting Business Needs:
Type of practice: Organizational commitments;
Key practice: 1. The organization has documented policies and
procedures for ensuring IT projects or systems that support the
organization's ongoing and future business needs;
Rating: Executed;
Summary of evidence: AMS and mission analysis guidance contain
documented policies and procedures for identifying the IT projects or
systems that support the organization's ongoing and future business
needs.
Type of practice: Prerequisites;
Key practice: 2. The organization has a documented business mission
with stated goals and objectives;
Rating: Executed;
Summary of evidence: The Federal Aviation Administration Flight Plan
(Strategic Plan) for fiscal years 2004 through 2008 defines the
agency's mission goals and objectives.
Type of practice: Prerequisites;
Key practice: 3. Adequate resources, including people, funding, and
tools, are provided for ensuring that IT projects and systems support
the organization's business needs and meet users' needs;
Rating: Executed;
Summary of evidence: FAA has adequate resources for ensuring that its
IT projects and systems support the organization's business needs and
meet users' needs. They include service organizations, the Mission
Analysis Steering Group, and the Corporate Mission Analysis
Organization. FAA also has detailed procedures and associated templates
for developing mission need statements.
Type of practice: Activities;
Key practice: The organization defines and documents business needs for
both proposed and ongoing IT projects and systems;
Rating: Executed;
Summary of evidence: AMS policy calls for business needs for both
proposed and ongoing IT projects and systems to be specified in the
mission need statement. We verified that business needs were defined
and documented in mission need statements for the three projects we
reviewed.
Type of practice: Activities;
Key practice: The organization identifies specific users and other
beneficiaries of IT projects and systems;
Rating: Executed;
Summary of evidence: FAA policy and procedures call for specific users
and other beneficiaries of IT projects and systems to be identified. We
verified that specific users and other beneficiaries were identified
for the three projects we reviewed.
Type of practice: Activities;
Key practice: 3. Users participate in project management throughout an
IT project's or system's life cycle;
Rating: Executed;
Summary of evidence: FAA policies and procedures call for users to
participate in project management throughout an IT project's or
system's life cycle. We verified that users participated in project
management throughout the life cycle of the three projects we reviewed.
Type of practice: Activities;
Key practice: 4. The investment board periodically evaluates the
alignment of its IT projects and systems with the organization's
strategic goals and objectives and takes corrective actions when
misalignment occurs;
The investment board periodically evaluates the alignment of its IT
projects and systems with the organization's strategic goals and
objectives and takes corrective actions when misalignment occurs;
Rating: Not executed;
Summary of evidence: The JRC evaluates the alignment of systems through
development and 2 years into operations with the organization's
strategic goals and objectives--through the annual budget formulation
process--and takes corrective actions when misalignment occurs.
However, there is no process for the JRC to periodically evaluate the
alignment of investments later in their life cycles.
Source: GAO.
[End of table]
FAA Has a Disciplined Process for Selecting New IT Proposals but Lacks
a Similar Process for Reselecting Ongoing Investments:
Selecting new IT proposals and reselecting ongoing investments requires
a well-defined and disciplined process to provide the agency's
investment board, business units, and developers with a common
understanding of the process and the cost, benefit, schedule, and risk
criteria that will be used both to select new projects and to reselect
ongoing projects for continued funding. According to ITIM, this
critical process requires, among other things, (1) making funding
decisions for new proposals according to an established process; (2)
providing adequate resources for investment selection activities; (3)
using a defined selection process to select new investments and
reselect ongoing investments; (4) establishing criteria for analyzing,
prioritizing, and selecting new IT investments and for reselecting
ongoing investments; and (5) creating a process for ensuring that the
criteria change as organizational objectives change. (The complete list
of key practices is provided in table 6.):
FAA has executed 7 of the 10 key practices associated with selecting an
investment. For example, the AMS establishes two processes--mission
analysis and investment analysis--that together constitute a set of
policies and procedures, as well as guidance that is designed to
enhance the agency's ability to select investments. In addition, FAA
has policies and procedures for its annual F&E budget formulation
process to reselect ongoing IT projects. Also, FAA's AMS sets forth
policies and procedures for reselecting ongoing IT investments by
identifying their capability shortfalls and addressing them as new
investments.
The AMS also integrates funding with the process of selecting an
investment by requiring the Systems Engineering/Operational Analysis
Team to perform affordability assessments for new proposed investment
programs; it may recommend funding reallocations from lower priority
programs when an alternative solution cannot be funded within FAA
planning and budgeting baselines. This team also supports the JRC to
ensure that the executives' funding decisions are aligned with
selection decisions during the investment analysis activities.
Resources for proposal selection activities include the program
director, the Integrated Product Team, and the Investment Analysis
Team, as well as detailed procedures and a template that have been
defined for developing investment analysis reports. The investment
analysis reports identify the evaluation criteria used, the
alternatives analyzed, and the ranking of each alternative so that the
JRC can select the best overall solution identified in the mission need
statement. The criteria that were established during the initial
investment analysis phase are used by the Investment Analysis Team to
rank each proposed project on the basis of how well it meets the
agency's mission needs compared with other projects.
FAA uses the processes defined in the AMS for selecting new IT
investments. In addition, it uses two processes to reselect ongoing IT
investments. Specifically, the FAA uses its annual budget formulation
process for projects in development or in the first 2 years of
operations. It also uses the AMS process when a system's capability
shortfall is identified, and it treats the correction of the shortfall
as a new investment. The managers of the three projects we reviewed
confirmed that their projects were selected using the AMS process. One
project's officials stated that this included market, alternatives,
investment, and affordability analyses. The program managers also
stated that the annual F&E budget formulation process is used to
reselect their projects. These project officials also noted that if a
project is scheduled for a hardware replacement, a reselection is done.
The AMS process is followed to explore new alternatives and make sure
the replacement is in the best interest of the government.
Despite these strengths, FAA has not developed similarly strong
processes for NAS investments more than 2 years into their operations-
-those NAS systems that are in the in-service management phase. For
example, while FAA's F&E budget formulation process establishes
criteria for analyzing, prioritizing, and reselecting IT investments
for systems in development or up until 2 years into operations, neither
of the two processes used to reselect IT investments has established
criteria for investments beyond 2 years into operations. In addition,
while FAA uses its annual budget formulation process to reselect
projects that are part of the F&E budget, the agency does not have an
analogous reselection process as part of its operations budget
formulation. Until FAA establishes consistent criteria for reselecting
all of its IT investments, it will not be adequately assured that it is
consistently and objectively continuing to fund ongoing projects that
still meet the needs and priorities of the agency in a cost-effective
and risk-insured manner.
Table 6 shows the rating for each key practice required to implement
the critical process for selecting an investment at the Stage 2 level
of maturity and summarizes the evidence that supports these ratings.
Table 6: Selecting an Investment:
Type of practice: Organizational commitments;
Key practice: 1. The organization has documented policies and
procedures for selecting new IT proposals;
Rating: Executed;
Summary of evidence: FAA's AMS policy, mission analysis, and investment
analysis guidance has documented policies and procedures for selecting
new IT proposals.
Type of practice: Organizational commitments;
Key practice: 2. The organization has documented policies and
procedures for reselecting ongoing IT investments;
Rating: Executed;
Summary of evidence: FAA has documented policies and procedures for its
annual F&E budget formulation process, which is used to reselect
ongoing IT projects. In addition, FAA's AMS policy has documented
policies and procedures for reselecting ongoing IT investments
throughout the FAA's acquisition life cycle.
Type of practice: Organizational commitments;
Key practice: 3. The organization has documented policies and
procedures for integrating funding with the process of selecting an
investment;
Rating: Executed;
Summary of evidence: FAA's AMS and investment analysis guidance have
documented policies and procedures for integrating funding with the
process of selecting an investment.
Type of practice: Prerequisites;
Key practice: 1. Adequate resources, including people, funding, and
tools, are provided for identifying and selecting IT projects and
systems;
Rating: Executed;
Summary of evidence: Adequate resources are provided for identifying
and selecting IT projects and systems. They include the program
director, Integrated Product Teams, and the Investment Analysis Team.
FAA also has detailed procedures and associated templates for
developing investment analysis reports.
Type of practice: Prerequisites;
Key practice: 2. Criteria for analyzing, prioritizing, and selecting
new IT investment opportunities have been established;
Rating: Executed;
Summary of evidence: The Investment Analysis Team has established
criteria for analyzing, prioritizing, and selecting new IT investment
opportunities. The investment analysis report, which is submitted to
the JRC, identifies the evaluation criteria, the alternatives analyzed,
and the ranking for each alternative.
Type of practice: Prerequisites;
Key practice: 3. Criteria for analyzing, prioritizing, and
reselecting[A] IT investment opportunities have been established;
Rating: Not executed;
Summary of evidence: While FAA's F&E budget formulation process has
established criteria for analyzing, prioritizing, and reselecting IT
investments that are part of that budget, neither of the two processes
used to reselect IT investment opportunities has established criteria
for investments beyond 2 years into operations.
Type of practice: Prerequisites;
Key practice: 4. A mechanism exists to ensure that the criteria
continue to reflect organizational objectives;
Rating: Not executed;
Summary of evidence: While FAA ensures that the criteria continue to
reflect organizational objectives for selecting new IT investments,
there are no consistent criteria used by the JRC to reselect
investments more than 2 years into operations.
Type of practice: Activities;
Key practice: 1. The organization uses its defined selection process,
including predefined selection criteria, to select new IT investments;
Rating: Executed;
Summary of evidence: FAA uses the policies and procedures defined in
the AMS, mission analysis, and investment analysis guidance to select
new IT investments. We verified that the three projects we reviewed
were selected using the mission analysis and investment analysis
activities defined in the AMS, mission analysis and the investment
analysis guidance.
Type of practice: Activities;
Key practice: 2. The organization uses the defined selection process,
including predefined selection criteria, to reselect[A] ongoing IT
investments;
Rating: Not executed;
Summary of evidence: FAA has a process defined in the AMS to reselect
ongoing IT investments. It also uses its annual budget formulation
process to reselect projects that are part of the F&E budget. According
to the project managers of the three projects we reviewed, this budget
process is used to reselect their projects. However, FAA does not
consistently use these defined processes to reselect IT investments
more than 2 years into operations.
Type of practice: Activities;
Key practice: 3. Executives' funding decisions are aligned with
selection decisions;
Rating: Executed;
Summary of evidence: The Systems Engineering/Operational Analysis Team,
which is composed of representatives from FAA's service organizations,
supports the JRC in making funding decisions that are aligned with
selection decisions as part of FAA's investment analysis activities.
Source: GAO.
[A] According to the GAO ITIM framework, reselecting is the periodic
reconsideration of an investment's continuing value to the organization
and the decision to continue funding. It is a recurring process that
continues for as long as a project is receiving funding.
[End of table]
FAA Does Not Have a Process for Effectively Overseeing Investments in
All Phases of Their Life Cycles:
An organization should provide effective oversight for its IT projects
throughout all phases of their life cycles. Its investment board should
maintain adequate oversight and observe each project's performance and
progress toward predefined cost and schedule expectations as well as
each project's anticipated benefits and risk exposure. The investment
board should also employ early warning systems that enable it to take
corrective action at the first sign of cost, schedule, or performance
slippages. This board has ultimate responsibility for the activities
within this critical process. According to ITIM, effective project
oversight requires, among other things, (1) having written policies and
procedures for management oversight; (2) developing and maintaining an
approved management plan for each IT project; (3) making up-to-date
cost and schedule data for each project available to the oversight
boards; (4) having regular reviews by each investment board of each
project's performance against stated expectations; and (5) ensuring
that corrective actions for each underperforming project are
documented, agreed to, implemented, and tracked until the desired
outcome is achieved. (The complete list of key practices is provided in
table 7.):
FAA has in place 4 of the 7 key practices associated with effective
project oversight. The agency has developed written policies and
procedures for management oversight of its investments. These include
(1) AMS; (2) the integrated program plan, which is the detailed
planning document for all aspects of a program's implementation,
including program control; and (3) the Integrated Baseline
Establishment and Management Process document for reporting variances
from the performance expectations approved by the JRC in the
acquisition program baseline.
We verified that cost, schedule, benefit, and risk expectations were
documented in the acquisition program baseline and that the integrated
program plan contained details for project execution for En Route
Communications Gateway and FAA Telecommunications Infrastructure. For
the VSCS Control Subsystem Upgrade, performance expectations and
details on project execution were both captured in the integrated
program plan.[Footnote 20] In addition, the JRC Secretariat Team
maintains a tracking system for action items that are assigned during a
project's acquisition reviews, including the action to be taken, the
responsible FAA organization, and whether the underlying problem has
been resolved.
FAA has not established processes that bring investments before the JRC
for oversight on a regular basis. There is a process for reporting
variances from the performance expectations that were approved by the
JRC in the investment's acquisition program baseline. However, although
this process is carried out as part of the F&E budget formulation for
IT investments in development or less than 2 years into operations, it
is not being carried out for investments that are part of the
operations budget. Investments that are meeting performance
expectations may not return to the JRC for several years. FAA also
conducts acquisition reviews as a means for program offices to report
to agency executives on the status of investments compared to program
baselines. However, since program offices may select which investments
they wish to bring forward for review, many investments may never come
forward. Until FAA develops (1) procedures for reporting on an
investment throughout its entire acquisition life cycle and (2)
mechanisms for ensuring that all investments are reviewed regularly,
the agency is placing itself at risk that underperforming investments
will not be reported to the JRC in order for it to take appropriate
actions.
Table 7 shows the rating for each key practice that is required to
implement the critical process for project oversight at the Stage 2
level of maturity and summarizes the evidence that supports these
ratings.
Table 7: Providing Investment Oversight:
Type of practice: Organizational commitment;
Key practice: 1. The organization has documented policies and
procedures for management oversight of IT projects and systems;
Rating: Executed;
Summary of evidence: FAA has developed written policies and procedures
for management oversight of IT projects and systems. These include AMS,
the integrated program plan, and the Integrated Baseline Establishment
and Management Process document for reporting variances from the
performance expectations approved in the acquisition program baseline
for an investment program.
Type of practice: Prerequisites;
Key practice: 1. Adequate resources, including people, funding, and
tools, are provided for IT project oversight;
Rating: Executed;
Summary of evidence: FAA has adequate resources for providing IT
project oversight. The agency has staff for compiling monthly variance
reports submitted by the investment program areas, preparing quarterly
baseline variance reports for the JRC, and preparing semi-annual
baseline variance report for the FAA Administrator. An automated system
is used to facilitate the maintenance of information for these reports.
Type of practice: Prerequisites;
Key practice: 2. IT projects and systems, including those in steady
state (operations and maintenance), maintain approved project
management plans that include expected cost and schedule milestones and
measurable benefit and risk expectations;
Rating: Executed;
Summary of evidence: AMS policy calls for an acquisition program
baseline (APB) document and an integrated program plan to be available
at the JRC final investment decision point. The APB document serves as
the AMS cost/schedule/ technical performance/benefits/risks control
document. The integrated program plan specifies how the APB baselines
will be controlled and details the management, contracting, and
technical actions and activities to be performed in executing the
acquisition. We verified that cost, schedule, benefit, risk, and
performance expectations were documented in the APBs for ECG and FTI.
For VCSU, these expectations were documented in an integrated program
plan.
Type of practice: Activities;
Key practice: 1. Data on actual performance (including cost, schedule,
benefit, and risk performance) are provided to the appropriate IT
investment board;
Rating: Not executed;
Summary of evidence: FAA has established a process for reporting
variances to the JRC from the performance expectations that have been
approved by the JRC in the APB for an investment. This process is
carried out for IT investments that are part of the F&E budget, but it
is not being carried out for investments that are managed as part of
the operations budget.
Type of practice: Activities;
Key practice: 2. Using verified data, each investment board regularly
reviews the performance of IT projects and systems against stated
expectations;
Rating: Not executed;
Summary of evidence: FAA does not have a process that provides an
opportunity for the JRC to regularly review investment performance. It
has a process for conducting acquisition reviews where program offices
provide status information to agency executives on the progress of
investments against their acquisition program baselines. However, the
individual program offices choose which investments they want to
discuss at these reviews. Also, although the process for reporting
variances from the performance expectations approved by the JRC in the
acquisition program baseline is carried out for IT investments that are
part of the F&E budget, this process is not being carried out for
investment programs that are part of the operations budget, and it only
results in investments with variances to be reviewed.
Type of practice: Activities;
Key practice: 3. For each underperforming IT project or system,
appropriate actions are taken to correct or terminate the project or
system in accordance with defined criteria and the documented policies
and procedures for management oversight;
Rating: Not executed;
Summary of evidence: During acquisition reviews, action items are
identified for investment programs discussed, an organization assigned
responsibility to carry them out, and the items are tracked until the
appropriate action is taken, at which time they are closed out.
Similarly, variance reports are prepared quarterly for the JRC
identifying investments with a 10 percent or greater variance from the
established acquisition program baseline. An investment program is to
remain on the quarterly variance report until successful corrective
action is taken. However, FAA has no mechanism that provides assurance
that every program has an acquisition review regularly, since it is
left to the individual program offices to decide which programs they
want discussed at the reviews. Also, although variance reports are
prepared for IT investments that are part of the F&E budget, reports
are not prepared for investments that are part of the operations
budget.
Type of practice: Activities;
Key practice: 4. The investment board regularly tracks the
implementation of corrective actions for each underperforming project
until the actions are completed;
Rating: Executed;
Summary of evidence: The JRC's Secretariat Team maintains a tracking
system for action items assigned during a project's acquisition
reviews. This system identifies the action to be taken, what FAA
organization is to perform it, and whether it is open or closed.
Source: GAO.
[End of table]
FAA Has a Structured Process for Capturing Investment Information and
Using It to Support Investment Management:
To make good IT investment decisions, an organization must be able to
acquire pertinent information about each investment and store that
information in a retrievable format. During this critical process an
organization identifies its IT assets and creates a comprehensive
repository of investment information. This repository provides
information to investment decision makers to help them evaluate the
impacts and opportunities that would be created by proposed or
continuing investments. It can provide insights and trends about major
IT cost and management drivers. The repository can take many forms and
does not have to be centrally located, but the collection method should
identify each IT investment and its associated components. This
critical process may be satisfied by the information contained in the
organization's current enterprise architecture, augmented by
additional information--such as financial information and information
on risk and benefits--that the investment board may require to ensure
that informed decisions are being made. According to ITIM, effectively
managing this repository requires, among other things, (1) developing
written policies and procedures for identifying and collecting the
information, (2) assigning responsibility for ensuring that the
information being collected meets the needs of the investment
management process, (3) identifying IT projects and systems and
collecting relevant information to support decisions about them, and
(4) making the information easily accessible to decision makers and
others. (The complete list of key practices is provided in table 8.):
FAA's AMS guidance identifies specific information that is needed in
the investment management process, including information for its
investment analysis phase. FAA maintains a number of repositories of
relevant information, including its Simplified Program Information
Reporting & Evaluation database, which reports variances in cost,
schedule, performance, or benefits from an investment's approved
acquisition program baseline. The information that is collected is made
available to the JRC in several documents, including program plans and
the acquisition program baseline document. The JRC Secretariat Team
ensures that the investment board has all the relevant information it
needs for its decision-making process.
Table 8 shows the rating for each key practice required to implement
the critical process for capturing investment information at the Stage
2 level of maturity and summarizes the evidence that supports these
ratings.
Table 8: Capturing Investment Information:
Type of practice: Organizational commitments;
Key practice: 1. The organization has documented policies and
procedures for identifying and collecting information about IT projects
and systems to support the investment management process;
Rating: Executed;
Summary of evidence: FAA has developed policies and procedures for
identifying and collecting information to support the investment
management process. For example, AMS guidance indicates what
information is needed for the investment analysis phase of FAA's
investment management process.
Type of practice: Organizational commitments;
Key practice: 2. An official is assigned responsibility for ensuring
that the information collected during project and systems
identification meets the needs of the investment management process;
Rating: Executed;
Summary of evidence: AMS guidance specifies which officials are
responsible for approving the completion of reports containing
information prepared for the investment management process. The JRC
Secretariat Team ensures through its readiness process that all of the
necessary information is available to the JRC for its decision making.
Type of practice: Prerequisite;
Key practice: 1. Adequate resources, including people, funding, and
tools, are provided for identifying IT projects and systems and
collecting relevant investment information about them;
Rating: Executed;
Summary of evidence: FAA has adequate resources for meeting this key
practice. Several teams, including a mission analysis team and an
investment analysis team, collect the relevant investment information
needed by the JRC to make its decisions on which investments to
approve. The JRC Secretariat Team ensures that the JRC has all the
relevant information for its decision making.
Type of practice: Activities;
Key practice: 1. The organization's IT projects and systems are
identified, and specific information is collected to support decisions
about them;
Rating: Executed;
Summary of evidence: FAA maintains data relevant to the investment
management process in several sources, including its financial
management system, its Simplified Program Information Reporting &
Evaluation tool, and its Capital Investment Plan. Also, AMS guidance
identifies information to be collected for the investment management
process--including for the investment analysis phase--to aid the JRC in
its final investment decisions.
Type of practice: Activities;
Key practice: 2. The information that has been collected is easily
accessible and understandable to decision makers and others;
Rating: Executed;
Summary of evidence: Information collected for the JRC decision-making
process is compiled in documents such as detailed program plans,
acquisition program baselines, and investment analysis reports. The JRC
Secretariat Team ensures that the JRC has all the relevant information
for its decision making through its readiness process.
Type of practice: Activities;
Key practice: 3. The information repository is used by investment
decision makers and others to support investment management;
Rating: Executed;
Summary of evidence: The JRC Secretariat Team, through its JRC
readiness process, collects information for the JRC to use in its
decision-making process. The JRC also receives an investment analysis
report that contains all of the information that has been gathered
during investment analysis activities.
Source: GAO.
[End of table]
FAA Does Not Have Structured Processes to Manage Its Non-NAS
Investments:
FAA does not have a single set of processes for making consistent basic
selection and control decisions for its non-NAS investments (Stage 2
capabilities). As previously discussed in the background section of
this report, several business units within FAA make decisions about
non-NAS investments. We reviewed the investment management processes of
seven of these units--Information Services, Region and Center
Operations, Regulation and Certification, Financial Services, Research
and Acquisition, Air Traffic Services, and Human Resource Management.
Appendix II describes the investment management processes we found in
these units. The extent to which these processes comply with the ITIM
framework for Stage 2 varies considerably by business unit, and FAA
currently does not specify non-NAS investment management processes in a
coordinated manner. Since the ITIM framework calls for a consistent
investment management process, we assessed FAA's non-NAS investment
management capability at an aggregate level. That is, we assessed FAA's
capability to manage its non-NAS investments, not the capability of
each individual business unit. Even though individual business units
may have some of these processes in place, FAA as a whole has not yet
defined:
* an investment management structure that allows the agency to
consistently manage its non-NAS investments,
* a uniform process for ensuring that non-NAS investments are linked to
business needs and meet users' needs,
* a process for selecting new IT proposals and reselecting ongoing
investments,
* a single process for reviewing the progress of investments and taking
corrective action when performance expectations are not being met, or:
* a comprehensive inventory of project and system information to
support investment decisions.
According to FAA officials, the agency has not defined a coherent
investment management structure and a set of processes for non-NAS
investments in the past because many of these investments have not had
the agencywide impact of the NAS investments. However, because there is
now recognition that a disciplined approach to managing non-NAS
investments could help control FAA's IT assets and costs in general,
efforts are currently under way to address this weakness. As previously
discussed, an IT Executive Board (ITEB) has been chartered with
responsibility for, among other things, making decisions about non-NAS
IT investments, but it has not yet taken action on developing a
standard process. Until FAA fully establishes the consistent practices
it needs to make basic project selection and control decisions,
executives will be hampered in their ability to effectively manage non-
NAS investments and ultimately to find the opportunities to achieve the
cost savings they are seeking.
FAA Lacks Key Capabilities Needed to Manage All IT Investments as a
Portfolio and Does Not Conduct Postimplementation Reviews:
During Stage 3, the investment board enhances the IT investment
management process by developing a complete investment portfolio and
carrying out PIRs. An IT investment portfolio is an integrated,
agencywide collection of investments that are assessed and managed
collectively on the basis of common criteria. Managing investments
within the context of such a portfolio is a conscious, continuous, and
proactive approach to expending limited resources on an organization's
competing initiatives in light of the relative benefits expected from
these investments. Taking an agencywide perspective enables an
organization to consider its investments comprehensively, so that
collectively the investments optimally address the organization's
missions, strategic goals, and objectives. Managing IT investments with
a portfolio approach also allows an organization to determine
priorities and make decisions about which projects to fund, and
continue to fund, based on analyses of the relative organizational
value and risks of all projects, including projects that are proposed,
under development, and in operation. For an organization to reap the
full benefits of the portfolio process, it should collect all of its
investments into an enterprise-level portfolio that is overseen by its
senior investment board. Although investments may initially be selected
into subordinate portfolios--based on, for example, lines of business
or life cycle stages--and managed by subordinate investment boards,
they should ultimately be aggregated into this enterprise-level
portfolio.
The purpose of a PIR is to evaluate an investment after its development
has been completed (i.e., after its transition from the implementation
phase to the in-service management phase) in order to validate actual
investment results. This review is conducted to (1) examine differences
between estimated and actual investment costs and benefits and their
possible ramifications for unplanned funding needs in the future and
(2) extract "lessons learned" about the investment selection and
control processes that can be used as the basis for management
improvements. Similarly, PIRs should be conducted for investment
projects that were terminated before completion, to help to readily
identify potential management and process improvements.
According to ITIM, critical processes performed by Stage 3
organizations include (1) defining the portfolio criteria,
(2) creating the portfolio, (3) evaluating the portfolio, and
(4) conducting PIRs. Table 9 shows the purpose of each critical process
in Stage 3.
Table 9: Stage 3 Critical Processes--Developing a Complete Investment
Portfolio:
Critical process: Defining the portfolio criteria;
Purpose: To ensure that the organization develops and maintains IT
portfolio selection criteria that support its mission, organizational
strategies, and business priorities.
Critical process: Creating the portfolio;
Purpose: To ensure that IT investments are analyzed according to the
organization's portfolio selection criteria and that an optimal IT
investment portfolio with manageable risks and returns is selected and
funded.
Critical process: Evaluating the portfolio;
Purpose: To review the performance of the organization's investment
portfolio(s) at agreed-upon intervals and to adjust the allocation of
resources among investments as necessary.
Critical process: Conducting postimplementation reviews;
Purpose: To compare the results of recently implemented investments
with the expectations that were set for them and to develop a set of
lessons learned from these reviews.
Source: GAO.
[End of table]
FAA has executed only 1 of the 27 key practices associated with Stage 3
critical processes: it has a process for distributing portfolio
criteria to project management personnel and other stakeholders. The
remaining 26 key practices were not executed--primarily because FAA
does not involve the JRC in the regular oversight of non-NAS
investments or in NAS investments during their in-service management
phase, weaknesses that we noted in our assessment of Stage 2
requirements. Since Stage 3 requires an enterprisewide perspective, the
lack of oversight of these classes of investments precludes the
successful completion of most Stage 3 critical processes. In addition,
Stage 3 requires an enterprisewide perspective that FAA has not
adopted, which would enable the JRC to oversee all major IT
investments, regardless of life cycle phase or business unit. Although
it can be appropriate for FAA to manage its NAS, in-service NAS, and
non-NAS investments as separate subordinate portfolios--depending on
the successful execution of all Stage 2 key practices--its enterprise-
level portfolio should contain all major IT investments regardless of
life cycle stage or business line. In building this enterprise-level
portfolio, the JRC can choose whether to include specific investments
based on predetermined criteria, as described by the ITIM framework.
Until FAA fully implements the critical processes associated with
managing its investments as a complete portfolio, it will not have the
data or enterprisewide perspective it needs to make informed decisions
about all of its major IT investments.
In addition, FAA has not executed the six key practices for conducting
PIRs. In June 2004, in response to a recommendation contained in our
1999 report[Footnote 21] that FAA initiate PIRs for projects or
programs within 3 to 12 months of deployment or termination, the NAS
Configuration Management and Evaluation Staff developed a proposed
approach to PIRs, but this approach was not implemented. In November
2003, the life cycle management policy team proposed a change to the
AMS that would require conducting these reviews, but there has been no
action on the proposal. Although the JRC has recently reaffirmed its
commitment to implement PIRs, there is no policy and no established
process to carry them out. If PIRs are not conducted on a routine
basis, then FAA will not be able to effectively evaluate the results of
its IT investments; this will affect the agency's ability to determine
whether to continue, modify, or terminate an IT investment in order to
meet its stated mission objectives.
Table 10 summarizes the status of FAA's critical processes for Stage 3,
showing how many associated key practices it has executed.
Table 10: Status of Stage 3 Critical Processes:
Critical process: Defining the portfolio criteria;
Key practices executed: 1;
Total required by critical process: 7;
Percentage of key practices executed: 14.
Critical process: Creating the portfolio;
Key practices executed: 0;
Total required by critical process: 7;
Percentage of key practices executed: 0.
Critical process: Evaluating the portfolio;
Key practices executed: 0;
Total required by critical process: 7;
Percentage of key practices executed: 0.
Critical process: Conducting postimplementation reviews;
Key practices executed: 0;
Total required by critical process: 6;
Percentage of key practices executed: 0.
Totals;
Key practices executed: 1;
Total required by critical process: 27;
Percentage of key practices executed: 4.
Source: GAO.
[End of table]
FAA Has Not Established a Process for Managing the Succession of Its
Information Systems:
Once an agency has attained Stage 3 maturity, it evaluates its IT
investment processes and portfolios to identify opportunities for
improvement (Stage 4 capabilities). This entails (1) improving the
portfolio's performance and (2) managing systems and technology
succession. We did not assess FAA's capability for improving the
portfolio's performance, because it did not claim to be executing any
of the relevant key practices in its self-assessment.
According to ITIM, regarding system and technology succession
management includes (1) defining policies and procedures for managing
the IT succession process, (2) assigning responsibility for the IT
succession process, (3) developing criteria for identifying IT
investments that may meet succession status, and (4) periodically
analyzing IT investments to determine whether they are ready for
succession. This critical process enables an organization to recognize
low-value or high-cost IT investments and augments the routine
replacement of systems at the end of their useful lives. It also
promotes the development of a forward-looking, solution-oriented view
of IT investments that anticipates future resource requirements and
allows the organization to plan appropriately. This process differs
from the reselection activity in Stages 2 and 3 in that it focuses on
anticipating and planning for the retirement of legacy systems and on
meeting remaining requirements with other, perhaps new, systems. In
addition, succession management takes place at the end of a system's
life cycle.
FAA has not executed any of the nine key practices required to
implement this critical process. Although the agency has defined
procedures in AMS for retiring investments, it still needs to describe
how to regularly review systems that are in operations in order to
identify candidates for retirement. According to FAA, decisions on
succession are made by the service organizations. However, no
individual or group has been assigned responsibility for managing the
succession process from an enterprise perspective, which would allow
the FAA to better anticipate and plan for future resource requirements.
Without an institutionalized process for succession management, the FAA
may not be able to identify those IT investments that are eligible for
succession in enough time to minimize the effect of the transition on
their successors. In addition, by establishing an effective succession
management process, the agency can identify systems for retirement,
freeing resources for other, superior, investments.
FAA Has Initiated Efforts to Improve Its Investment Management Process:
We have previously reported that to effectively implement IT investment
management processes, organizations need to be guided by a plan that
(1) is based on an assessment of strengths and weaknesses;
(2) specifies measurable goals, objectives, and milestones;
(3) specifies needed resources; (4) assigns clear responsibility and
accountability for accomplishing tasks; and (5) is approved by senior
management.
FAA has begun to take steps to resolve some of the weaknesses
identified in this report. For example, at a June 10, 2004, meeting,
the JRC decided to incorporate budget justification documents
(Exhibit 300s), which are currently prepared for the Office of
Management and Budget (OMB) as part of the President's Budget
formulation process, into the AMS process for managing NAS investments.
The Exhibit 300 will become the board's decision-making document, and
essential information from existing AMS-required documents--the
investment management report, the acquisition strategy paper, the
integrated program plan, and the requirements documents--will be
incorporated into the Exhibit 300. The JRC also recently decided to
implement PIRs in order to track metrics during program implementation.
Finally, at that same meeting, the board decided to collectively
determine, at the meeting where the F&E budget is approved, which F&E
and OPS programs should be brought forward for review the following
year. This decision serves to bring certain investments in the
in-service management phase under the JRC's direct purview, although it
does not specify that consistent criteria be established, as the ITIM
framework requires.
FAA has also begun to initiate steps to bring more clarity to the
ITEB's responsibilities, although the specifics have yet to be defined.
In its charter, the ITEB is charged with making investment decisions
about non-NAS IT investments. This action would begin to bring all of
the non-NAS investments under a single authority. The charter suggests
that the ITEB choose among three options: (1) to send major non-NAS
investment decisions to the JRC, (2) to make the decision itself, given
an acceptable review process similar to the JRC processes, or (3) have
the CIO, Chief Financial Officer, and owning assistant/associate
administrator make the decision jointly. This description of the ITEB's
roles and responsibilities further alludes to the senior board's
evolving responsibility toward major non-NAS IT investments, although
it falls short of laying out specific criteria for selecting which
investments should be sent forward to the JRC. The ITEB has been given
responsibility for four short-term initiatives as well, including
establishing an agencywide cost control program for non-NAS
expenditures and ensuring that all OMB Exhibit 300s receive a passing
grade for the 2006 budget year. The ITEB has been charged with the
long-term initiative of clearly delineating the roles it plays and its
relationship with the more senior board. The successful completion of
this initiative is likely to satisfy the single key practice that FAA
has not yet executed in the Instituting the Investment Board critical
process of the ITIM.
The Chief Operating Officer's recent reorganization of the ATO is
intended to make the heads of the service units responsible for IT
projects from their inception through the in-service management phase.
This new organization is designed to support his expressed intentions
to increase accountability for systems in operation in order to manage
costs more effectively. According to the Chief Operating Officer, FAA
recognizes that good processes are needed for both NAS and non-NAS to
improve the way the agency manages its investments.
While FAA has initiated these improvement efforts, it has not linked
them together in a plan with the characteristics listed above that
would help coordinate and guide the efforts. Until FAA develops a plan
that would allow for the systematic prioritization, sequencing, and
evaluation of improvement efforts, the agency risks not being able to
effectively establish mature investment management processes.
DOT is Taking Steps to Integrate Oversight of FAA's IT Investments:
DOT has recently initiated several efforts that can serve to provide
better departmental oversight of FAA investments. This fiscal year DOT
and FAA reached an agreement by which DOT reviews FAA's Exhibit 300s as
part of the department's annual budget process, in which all
departmental components participate. Under this agreement, DOT conducts
a review of all FAA Exhibit 300s starting in June of each budget year
and culminating in the review of all Exhibit 300s by the Department
Investment Review Board in late August, prior to the submission of the
budget to OMB in September. As part of this agreement, DOT has outlined
a process and schedule for reviewing the fiscal year 2006 budget
justifications for major FAA programs and is monitoring FAA's progress
in meeting this schedule. In addition, the department has identified
about a dozen programs that it plans to monitor regularly and has begun
reviewing these programs through its senior investment management
decision-making board, on which the FAA Administrator is a voting
member. DOT has also requested that FAA set reasonable expectations for
cost, schedule, and performance for its major projects and that it then
report quarterly on variances to those expectations. FAA submitted its
first quarterly report as of June 2004. These regular reports are
intended to help DOT maintain oversight of FAA's processes and ensure
that they are appropriate and consistent with OMB's requirements.
Furthermore, the department is currently planning to issue an
investment management guide that specifies minimum expectations that
its operating administrations (including FAA) are to follow in managing
their investments. According to DOT officials, FAA has been complying
with the department's requests for information to facilitate its
oversight process.
Department officials are attributing their increased oversight--and
cooperation from FAA--to the fact that the department has recently
reinstituted its own investment management processes. In addition, DOT
officials said that FAA now understands the role the department can
play in helping it to obtain the funding it needs for its programs.
Conclusions:
FAA has established most of the project selection and control
capabilities needed to manage its NAS investments. This should help
provide the executive-level decision-making and oversight capabilities
required to establish accountability and guide major IT investments
through most of their life cycles. However, weaknesses remain. For
example, although business units are involved in the regular review of
investments throughout their life cycles, the JRC may not review the
performance of operations systems for several years unless they require
significant additional funds. Also, FAA has yet to define and implement
the practices it needs to select and control its non-NAS investments.
Ultimately, because the JRC does not regularly review NAS systems
during the in-service management phase and does not regularly review
the non-NAS systems in general, significant portions of FAA's
approximately $2.5 billion investment in IT go without top-level
executive oversight and are not viewed as part of an enterprisewide
portfolio. FAA has taken some initial steps to implement PIRs, but it
has not yet established a process to carry them out.
The agency has begun to take some steps to develop improvements to
address some of these weaknesses, such as establishing an Information
Technology Executive Board with relevant responsibilities. In addition,
the JRC has begun integrating some budgeting and oversight processes,
and the Chief Operating Officer has begun to articulate a vision that
includes additional accountability for investments in operations. But
FAA has not developed a comprehensive plan to guide all improvement
efforts. Such a plan would help coordinate and prioritize improvement
efforts and help sustain commitment to the efforts under way. The
increasing collaboration between FAA and DOT further contributes to the
likelihood that the management of FAA's investments will improve as
FAA's Exhibit 300s have the benefit of department-level review and the
departmental investment review board conducts periodic reviews of
selected projects.
Recommendations for Executive Action:
To strengthen FAA's investment management capability and address the
weaknesses discussed in this report, we recommend that the Secretary of
the Department of Transportation direct the FAA Administrator to
develop and implement a plan for improving FAA's IT investment
management processes. The plan should address the weaknesses described
in this report, beginning with those we identified in our Stage 2
analysis and continuing with those we identified in our Stage 3. The
plan should also draw together ongoing efforts as well as instituting
new initiatives where called for. The plan should, at a minimum,
provide for accomplishing the following:
In Stage 2:
* Define procedures for aligning the JRC and the newly established
ITEB.
* Establish a process for the JRC to periodically reevaluate the
alignment of projects in the in-service management phase with strategic
goals and objectives.
* Establish a process for the JRC to regularly review the performance
of IT systems throughout their life cycles and take corrective actions
when expected performance is not being met.
* Define and implement an IT investment management structure, including
an investment management board and a disciplined process for managing
all non-NAS investments.
In Stage 3:
* Define and implement processes for managing major investments as part
of an enterprise-level portfolio, including NAS F&E investments, NAS
investments in the in-service management phase, and non-NAS
investments.
* Define and implement processes for carrying out PIRs on investments
as they enter the in-service management stage.
In developing the plan, the FAA Administrator should ensure that it
(1) specifies measurable goals, objectives, and milestones;
(2) specifies needed resources; (3) assigns clear responsibility and
accountability for accomplishing tasks; and (4) is approved by senior
management. In implementing the plan, the FAA Administrator should
ensure that the needed resources are provided to carry out the plan and
that progress is measured and reported periodically to the Secretary of
Transportation.
Agency Comments:
In commenting on a draft of this report, DOT's Director of Audit
Relations stated via e-mail that DOT appreciated the opportunity to
review and offer comment on our report and that GAO had done a good job
keeping the report balanced and fair, showing where FAA has many
capabilities in place and identifying areas that need improvement. The
Director also provided a technical comment, which we have incorporated
into the report.
As agreed with your offices, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days
from the date of this letter. At that time, we will send copies to
other interested congressional committees, the Director of the Office
of Management and Budget, the Secretary of Transportation, FAA's
Administrator and CIO, and other interested parties. We also will make
copies available to others upon request. In addition, the report will
be available at no charge on the GAO Web site at
[Hyperlink, http://www.gao.gov].
Should you or your offices have questions on matters discussed in this
report, please contact me at (202) 512-9286 or Lester P. Diamond,
Assistant Director, at (202) 512-7957. We can also be reached by e-mail
at [Hyperlink, pownerd@gao.gov], or [Hyperlink, diamondl@gao.gov],
respectively. Key contributors to this report were William G. Barrick,
Niti Bery, Joanne Fiorino, Michael Giannone, Sabine R. Paul, and Nik
Rapelje.
Signed by:
David A. Powner:
Director, IT Management Issues:
[End of section]
Appendixes:
Appendix I: Objectives, Scope, and Methodology:
The objectives of our review were to (1) evaluate FAA's capabilities
for managing its IT investments, (2) determine what plans the agency
might have for improving these capabilities, and (3) describe how DOT
oversees FAA's investments and investment process. Because FAA told us
that it managed its NAS and non-NAS investments differently, we
performed separate assessments for the practices to evaluate FAA's
capabilities for managing IT investments.
To address the first objective, for the NAS investments we reviewed the
results of the agency's self-assessment of Stages 2, 3, and 4 practices
using GAO's ITIM framework[Footnote 22] and validated and updated the
results of the self-assessment through document reviews and interviews
with officials. We reviewed written policies, procedures, and guidance
and other documentation providing evidence of executed practices,
including FAA's Acquisition Management System guidance, mission
analysis and investment analysis guidance, and memorandums. We also
reviewed JRC guidance and records of decision, acquisition review
guidance and meeting minutes, and variance reporting procedures and
reports. We did not assess FAA's progress in establishing the
capabilities found in one of the two Stage 4 critical processes,
entitled Improving the Portfolio's Performance, or in any of the Stage
5 critical processes, because FAA acknowledged that it had not executed
any of the key practices in these critical processes. For the non-NAS
investments, we reviewed the results of FAA's self-assessments of
Stage 2 practices using GAO's ITIM framework and conducted interviews
to clarify and update the results. We did not perform a detailed
assessment of these practices because they most likely will be
superseded by a new process (when it is defined) for managing non-NAS
investments, and non-NAS investments are of lower cost and impact to
FAA.
As part of our analysis, we selected three IT projects as case studies
to verify that the critical processes and key practices were being
applied. We selected projects that (1) supported different FAA
functional areas, (2) were in different life cycle phases, and
(3) required different levels of funding. The three projects are
described below:
* FAA Telecommunications Infrastructure (FTI)--FTI is a performance-
based telecommunications services contract for voice, video, and data
point-to-point support for telecommunications for the National Airspace
System and its support system. It contributes to both the separation of
aircraft (the mission-support network) and other FAA uses (the
operational network, e.g., e-mail and phone). FTI will replace the
current telecom system. FTI will eliminate the need for other
subnetworks, of which there are currently eight or nine, and therefore
eliminate the management overhead associated with operating so many
networks. The integration of multiple networks and subnetworks will
provide a single source and single vehicle for telecom. FTI is in the
Technical Operations unit and has estimated life cycle costs of
$2 billion. The contract for FTI was awarded in June 2002.
* En Route Communications Gateway (ECG)--ECG is a mission critical
gateway, or interface, for data from radar sites to Air Route Traffic
Control Centers. ECG will serve as a single domain communications
gateway and will provide the path for exchanging flight plan data from
outside sources and transfer data among systems. ECG provides a
commercial-off-the-shelf nondevelopmental item digital gateway using a
modern, open and extensible platform consisting of modular scalable
hardware components. ECG will incorporate interface capability to
support legacy and future systems and will provide the capability to
transition to modern network communications and access more
surveillance sources. The flexibility provided by the ECG system
architecture will facilitate the evolution of the En Route domain
modernization. ECG will replace the Peripheral Adapter Module
Replacement Item system and provide a modern domain gateway that will
support the current and future En Route infrastructure. ECG is in the
En Route & Oceanic Service group and has estimated life cycle costs of
$442.5 million through September 2015.
* Voice Switching and Control System (VSCS)--In our review of the VSCS
program, we focused our review on one of VSCS's subcomponents, the VSCS
Control Subsystem Upgrade (VCSU). The VCSU program, part of the
Technical Operations Communications service group, is designed to
maintain overall supportability of VSCS[Footnote 23] by replacing the
hardware for the existing control subsystem, associated VSCS
operational and application software, required software licenses, and
supporting software and hardware documentation. Deliverables for the
VCSU program include all hardware, spare parts, software, software
licenses, system baseline documentation, training, and other technical
documentation necessary to support the product at 21 locations.
According to FAA, the VCSU program has a funding baseline of over $59
million and is in the operations and maintenance phase.
For these projects, we reviewed project management documentation, such
as mission needs statements, acquisition program baselines, and
integrated program plans. We also interviewed the project managers for
these projects.
We compared the evidence collected from our document reviews and
interviews to the key practices in ITIM. We rated the key practices as
"executed" on the basis of whether the agency demonstrated (by
providing evidence of performance) that it had met the criteria of the
key practice. A key practice was rated as "not executed" when we found
insufficient evidence of a practice during the review or when we
determined that there were significant weaknesses in FAA's execution of
the key practice.
To address our second objective, we obtained and evaluated documents
showing what management actions had been taken and what initiatives had
been planned by the agency. This documentation included JRC records of
decisions, the agency's capital investment guidance, and the recently
formed ITEB charter and meeting minutes. We also interviewed the Chief
Information Officer, other members of the JRC, and the Chief Operating
Officer to determine what efforts FAA had undertaken to improve IT
investment management processes.
To address our third objective, we reviewed documentation on DOT's
process for reviewing FAA's budget proposals and capital planning and
investment control reviews. We also conducted interviews with both FAA
and DOT officials, including DOT's CIO and Director for Capital
Planning and Investment Control to determine DOT's oversight role in
FAA's investments and investment management processes.
We conducted our work at FAA Headquarters in Washington, D.C., from
October 2003 through July 2004, in accordance with generally accepted
government auditing standards.
[End of section]
Appendix II: Investment Management Process Used by Some Organizational
Units to Manage Non-NAS Investments:
Financial Services (ABA):
Instituting the investment board; ABA has an investment board that
conducts periodic and monthly program reviews for all IT programs to
determine whether a program will be approved as an IT investment. A
life cycle process guide is now in place to direct the activities of
the investment board along with providing oversight of IT projects
within ABA.
Meeting business needs; The business needs of a project within ABA,
along with the dates for achieving them, need to be aligned with the
strategic goals established in the FAA Flight Plan. Projects or systems
that are no longer aligned with the Flight Plan will be decommissioned;
A project management plan identifies, among other things, the system's
users, customers, and types of services to be provided.
Selecting an investment; Selecting and reselecting an IT investment
within ABA involves both the executive management team and ABA's CIO
team. The executive management team reviews the business needs of the
investment and compares them against the ABA's IT budget, while ABA's
CIO team is involved with the selecting and reselecting processes by
analyzing the technical costs associated with the IT investment and
comparing those technical costs against the ABA's IT budget.
Providing investment oversight; ABA uses its life cycle process guide
to help manage its $25 million IT budget, which consists of 22 or 23
financial systems, 5 or 6 of them considered major programs under OMB's
definition of a major IT investment. A requirement of the life cycle
process guide is for every critical system in ABA to have a detailed
project management plan that addresses performance measures such as
cost, schedule, benefits, and risks; The day-to-day progress of IT
projects is tracked against critical milestones that have been already
established through weekly summary reviews with IT staff. For major IT
projects, biweekly meetings are conducted that address any concerns
with meeting the performance measures.
Capturing the investment information; ABA captures its IT asset
information using its Information Technology Investment Portfolio
System (ITIPS), which is available to all ABA management and system
support personnel. The information in ITIPS is used to manage projects
that are in production as well as ensuring that the life cycle
activities are in alignment with FAA's mission statements.
Research and Acquisitions (ARA):
Instituting the investment board; ARA uses its Operations Resource
Management Team guide to select, control, and evaluate ARA IT
investments. The team composed of representatives from ARA service
units. ARA investments are controlled and tracked through quarterly
reviews. These reviews look at the cost, schedule, and overall
performance of the investment.
Meeting business needs; The business needs for ARA investments need to
be mapped back to the Flight Plan. A monthly status review report is
prepared in order to ensure that the business needs are tracking back
to the Flight Plan.
Selecting an investment; ARA does not have any well-defined selection
criteria since each program uses its own configuration management plan.
ARA Ops build process guides the establishment of new projects.
Providing investment oversight; A project plan does exist, along with
established expenditures, which the program managers submit to the ARA
CIO on a monthly basis. These monthly status reports occur between the
CIO and the program managers to decide if an investment's resources,
such as funding, need to be reallocated. Once the CIO and program
manager decide that it is necessary for an investment's resources to be
reallocated, the CIO will discuss the need further with the Deputy
Associate Administrator for ARA, who ultimately will determine whether
a program will receive additional resources, such as funding; With
respect to the level of interaction that ARA has had with the JRC in
the past, only one program from ARA, NextGen, has gone before the JRC.
According to the ARA CIO, in order for a program to go to the JRC,
there must be justification made to the council that the program is
fully operational and is considered to be a benefit and a priority to
FAA. The ARA Deputy Associate Administrator will determine if a program
should go before the JRC for approval and funding.
Capturing investment information; The configuration control board uses
a database to capture asset inventory data about the systems that are
owned by the ARA CIO. According to the ARA CIO, in order for IT assets
to be effectively managed in ARA, there needs to be vision from AIO
about what programs to invest in over the next 5 years.
Air Traffic Services (ATS):
Instituting the investment board; The Information Resource Management
Executive Board is responsible for selecting, controlling, and
evaluating ATS IT investments.
Meeting business needs; Not all services within ATS have defined their
business needs. Even though ATS has the NAS Support Integration Process
(NSIP) data repository available for capturing IT asset information,
including business needs, and for defining system users, there is no
consistency in terms of the records being complete because there are
systems within ATS that have not registered with NSIP.
Selecting an investment; The ATS CIO manages the selection process,
which begins with the NSIP registration criteria.
Providing investment oversight; Each business unit within ATS has its
own project management plan and procedures. The day-to-day tracking of
projects as well as the monitoring of whether corrective actions are
being executed is also the responsibility of the individual business
units. Even though the individual business units are tasked with this
level of responsibility, the ATS CIO does play an oversight role by
setting the criteria and policies for the investments to be made for
the projects.
Capturing investment information; ATS uses the NSIP meta data
repository to collect any changes to the IT projects and systems by
providing a full declaration of the project or system. This includes
providing information to help ATS avoid unwanted costs due to systems
having redundant functionality and determining whether a system's or a
project's functions match the stated mission goals for ATS. NSIP also
handles the technical rollover for ATS systems or projects.
Information Services (AIO):
Instituting the investment board; AIO's investment management process
can be characterized as iterative and well managed, but undocumented.
The AIO Business Plan and IT Strategy are used to ensure that when
funds are appropriated and allocated that they map back to the Flight
Plan. Investments are controlled or tracked by the Deputy CIO on a
monthly basis to get an indication as to where the program is in the
process against the expenditures that have been already established.
Weekly meetings are held with the unit's CIO to discuss any issues
regarding AIO's investment management process.
Meeting business needs; AIO does not have any written policies or
procedures for identifying business needs for its IT projects. Only one
of its major projects, NAS Adaptation Service and Environment, has
documented its requirements, which includes specific users.
Selecting an investment; AIO uses an undocumented process for reviewing
new IT proposals to reach an agreement on selection.
Providing investment oversight; There are no AIO-wide policies or
procedures for managing projects or investment oversight. The
Information Technology Executive Board (ITEB)[A] has been formed to
provide a governing structure for non-NAS programs. One of the targets
for ITEB is to look at cost control and cross-cutting IT initiatives by
involving the heads of the lines of business. The ITEB is also going to
be involved with improving the scores on the Exhibit 300 business cases
for OMB.
Capturing the investment information; AIO uses ITIPS to track its asset
inventory and IT investments. The Deputy CIO of AIO is responsible for
ensuring that the inventory located in ITIPS meets the needs of AIO's
investment management process. According to AIO, the information within
ITIPS is updated at least twice a year.
Human Resource Management (AHR):
Instituting the investment board; AHR does not have an investment
board. Instead, AHR's senior management[B] is responsible for
selecting, controlling, and evaluating all IT investments by using
established agency acquisition policies and procedures to conduct
investment management decisions.
Meeting business needs; Business needs and specific users for each
project are identified within the project plan and are aligned with the
AHR Strategic Plan, the FAA Flight Plan, and the AIO Plan. AHR is also
aligning its business needs to the ITEB plans. Business needs are re-
evaluated on a quarterly basis to ensure that a project is aligned with
FAA's strategic goals and objectives.
Selecting an investment; AHR senior management uses its prioritization
process to evaluate and select investments for funding. The office and
center directors determine their requirements and then a budget request
is submitted for proposal funding. AHR receives an allowance amount
from the budget office. The first priority is to handle personnel
payments. The remaining balance is then redistributed to the business
divisions. The "building blocks" process starts at this point. This is
when base funding is reviewed to decide if a current investment needs
continued funding by asking questions about the importance of
continuing the funding of a particular project by looking at the
project activity and what the impact will be if this project is no
longer funded. Each division will submit a list of prioritized projects
with costs to the directorate. This list may exceed the budget level.
The directorate will reprioritize the original list.
Providing investment oversight; AHR has a Human Resource Management
Automation Plan that contains procedures for approving IT projects, and
describes the policies and procedures that AHR uses for project
management. Despite having project management policies and procedures,
not all projects within AHR have a formal project plan. The size and
scope of the project are two factors that help determine whether a
project has a formal project plan. AHR Division Managers ensure that
projects are on time by performing quarterly reviews that assess a
project's cost and schedule. AHR uses a color scheme (red, green, and
yellow) to indicate the schedule status of major milestones.
Capturing the investment information; AHR uses the ITIPS as its
inventory for making investment management decisions. AHR projects are
listed in ITIPS, along with business cases.
Regions and Center Operations (ARC):
Instituting the investment board; The IT Configuration Management Board
is ARC's investment review board. The board's charter has recently been
redone to provide more traceability back to the Flight Plan. The board
functions include evaluating potential IT investment options for ARC,
making recommendations on IT investment, establishing ARC-wide IT
standards, and developing and maintaining investment policies and
procedures. The board is led by the unit's CIO and includes four IT
managers from the regional offices and aeronautical center and two
members from the ARC Management Team. The ARC Management Team makes the
final-selection decisions. The IT investment management decisions are
then incorporated into the ARC Business Plan. The ARC unit is also
involved with cross-organizational investment decisions for FAA through
its membership on the FAA CIO Council.
Meeting business needs; Business needs are identified through entries
made in ITIPS, along with documentation from Exhibit 300s and Exhibit
53s.
Selecting an investment; ARC does not have its selection criteria
documented. To evaluate and select IT investments, the ARC IT
Configuration Management Board considers such things as benefits to ARC
across the regions, expected return on investment, technical
feasibility, and risk. The ARC business plan and the Flight Plan are
the documents that address these priorities.
Providing investment oversight; ARC does not have policies or
procedures for project management. Instead, ARC uses a weekly
teleconference to address expectations and progress of ARC-wide IT
initiatives at the IT manager level across ARC. According to the ARC
CIO, a second teleconference has been added to discuss portfolio
management--schedule, budget, training, and deployment along with
whether the project will be integrated with other lines of business.
Capturing the investment information; ARC uses ITIPS as its
standardized repository for collecting asset information that will be
useful for ARC's IT investment management decisions by providing
information about what types of systems and functions are available and
how they are supporting a specific business issue.
Regulation and Certification (AVR):
Instituting the investment board; Similar to an IT investment board,
AVR has a two-tiered management structure that is composed of the AVR
management team and the CIO management team. The AVR management team
includes the Associate Administrator and the Service Directors who make
the final decisions based upon recommendations and input from the CIO
management team and its business partners from each of the service
units. According to AVR, its IT investment process guide is still under
development and will be completed at the end of Fiscal Year 2004.
Meeting business needs; Each line of business within AVR identifies and
documents its business needs including project requirements and
specific users. Once the business needs have been identified, the IT
Management and Resources section prioritizes them for funding.
Selecting an investment; Programs in AVR are reviewed quarterly. For
major projects, meetings are designed to look at project milestones to
see if they are being met. These meetings are carried out biweekly and
presented to the AVR management team.
Providing investment oversight; The AVR CIO management team is
responsible for monitoring projects and reporting to the AVR Management
team. Biweekly meetings are held for major projects within AVR.
Capturing the investment information; AVR's system inventory is a part
of its enterprise architecture. The system inventory is being used
primarily in developing the Exhibit 300s. The performance of IT
projects in AVR is monitored daily, based upon each project's
individual plan, using project management tools such as MS Project.
According to AVR, not all projects have a project plan in place, but
AVR is trying to make it a requirement.
Source: GAO, based on information from FAA.
[A] ITEB is a board that can provide a governing structure so that
information technology is used as an agency wide strategic asset.
[B] Composed of Assistant Administrator; two Deputy Assistant
Administrators; three Office Directors; the Director, Center for
Management and Development; and the AHR Business Officer.
[End of table]
(310456):
FOOTNOTES
[1] See GAO, High-Risk Series: An Overview, GAO/HR-95-1 (Washington,
D.C.: February 1995); GAO, High-Risk Series: Information Management and
Technology, GAO/HR-97-9 (Washington, D.C.: February 1997); GAO, High-
Risk Series: An Update, GAO/HR-99-1 (Washington, D.C.: January 1999);
GAO, High-Risk Series: An Update, GAO-01-263 (Washington, D.C.: January
2001); and GAO, High-Risk Series: An Update, GAO-03-119 (Washington,
D.C.: January 2003).
[2] GAO, Information Technology Investment Management: A Framework for
Assessing and Improving Process Maturity, GAO-04-394G (Washington,
D.C.: March 2004).
[3] The 10 service units that make up the ATO include Safety,
Communications, Operations Planning, Finance, Acquisition & Business
Services, En Route and Oceanic Services, Terminal Services, Flight
Services, System Operations Services, and Technical Operations
Services.
[4] FAA uses three types of facilities to control traffic: airport
towers, terminal radar approach control facilities, and en route
centers. Airport towers direct traffic on the ground, before landing,
and after takeoff within 5 nautical miles from the airport and about
3,000 feet above the airport. Terminal radar approach control
facilities sequence and separate aircraft as they approach and leave
airports, beginning about 5 nautical miles and ending about 50 nautical
miles from the airport and generally up to 10,000 feet above the
ground. Air route traffic control centers, called en route centers,
control planes in transit and during approaches to some airports,
generally controlling air space that extends above 18,000 feet for
commercial aircraft.
[5] GAO-03-119.
[6] GAO, Air Traffic Control: FAA's Modernization Investment Management
Approach Could Be Strengthened, GAO/RCED/AIMD-99-88 (Washington, D.C.:
Apr. 30, 1999); GAO, Air Traffic Control: FAA's Modernization Efforts-
-Past, Present, and Future, GAO-04-227T (Washington, D.C.: Oct. 30,
2003).
[7] 49 U.S.C. 40110(d).
[8] May be delegated to an associate administrator.
[9] These teams may operate as entities or be organized into
subintegrated product teams or product teams to develop, procure, and
deliver products and services for users or customers. They are
responsible for the acquisition of new or improved capability for
services and products throughout their life cycles and for developing
cost and schedule baselines for candidate solutions during investment
analysis.
[10] The investment analysis staff assists and oversees the work of all
the investment analysis teams, is responsible for all investment
analyses, and is responsible for developing the tools, techniques, and
databases to ensure quality performance of investment analysis on
behalf of the JRC.
[11] A service organization is any organization within FAA that
delivers a service, whether it is a business unit, project office,
program directorate, or integrated product team, or whether it is
engaged in air traffic services, security, regulation, certification,
operations, commercial space transportation, or airport development.
These organizations are responsible and accountable for managing
service delivery throughout the life cycle. Investment decisions are
made to support service delivery. Specifically, after the investment
decision has been made, the service organization assumes responsibility
for the investment program, implements the selected solution, and
manages the product throughout the in-service management phase of its
life cycle.
[12] Non-NAS business units include Information Services (AIO), Region
and Center Operations (ARC), Regulation and Certification (AVR),
Financial Services (ABA), Research and Acquisition (ARA), Air Traffic
Services (ATS), and Human Resource Management (AHR).
[13] GAO-04-394G.
[14] GAO, Information Technology: DLA Needs to Strengthen Its
Investment Management Capability, GAO-02-314 (Washington, D.C.: Mar.
15, 2002); GAO, United States Postal Service: Opportunities to
Strengthen IT Investment Management Capabilities, GAO-03-3
(Washington, D.C.: Oct. 15, 2002); GAO, Information Technology:
Departmental Leadership Crucial to Success of Investment Reforms at
Interior, GAO-03-1028 (Washington, D.C.: Sept. 12, 2003); and GAO,
Bureau of Land Management: Plan Needed to Sustain Progress in
Establishing IT Investment Management Capabilities, GAO-03-1025
(Washington, D.C.: Sept. 12, 2003).
[15] An IT investment board is a decision-making body, made up of
senior program, financial, and information managers, that is
responsible for making decisions about IT projects and systems based on
comparisons and trade-offs among competing projects, with an emphasis
on meeting mission goals.
[16] 49 U.S.C. 40110(d).
[17] An advisory group, composed of representatives from each line of
business, that establishes guidelines for conducting mission analysis
and developing mission need statements as well as resolving agencywide
mission analysis issues.
[18] We reviewed the FAA Telecommunications Infrastructure, En Route
Communications Gateway, and Voice Switching and Control System (VSCS)
Control Subsystem Upgrade (VCSU) projects. The projects are described
in appendix I.
[19] VCSU is a subcomponent of the VSCS project. We decided to review
VCSU because its investment management process was carried out using
FAA's AMS, whereas the VSCS project was funded before the AMS became
part of the FAA's investment management process.
[20] According to FAA, no acquisition program baseline was prepared for
VCSU.
[21] GAO/RCED/AIMD-99-88.
[22] GAO-04-394G.
[23] VSCS is FAA's highly distributed, computer-controlled
communications and control system for U.S. air traffic management that
allows air traffic controllers to establish all air-to-ground and
ground-to-ground communications with pilots and other air traffic
controllers.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
