Information Technology

FAA Has Many Investment Management Capabilities in Place, but More Oversight of Operational Systems Is Needed Gao ID: GAO-04-822 August 20, 2004

The Federal Aviation Administration's (FAA) mission is to promote the safe, orderly, and expeditious flow of air traffic in the United States airspace system, commonly referred to as the National Airspace System (NAS). To maintain its ability to effectively carry out this mission FAA embarked, in 1981,on a multi-billion dollar effort to modernize its aging air traffic control (ATC) system, the principle technology component of the NAS. Yet the NAS modernization has continued to be plagued by cost increases, schedule delays, and performance shortfalls. To gain insight into how FAA is meeting its management challenges, congressional requesters asked GAO to evaluate FAA's processes for making IT investment management decisions. The objectives of this review included (1) evaluating FAA's capabilities for managing its IT investments and (2) determining what plans, if any, the agency might have for improving these capabilities.

Judged against the criteria of GAO's framework for information technology investment management (ITIM), which measures the maturity of an organization's investment management processes, FAA has established about 80 percent of the basic selection and control practices that it needs to manage its mission-critical investments. For example, business lines actively monitor projects throughout their life cycles. However, the agency's senior IT investment board does not regularly review investments that are in the "in-service management," or operational, phase of their life cycles, and this creates a weakness in FAA's ability to oversee more than $1 billion of its IT investments. In addition, the agency has not yet established the key practices that would allow it to manage all of its investments as one portfolio--an integrated set of competing options. Until FAA has established the practices that would enable it to effectively manage its annual IT budget of about $2.5 billion, agency executives lack assurance that they are selecting and managing the mix of investments that best meets the agency's needs and priorities. The agency has initiated efforts to improve its investment management processes, but it has not yet developed and implemented a comprehensive plan--supported by management--to guide all of its improvement efforts. Such a plan is crucial in helping FAA to coordinate and prioritize its improvement efforts and sustain its commitment to the efforts it already has under way. Without such a plan--and controls for implementing it--FAA will be unlikely to develop a mature investment management capability.

Recommendations

Our recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.

Director: Team: Phone:


GAO-04-822, Information Technology: FAA Has Many Investment Management Capabilities in Place, but More Oversight of Operational Systems Is Needed This is the accessible text file for GAO report number GAO-04-822 entitled 'Information Technology: FAA Has Many Investment Management Capabilities in Place, but More Oversight of Operational Systems Is Needed' which was released on September 20, 2004. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. GAO Highlights: Highlights of GAO-04-822, a report to congressional requesters Report to Congressional Requesters: August 2004: INFORMATION TECHNOLOGY: FAA Has Many Investment Management Capabilities in Place, but More Oversight of Operational Systems Is Needed: [Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-822]: Why GAO Did This Study: The Federal Aviation Administration‘s (FAA) mission is to promote the safe, orderly, and expeditious flow of air traffic in the United States airspace system, commonly referred to as the National Airspace System (NAS). To maintain its ability to effectively carry out this mission FAA embarked, in 1981,on a multi-billion dollar effort to modernize its aging air traffic control (ATC) system, the principle technology component of the NAS. Yet the NAS modernization has continued to be plagued by cost increases, schedule delays, and performance shortfalls. To gain insight into how FAA is meeting its management challenges, congressional requesters asked GAO to evaluate FAA‘s processes for making IT investment management decisions. The objectives of this review included (1) evaluating FAA‘s capabilities for managing its IT investments and (2) determining what plans, if any, the agency might have for improving these capabilities. What GAO Found: Judged against the criteria of GAO‘s framework for information technology investment management (ITIM), which measures the maturity of an organization‘s investment management processes, FAA has established about 80 percent of the basic selection and control practices that it needs to manage its mission-critical investments (see table below). For example, business lines actively monitor projects throughout their life cycles. However, the agency‘s senior IT investment board does not regularly review investments that are in the ’in-service management,“ or operational, phase of their life cycles, and this creates a weakness in FAA‘s ability to oversee more than $1 billion of its IT investments. In addition, the agency has not yet established the key practices that would allow it to manage all of its investments as one portfolio”an integrated set of competing options. Until FAA has established the practices that would enable it to effectively manage its annual IT budget of about $2.5 billion, agency executives lack assurance that they are selecting and managing the mix of investments that best meets the agency‘s needs and priorities. The agency has initiated efforts to improve its investment management processes, but it has not yet developed and implemented a comprehensive plan”supported by management”to guide all of its improvement efforts. Such a plan is crucial in helping FAA to coordinate and prioritize its improvement efforts and sustain its commitment to the efforts it already has under way. Without such a plan”and controls for implementing it”FAA will be unlikely to develop a mature investment management capability. Summary of Results for Foundational Critical Processes and Key Practices: [See PDF for table] Source: GAO. [End of table] What GAO Recommends: To strengthen FAA‘s investment management capability, GAO recommends that FAA develop and implement a plan to address the weaknesses identified in this report. In commenting on a draft of this report, the Department of Transportation commented that the report was balanced and fair, showing where FAA has many capabilities in place and identifying areas that need improvement. www.gao.gov/cgi-bin/getrpt?GAO-04-822. To view the full product, including the scope and methodology, click on the link above. For more information, contact David Powner, 202-512- 9286, pownerd@gao.gov or Lester Diamond, 202-512-7957, diamondl@gao.gov. [End of section] Contents: Letter: Results in Brief: Background: FAA Has Established an IT Management Structure to Manage Its NAS Investments: FAA Has Initiated Efforts to Improve Its Investment Management Process: DOT is Taking Steps to Integrate Oversight of FAA's IT Investments: Conclusions: Recommendations for Executive Action: Agency Comments: Appendixes: Appendix I: Objectives, Scope, and Methodology: Appendix II: Investment Management Process Used by Some Organizational Units to Manage Non-NAS Investments: Tables: Table 1: NAS and Non-NAS IT Investments: Table 2: Stage 2 Critical Processes--Building the Investment Foundation: Table 3: Summary of Results for Stage 2 Critical Processes and Key Practices for NAS Investments: Table 4: Instituting the Investment Board: Table 5: Meeting Business Needs: Table 6: Selecting an Investment: Table 7: Providing Investment Oversight: Table 8: Capturing Investment Information: Table 9: Stage 3 Critical Processes--Developing a Complete Investment Portfolio: Table 10: Status of Stage 3 Critical Processes: Figures: Figure 1: FAA's Life Cycle Management Process: Figure 2: Detailed Breakdown of FAA's Life Cycle Management Process: Figure 3: The Five ITIM Stages of Maturity with Critical Processes: Abbreviations: ABA: Financial Services unit: AHR: Human Resource Management unit: AIO: Information Services unit: AMS: Acquisition Management System: APB: acquisition program baseline: ARA: Research and Acquisition unit: ARC: Region and Center Operations unit: ATC: air traffic control: ATO: Air Traffic Organization: ATS: Air Traffic Services unit: AVR: Regulation and Certification unit: CIO: Chief Information Officer: DOT: Department of Transportation: ECG: En Route Communications Gateway: F&E: facilities and equipment (predeployment stage of system life cycle): FAA: Federal Aviation Administration: FTI: FAA Telecommunications Infrastructure: IAT: Investment Analysis Team: IT: information technology: ITEB: Information Technology Executive Board: ITIM: Information Technology Investment Management framework: ITIPS: Information Technology Investment Portfolio System: JRC: Joint Resources Council: NAS: National Airspace System: NSIP: NAS Support Integration Process: OMB: Office of Management and Budget: OPS: operations (postdeployment stage of system life cycle): PIR: postimplementation review: VCSU: VSCS Control Subsystem Upgrade: VSCS: Voice Switching and Control System: Letter August 20, 2004: The Honorable Tom Davis: Chairman: Committee on Government Reform: U.S. House of Representatives: The Honorable Adam H. Putnam: Chairman: Subcommittee on Technology, Information Policy, Intergovernmental Relations and the Census: Committee on Government Reform: U.S. House of Representatives: The Federal Aviation Administration's (FAA) mission is to promote the safe, orderly, and expeditious flow of air traffic in the United States airspace system, commonly referred to as the National Airspace System (NAS). To maintain its ability to effectively carry out this mission FAA embarked, in 1981, on a multibillion dollar effort to modernize its aging air traffic control (ATC) system, the principle technology component of the NAS. Over the past 2 decades, individual FAA modernization projects have experienced cost overruns, schedule delays, and performance shortfalls of large proportions. Because of the size, complexity, cost, and problem-plagued past of FAA's modernization program, we have designated it a high-risk information technology investment since 1995.[Footnote 1] This report is one in a series of reports responding to your request to evaluate FAA's efforts to address the information technology (IT) management challenges it faces as it continues to modernize the ATC system. It focuses on FAA's processes for making IT investment management decisions and uses our Information Technology Investment Management (ITIM) framework,[Footnote 2] which was released at a hearing of the subcommittee on March 3, 2004. The framework provides a method for evaluating and assessing how well an agency is selecting and managing its IT resources. As agreed, our objectives were to (1) evaluate FAA's capabilities for managing its IT investments, (2) determine what plans the agency might have for improving these capabilities, and (3) describe the Department of Transportation's (DOT) oversight of FAA's investments and investment management process. To address these objectives we analyzed documents and interviewed agency officials to (1) validate and update FAA's self-assessments of the key practices in the framework, (2) evaluate FAA's plans for improving its capabilities, and (3) describe the department's oversight role. We performed our work from October 2003, through July 2004, in accordance with generally accepted government auditing standards. Appendix I contains further details on our objectives, scope, and methodology. Results in Brief: FAA has established most--about 80 percent--of the basic practices needed to manage its mission critical investments, including many of the foundational practices for selecting and controlling IT investments. These key practices provide additional assurance that the investments selected will meet organizational needs and will be completed on time and within budget. The practices also will enable the agency to manage its IT investments as a portfolio, or integrated set of competing options. Even with these many capabilities in place, weaknesses remain in several areas. Specifically, FAA: * does not involve its senior IT investment board in regular reviews of investments that have entered the in-service management phase, that is, those systems that have completed development and become operational; * does not have standard practices for managing its mission-support and administrative investments; * has not developed a process where the senior IT investment board regularly reviews the full portfolio of investments; and: * has not implemented postimplementation reviews of its major investments to validate that they are providing the expected benefits after they become operational. FAA has begun to act to resolve the weaknesses described above, but until FAA establishes the practices it needs to effectively manage its IT investments, executives cannot be assured that they are selecting and managing the mix of investments that best meets the agency's needs and priorities. Establishing the capabilities needed to effectively manage investments requires the development and implementation of a plan, supported by management, that defines and prioritizes improvements to the investment process. While FAA has initiated a series of efforts to improve its investment management processes, it does not have such a plan. Without this plan--and controls for implementing it--it is unlikely that the agency will effectively establish mature investment management capabilities. The Department of Transportation has recently initiated several efforts that can serve to provide better departmental oversight of FAA investments. For example, DOT is finalizing capital investment guidance for all of its operating administrations to follow in implementing their investment management processes, and it has initiated a process for reviewing the fiscal year 2006 budget justifications for major programs, including those of FAA. The department has also identified about a dozen programs it plans to monitor on a regular basis and has asked FAA to report cost, schedule, and performance data on some of its programs quarterly. To further strengthen FAA's investment management capability, we are recommending that the agency develop and implement a plan aimed at addressing the weaknesses identified in this report. In commenting on a draft of this report, the Department of Transportation's Director of Audit Relations stated that GAO did a good job of keeping the report balanced and fair, showing where FAA has many capabilities in place and identifying areas that need improvement. The agency also provided a technical comment, which we have incorporated into the report. Background: FAA's Mission and Organizational Structure: As an agency of the Department of Transportation, FAA's mission is to promote the safe, orderly, and expeditious flow of air traffic in the national airspace. To fulfill its mission requires the extensive use of technology. The achievement of the agency's mission is also dependent in large part on the skills and expertise of its workforce. Its workforce of nearly 50,000 people provides aviation services that include air traffic control; maintenance of air traffic control equipment; and certification of aircraft, airline operations, and pilots. FAA is organized into several staff support offices (examples include the Office of Information Services and the Office of Human Resource Management) and five lines of business, which include Airports, Regulation and Certification, Commercial Space Transportation, the Office of Security and Hazardous Materials, and the newly formed Air Traffic Organization (ATO). The ATO was formed on February 8, 2004, to better provide safe, secure, and cost-effective air traffic services now and into the future. The Air Traffic Services and the Research and Acquisitions units, which had been primarily responsible for managing air traffic services within FAA, were combined into one performance- based organization to create ATO. ATO is led by FAA's Chief Operating Officer and consists of 10 service units.[Footnote 3] FAA's Use of IT: FAA relies extensively on information technology to carry out its NAS operations. It constantly depends on the adequacy and reliability of the nation's ATC system, which comprises a vast network of radars; automated data processing, navigation, and communications equipment; and ATC facilities.[Footnote 4] Through this system, FAA provides services such as controlling takeoffs and landings and managing the flow of traffic between airports. For example, the Integrated Terminal Weather System is employed to allow maximum use of airport runways in all kinds of weather through a variety of weather sensors. The Wide Area Augmentation System is used to provide vertically guided landing to aircraft at thousands of airports and airstrips where there is currently no vertically guided landing capability. FAA also relies on IT to carry out its mission-support and administrative operations (non-NAS operations). For example, FAA uses IT to support accident and incident investigations, security inspections, and personnel and payroll functions. With an IT budget of about $2.5 billion for fiscal year 2004, FAA accounts for over 90 percent of the Department of Transportation's IT budget. The amount of investments in both NAS and non-NAS IT is shown in the table 1 below. Table 1: NAS and Non-NAS IT Investments: Type of investment: NAS; Funding of IT investments: Facilities and Equipment (F&E) (development through 2 years of operations); Total IT investment in fiscal year 2004 in billions of dollars: $1.464. Type of investment: NAS; Funding of IT investments: Operations (OPS) (through the rest of the life cycle); Total IT investment in fiscal year 2004 in billions of dollars: $0.834. Type of investment: Non-NAS; Funding of IT investments: Operations; Total IT investment in fiscal year 2004 in billions of dollars: $0.350-0.500[A]. Source: FAA. [A] According to FAA, these numbers will be verified via baselining by the end of Fiscal Year 2004 to reflect non-NAS IT assets and their costs. [End of table] Prior Reviews Identified Weaknesses in the Agency's IT Investment Management Process: In 1995, we designated FAA's modernization of its air traffic control system, the principle technology component of the NAS, as a high-risk area because of the size and complexity of the program and FAA's many failures in meeting projects' cost, schedule, and performance goals. In our latest High-Risk Series, issued in January 2003,[Footnote 5] we addressed the critical need for FAA to continue to improve its investment management practices--the management processes the agency uses to select, control and evaluate the benefits realized from its IT spending--because the agency would be spending nearly $16 billion more through FY 2007, after having already spent $35 billion since 1981. Other reports have also noted weaknesses in FAA's IT investment management processes and have made a number of recommendations to address this area.[Footnote 6] For instance, last year we reported that while FAA had improved its processes, several issues remained unresolved. We noted, for example, that the agency had not yet implemented processes for evaluating projects after implementing them, in order to identify lessons learned and improve the investment management process. FAA's Current Approach to Investment Management: FAA's process for managing an IT investment varies depending on the type of investment--NAS systems in development through the second year of operation (F&E), NAS systems in operation after the second year (OPS), and non-NAS systems each follow different processes. NAS investments are managed through a standardized process, the FAA Acquisition Management System (AMS), and non-NAS investments are managed through a number of different processes. Process for Managing NAS Investments: In April 1996, FAA implemented its AMS in response to legislation that directed the agency to develop a new acquisition management system.[Footnote 7] Because of FAA's contention that some of its modernization problems were caused by federal acquisition regulations, the Congress enacted legislation in November 1995 that exempted the agency from most federal procurement laws and regulations and directed FAA to develop and implement a new acquisition management system that would address the unique needs of the agency. AMS was intended to reduce the time and cost for fielding new products and services by introducing (1) a new investment management system that spans the entire life cycle of an acquisition, (2) a new procurement system that provides flexibility in selecting and managing contractors, and (3) organizational and human capital reforms that support the new investment and procurement systems. AMS provides high-level acquisition policy and guidance for selecting and controlling FAA's NAS investments through all phases of the acquisition life cycle, which is organized into a series of phases and decision points that include (1) mission analysis, (2) investment analysis, (3) solution implementation, and (4) in-service management. To select investments, FAA has established two processes--mission analysis and investment analysis--which together constitute a set of policies, procedures, and guidance that enhance the agency's ability to screen projects that are submitted for funding. Also, through these two processes FAA is to assess and rank each project based on its relative costs, benefits, risks, and contribution to FAA's mission, and a senior, corporate-level decision-making group selects projects for funding. After a project has been selected, FAA officials are required to formally establish the life cycle cost, schedule, benefits, and performance baselines that are used to monitor the project's status throughout the remaining phases of the acquisition management life cycle. See figure 1 for a graphic depiction of FAA's life cycle management process. Figure 1: FAA's Life Cycle Management Process: [See PDF for image] Note: During the front end of the life cycle, research and system analysis activities are undertaken to discover applications of new technology for FAA's present services, explore new opportunities for service delivery, solve problems within current operations, and define requirements. [End of figure] Several groups are involved in managing FAA's NAS investments; they perform functions from analysis of mission needs and alternative investments through system development, implementation, operation, and, ultimately, disposal. The roles and responsibilities of each group are described below: Joint Resources Council (JRC)--This board makes corporate-level resource and investment decisions and establishes investment programs. Members include Associate Administrators representing FAA's lines of business, the FAA Acquisition Executive,[Footnote 8] the Chief Financial Officer, the Chief Information Officer (CIO), and the Assistant Administrators for System Safety, for Policy, Planning and International Aviation, and for Region and Center Operations. The board is supported by the JRC Secretariat Team, a group that facilitates the board's processes by maintaining the meeting calendar and guidance documents, developing records of decisions, and providing advisory and liaison support to programs. Systems Engineering/Operational Analysis Team--This team performs affordability assessments for newly proposed investments and prepares recommendations for the reprogramming of funds from lower priority programs. It also prepares annual budget submissions for approval by the JRC. This team is composed of representatives from each line of business and from other functional disciplines and is chaired by the Director, System Architecture and Investment Analysis. Investment Analysis Team (IAT)--This team is assembled for a relatively short period for each specific investment being considered, to conduct the detailed analysis of alternatives that will lead to selecting and recommending a preferred acquisition solution. The team draws experts from the integrated product teams,[Footnote 9] the organizational unit with the need, the investment analysis staff,[Footnote 10] and other organizations. Corporate Mission Analysis Organization--Performs agency-level mission analysis and coordinates service area analysis, an activity that is conducted during mission analysis to (1) identify capability shortfalls for or in conjunction with service organizations,[Footnote 11] (2) ensure alignment with agency strategic goals, and (3) eliminate redundant activity, duplicate benefits, service gaps, and service overlaps. It also develops and maintains standards and tools for conducting service area analysis, and it assists service organizations in establishing a service area analysis capability. In addition to identifying the roles and responsibilities of the groups involved in the management process, AMS provides guidance on the documents and decisions that result from each of the life cycle phases. For example, through the mission analysis phase, FAA identifies critical needs that the agency must meet for improving the safety, capacity, efficiency, and effectiveness of the NAS. Approval of a mission need statement by the JRC signifies that the agency agrees that the need is critical enough to proceed to the next phase--investment analysis. During the investment analysis phase, the IAT is to analyze and recommend a solution that best satisfies FAA's performance goals and customer service needs. This team is then to rank each proposed project based on a number of factors, including how well it meets mission needs compared to other projects and whether it has a favorable cost-benefit ratio. As part of the JRC selection process, the life cycle cost, schedule, benefits, and performance baselines are established in a formal document called the acquisition program baseline (APB), which is designed to be used by program offices to monitor a project's status in achieving those baselines throughout the remaining phases of the acquisition management life cycle. The solution implementation phase begins when the JRC approves and funds a project, establishes its acquisition program baseline, and authorizes the service organizations to implement and manage the project over its life cycle. After the project has been implemented and is in operation (FAA's in-service management phase), the service organizations monitor and assess operational performance. Also during this phase, the project is monitored to determine whether the current capability satisfies the demand for services or whether another solution offers the potential for improving safety or effectiveness or for significantly lowering costs. If the current capability is lacking, FAA initiates a process whereby the mission need would be revalidated and the investment analysis process begun again, possibly leading to a new investment decision. Figure 2 provides detail on the phases of FAA's IT investment management process and decision points. The highlighted decision points represent those for which the JRC must make an approval decision before a project can move forward. Senior executives have stated that with the reorganization of the ATO in February 2004, discussions have been held about realigning the investment management process to make the heads of the service units responsible and accountable for managing programs' capital investments and operating costs from inception to retirement. In the past, the business units have been organized to manage either capital investments or operating costs, but not both. These discussions have not yet led to specific changes in FAA's investment management processes and responsibilities. Figure 2: Detailed Breakdown of FAA's Life Cycle Management Process: [See PDF for image] [A] Corporate requirements organization takes the lead in planning for the concept and requirements definition phase of the life cycle management process. [B] In a February 2004 memorandum, FAA's Chief Operating Officer assumed the in-service decision authority and stated that he would delegate this responsibility to the vice presidents of the service organizations, unless the JRC retained the in-service decision authority. If the JRC retains this authority, it determines the in- service decision authority at the time of the final investment decision. [End of figure] Process for Managing Non-NAS Investments: While the AMS was intended to apply to all FAA investment programs, it has not been implemented for non-NAS investments. Each of the agency's business line and staff offices that manage non-NAS investments[Footnote 12] has implemented its own processes for managing these investments. Examples of these various non-NAS investment processes include the following: * Regarding an investment management board structure, the Financial Services staff office has an informal board consisting of the Chief Financial Officer, Deputy Chief Financial Officer, and heads of offices within Financial Services. The Financial Services life cycle process guide directs the board's operations. In the Regulation and Certification unit, the senior management team makes investment management decisions with input from the Chief Information management team. This unit is developing an IT investment management processes guide, which is expected to be completed by the end of the fiscal year. * When selecting investments, the Human Resource Management unit uses its established annual budget formulation process, while the Region and Center Operations unit is moving toward a new process whereby in order to be selected investments need to demonstrate, at a minimum, that they (1) are compliant with FAA's architecture, (2) have a business sponsor, (3) have a solid business case, and (4) can be funded. * In controlling investments, Information Services has developed processes to monitor contract expenditures, and unit managers regularly perform financial management reviews of the programs under their purview, but there is no structured process for oversight of projects' performance against expectations. In the Human Resource Management unit, division managers hold quarterly reviews to assess projects' progress in meeting cost and schedule expectations and aligning with strategic goals. Descriptions of the processes used by each of the units responsible for managing non-NAS investments can be found in appendix II. In January 2004, the FAA Administrator established the Information Technology Executive Board (ITEB) to "strengthen FAA's ability to use IT as an agencywide strategic asset" and "guide fundamental changes in the governance of IT assets." Its charter calls for the ITEB to assume responsibility for making investment decisions about non-NAS IT investments. However, the ITEB has not yet implemented this aspect of its charter. Therefore, at the current time there is no single board or investment management process for non-NAS investments that would be analogous to the JRC board and AMS process that are used for NAS investments. ITIM Maturity Framework: The ITIM framework is a maturity model composed of five progressive stages of maturity that an agency can achieve in its investment management capabilities.[Footnote 13] It was developed on the basis of our research into the IT investment management practices of leading private-and public-sector organizations. The framework identifies critical processes for making successful IT investments, organized into the five increasingly mature stages. These maturity stages are cumulative; that is, in order to attain a higher stage of maturity, the agency must have institutionalized all of the requirements for all of the lower stages, in addition to those for the higher stage. The ITIM can be used both to assess the maturity of an agency's investment management processes and as a tool for organizational improvement. The overriding purpose of the framework is to encourage investment processes that increase business value and mission performance, reduce risk, and increase accountability and transparency in the decision process. We have used the framework in several of our evaluations,[Footnote 14] and a number of agencies have adopted it. These agencies have used ITIM for purposes ranging from self-assessment to redesign of their IT investment management processes. ITIM's five maturity stages represent steps toward achieving stable and mature processes for managing IT investments. Each stage builds on the lower stages; the successful attainment of each stage leads to improvement in the organization's ability to manage its investments. With the exception of the first stage, each maturity stage is composed of "critical processes" that must be implemented and institutionalized in order for the organization to achieve that stage. These critical processes are further broken down into key practices that describe the types of activities that an organization should be performing to successfully implement each critical process. An organization may be performing key practices from more than one maturity stage at the same time. This is not unusual, but efforts to improve investment management capabilities should focus on becoming compliant with lower-stage practices before addressing higher-stage practices. Stage 2 of the ITIM framework encompasses building a sound investment management process by establishing basic capabilities for selecting new IT projects. It also involves developing the capability to control projects so that they finish predictably within established cost and schedule expectations and the capability to identify potential exposures to risk and put in place strategies to mitigate that risk. The basic selection processes established in Stage 2 lays the foundation for more mature selection capabilities in Stage 3. Stage 3 requires that an organization continually assess both proposed and ongoing projects as parts of a complete investment portfolio--an integrated and competing set of investment options. It focuses on establishing a consistent, well-defined perspective on the IT investment portfolio and maintaining mature, integrated selection (and reselection), control, and evaluation processes, which are to be evaluated during postimplementation reviews (PIR). This portfolio perspective allows decision makers to consider the interaction among investments and the contributions to organizational mission goals and strategies that could be made by alternative portfolio selections, rather than relying exclusively on the balance between the costs and benefits of individual investments. Stages 4 and 5 require the use of evaluation techniques to continuously improve both the investment portfolio and investment processes in order to better achieve strategic outcomes. At Stage 4 maturity an organization has the capacity to conduct IT succession activities and therefore can plan and implement the deselection of obsolete, high- risk, or low-value IT investments. An organization with Stage 5 maturity conducts proactive monitoring for breakthrough information technologies that will enable it to change and improve its business performance. Organizations implementing Stages 2 and 3 have in place the selection, control, and evaluation processes that are required by the Clinger-Cohen Act. Stages 4 and 5 define key attributes that are associated with the most capable organizations. Figure 3 shows the five maturity stages and the critical processes associated with each. Figure 3: The Five ITIM Stages of Maturity with Critical Processes: [See PDF for image] [End of figure] As defined by the model, each critical process consists of "key practices" that must be executed to implement the critical process. FAA Has Established Many Key Practices for Managing NAS Investments but Lacks Oversight of Operational Systems: In order to have the capabilities to effectively manage IT investments, an agency should, at a minimum, (1) build an investment foundation by putting basic, project-level control and selection practices in place (Stage 2 capabilities) and (2) manage its projects as a portfolio of investments, treating them as an integrated package of competing investment options and pursuing those that best meet the strategic goals, objectives, and mission of the agency; and it should also conduct PIRs to maintain mature, integrated selection, control, and evaluation processes (Stage 3 capabilities). In addition, an agency would be well served by implementing capabilities for improving its investment process through performance evaluations of its portfolio and succession management of current investments (Stage 4 capabilities). In order to develop the capabilities to effectively manage its investments, FAA would, at minimum, need to implement Stage 2 capabilities for both its NAS and non-NAS investments and Stage 3 capabilities for its portfolio of investments. FAA's investment management capabilities vary depending on whether an investment is considered to be NAS or non-NAS. Specifically: * For NAS investments, FAA has executed 30 of the 38 Stage 2 key practices that are required to establish a foundation for investment management maturity. For these investments, the agency has in place a strong set of processes to support investment management, although the JRC does not regularly review investments that have passed into the in- service management phase (i.e., operational systems). * For its non-NAS investments, the agency has not yet adequately implemented a single management line of responsibility and the standard processes needed to manage in a consistent manner. Although some structured processes exist within individual business units, this lack of consistency undermines the agency's maturity. * In Stage 3, the lack of regular JRC oversight of operational systems and the absence of a structured approach to managing non-NAS investments prevent FAA from managing its investments as a portfolio that includes all major NAS and non-NAS investments. In addition, the agency is not conducting PIRs on its major investments. * FAA has not executed any of the Stage 4 key practices for managing the succession of its information systems, although the agency has begun to address this weakness by defining procedures for retiring investments in the AMS. When FAA implements all of the key practices associated with building the investment foundation and managing its investments as a portfolio, the agency will have greater assurance that it has selected the mix of investments that best supports its strategic goals and that it will be able to manage the investments to successful completion. FAA Has Established Much of the Foundation Needed to Manage Its NAS Investments: At the ITIM Stage 2 level of maturity, an organization has attained repeatable, successful IT project-level investment control processes and basic selection processes. Through these processes, the organization can identify expectation gaps early and take appropriate steps to address them. According to ITIM, critical processes at Stage 2 include (1) defining IT investment board[Footnote 15] operations, (2) identifying the business needs for each IT investment, (3) developing a basic process for selecting new IT proposals and reselecting ongoing investments, (4) developing project-level investment control processes, and (5) collecting information about existing investments. Table 2 describes the purpose of each of the Stage 2 critical processes. Table 2: Stage 2 Critical Processes--Building the Investment Foundation: Critical process: Instituting the investment board; Purpose: To define and establish an appropriate IT investment management structure and the processes for selecting, controlling, and evaluating IT investments. Critical process: Meeting business needs; Purpose: To ensure that IT projects and systems support the organization's business needs and meets users' needs. Critical process: Selecting an investment; Purpose: To ensure that a well-defined and disciplined process is used to select new IT proposals and reselect ongoing investments. Critical process: Providing investment oversight; Purpose: To review the progress of IT projects and systems, using pre- defined criteria and checkpoints, in meeting cost, schedule, risk, and benefit expectations and to take corrective action when these expectations are not being met. Critical process: Capturing investment information; Purpose: To make available to decision makers information to evaluate the impacts and opportunities created by proposed (or continuing) IT investments. Source: GAO. [End of table] To its credit, FAA has put in place about 80 percent of the key practices associated with managing its NAS investments through the Stage 2 critical processes. The agency has satisfied all of the key practices associated with capturing investment information and most of those associated with instituting the investment board, meeting business needs, selecting an investment, and providing investment oversight. Most of the weaknesses in these critical processes relate to NAS investments in the in-service management phase. Table 3 summarizes the status of FAA's critical processes for Stage 2, showing how many key practices FAA has executed in managing its NAS investments. Table 3: Summary of Results for Stage 2 Critical Processes and Key Practices for NAS Investments: Critical process: Instituting the investment board; Key practices executed: 7; Total required by critical process: 8; Percentage of key practices executed: 88%. Critical process: Meeting business needs; Key practices executed: 6; Total required by critical process: 7; Percentage of key practices executed: 86%. Critical process: Selecting an investment; Key practices executed: 7; Total required by critical process: 10; Percentage of key practices executed: 70%. Critical process: Providing investment oversight; Key practices executed: 4; Total required by critical process: 7; Percentage of key practices executed: 57%. Critical process: Capturing investment information; Key practices executed: 6; Total required by critical process: 6; Percentage of key practices executed: 100%. Critical process: Total; Key practices executed: 30; Total required by critical process: 38; Percentage of key practices executed: 79%. Source: GAO. [End of table] FAA Has Established an IT Management Structure to Manage Its NAS Investments: The establishment of decision-making bodies or boards is a key component of the IT investment management process. At the Stage 2 level of maturity, organizations define one or more boards, provide resources to support their operations, and appoint members who have expertise in both operational and technical aspects of proposed investments. The boards operate according to a written IT investment process guide that is tailored to the organization's unique characteristics, thus ensuring that consistent and effective management practices are implemented across the organization. Once board members are selected, the organization ensures that they are knowledgeable about policies and procedures for managing investments. Organizations at the Stage 2 level of maturity also take steps to ensure that executives and line managers support and carry out the decisions of the IT investment board. According to ITIM, an IT investment management process guide should be a key authoritative document that the organization uses to initiate and manage IT investment processes and should provide a comprehensive foundation for the policies and procedures that are developed for all of the other related processes. (The complete list of key practices is provided in table 4.) FAA has executed 7 of the 8 key practices for this critical process. For example, in 1996, Congress directed FAA to develop a new acquisition management system as part of a broad mandate for acquisition reform at the agency.[Footnote 16] In response, FAA implemented AMS in April 1996. AMS establishes policy and guidance for all aspects of the agency's acquisition life cycle and documents the investment management process used for NAS investments. The agency established the JRC as its corporate-level investment board for the NAS investments. The JRC makes select and control decisions, including corporate decisions on mission needs, acquisition investments, and acquisition program baseline changes; it also reviews and recommends approval of the agency's F&E budget submission. The board is adequately resourced to support its operations. The JRC Secretariat Team supports the board in such ways as developing and updating guidance, scheduling meetings, and preparing and executing the JRC readiness process. In addition, the Mission Analysis Steering Group[Footnote 17] is responsible for assisting the board in prioritizing mission needs, while the Systems Engineering/Operational Analysis Team is to assist in addressing budget issues among investments. The JRC consists of senior officials from both business and IT areas, including the Chief Information Officer and the associate administrators representing FAA lines of business. These members are to exhibit the core competencies required by FAA in selecting executives and in assessing executive training needs. In addition, the agency offers a 3-day AMS overview course for all employees, including JRC members. Although the board as an entity does not oversee the development and maintenance of AMS, it is involved through FAA's Acquisition System Advisory Group, which evaluates all proposed changes to AMS. To ensure that the board's decisions are carried out, an acquisition program baseline document is approved at the JRC final investment decision point; this document identifies the capabilities, benefits, costs, and schedule for the approved investment, which are monitored by FAA through its variance reporting process. Despite these strengths, FAA has not yet clearly defined the relationship between the JRC and the newly formed ITEB. Although the ITEB was established by the Administrator to function as the central authority responsible for assuring that FAA IT investments are based on sound business practices, FAA has not yet clearly delineated the specific roles the ITEB is to play and the relationship it will have with the JRC. This task has been assigned to the ITEB as a longer-range initiative. Table 4 shows the rating for each key practice required to implement the critical process for instituting the investment board at the Stage 2 level of maturity. Each of the "Executed" ratings shown below represents instances where, based on the evidence provided by FAA officials, we concluded that the specific key practices were executed by the organization. Table 4: Instituting the Investment Board: Type of practice: Organizational commitments; Key Practice: 1. An enterprisewide IT investment board composed of senior executives from IT and business units is responsible for defining and implementing the organization's IT investment governance process; Rating: Executed; Summary of evidence: The JRC, FAA's corporate-level investment board for the NAS investments, is responsible for defining and implementing the agency's IT investment governance process. It consists of the agency's most senior executives, including the CIO, Chief Financial Officer, FAA's Acquisition Executive, and Associate Administrators from its lines of business. Type of practice: Organizational commitments; Key Practice: 2. The organization has a documented IT investment process directing each investment board's operations; Rating: Executed; Summary of evidence: FAA's AMS sets forth acquisition policy and processes for the JRC. Also, the board has established its own guidance to implement AMS core JRC policy. Type of practice: Prerequisites; Key Practice: 1. Adequate resources, including people, funding, and tools, are provided for supporting the operations of each IT investment board; Rating: Executed; Summary of evidence: Adequate resources are provided to support the board's operations. The JRC Secretariat Team provides such operations support as developing and updating guidance, scheduling meetings, and preparing and distributing records of decisions. Two other groups support the JRC decision-making process. The Mission Analysis Steering Group assists the board in ranking mission needs, while the Systems Engineering/Operational Analysis Team provides assistance by performing affordability assessments for the JRC when it is considering alternatives during investment analysis. Type of practice: Prerequisites; Key Practice: 2. The board members understand the organization's IT investment management policies and procedures and the tools and techniques used in the board's decision-making process; Rating: Executed; Summary of evidence: JRC members are senior managers representing all agency lines of business. They include the CIO, Chief Financial Officer, and Associate Administrators representing FAA lines of business such as Air Traffic Services. Core executive competencies based on FAA's Executive Success Profile are used in selecting executives and in assessing executive training needs. A 3-day AMS overview course is also available. Type of practice: Prerequisites; Key Practice: 3. Each board's span of authority and responsibility is defined to minimize overlaps or gaps among the boards; Rating: Not executed; Summary of evidence: The JRC is FAA's corporate-level investment board for making decisions related to NAS investments. In January 2004, the ITEB was established to oversee the governance of the agency's IT assets. However, the ITEB has yet to take significant action on the charge in its charter to clearly delineate the roles it is to play and its relationship with the JRC. Type of practice: Activities; Key Practice: 1. The enterprisewide investment board has oversight responsibilities for the development and maintenance of the organization's documented IT investment process; Rating: Executed; Summary of evidence: Although the JRC does not directly oversee the development and maintenance of the FAA's documented investment process, it is involved in this process through FAA's Acquisition System Advisory Group, which is a corporate crossfunctional body that evaluates all proposed changes to AMS. Membership consists of representatives from each line of business as well as the JRC Secretariat Team. Policy changes that are endorsed by the Group are presented, via the FAA Acquisition Executive, who is on the JRC, to the Administrator for approval. Type of practice: Activities; Key Practice: 2. Each investment board operates in accordance with its assigned authority and responsibility; Rating: Executed; Summary of evidence: The JRC is operating in accordance with its assigned authority and responsibility as FAA's corporate-level investment board for making decisions related to NAS investments. The charter for the ITEB specifically indicates the ITEB's responsibilities, including for making decisions for non-NAS IT acquisitions. Type of practice: Activities; Key Practice: 3. The organization has established management controls for ensuring that investment boards' decisions are carried out; Rating: Executed; Summary of evidence: FAA has controls for ensuring that the JRC's investment decisions are carried out as approved. At the JRC final investment decision point, an acquisition program baseline document is finalized and approved, which represents the mutual agreement between the JRC, the provider organization, and the user organization concerning the expected capability, benefits, costs, and schedule for the investment program. It also establishes performance metrics for assessing the program's success. Source: GAO. [End of table] FAA Has a Process for Ensuring That its Investments Support Business Needs and Meet Users' Needs: Defining business needs for each IT project helps to ensure that projects and systems support the organization's business needs and meet users' needs. This critical process ensures that a link exists between the organization's business objectives and its IT management strategy. According to ITIM, effectively meeting business needs requires, among other things, (1) documenting business needs with stated goals and objectives, (2) identifying specific users and other beneficiaries of IT projects and systems, (3) providing adequate resources to ensure that projects and systems support the organization's business needs and meet users' needs, and (4) periodically evaluating the alignment of IT projects and systems with the organization's strategic goals and objectives. (The complete list of key practices is provided in table 5.) FAA has in place 6 of the 7 key practices for meeting business needs. The agency's AMS and mission analysis guidance calls for business needs for both proposed and ongoing IT projects and systems to be identified in the mission need statement developed during the mission analysis phase. FAA also has detailed procedures for developing this document that call for identifying business needs. Resources for ensuring that IT projects and systems support the organization's business needs and meet users' needs include service organizations, the Corporate Mission Analysis Organization, the Mission Analysis Steering Group, and detailed procedures and associated templates for developing mission need statements. FAA's specific business mission, with stated goals and objectives, is defined in the Federal Aviation Administration Flight Plan for fiscal years 2004 through 2008. Further, FAA defines and documents business needs for both proposed and ongoing IT projects and identifies users and other beneficiaries during its mission analysis activities. In addition, the AMS policy calls for users to participate in project management throughout the FAA life cycle management process. For the three projects we reviewed,[Footnote 18] we verified that business needs and specific users and other beneficiaries were identified and documented in mission needs statements as well as in other documents. In addition, users are involved in project management throughout the life cycle of the projects. For example, according to project officials, En Route Communications Gateway (ECG) users participate in project meetings, weekly integrated product team status meetings, and monthly En Route domain national deployment teleconferences. FAA Telecommunications Infrastructure's (FTI) end users are heavily involved in the "operational test" period, which determines whether the equipment can be safely implemented in NAS. VSCS Control Subsystem Upgrade[Footnote 19] users are involved in the project's life cycle via a Web site through which they review and comment on project documentation. Despite these strengths, the JRC has no process for evaluating the organizational alignment of NAS systems through most of their in- service management phase (and non-NAS investments, which are described separately in this report). While the JRC does evaluate the alignment of projects and systems with organizational goals throughout the systems' development and 2 years into their operations as part of the annual budget formulation process, it does not use any consistent process to review projects and systems after that point in their life cycles. For NAS systems in the in-service management phase, these activities are carried out within the business unit that owns the system, but the JRC does not regularly oversee these processes and may go for several years without reviewing a system's alignment with organizational goals. In-service NAS systems only return to the JRC if they are judged to require additional funds for correction. Until FAA establishes a process for periodic evaluation of systems throughout the in-service management phase and takes corrective actions when misalignment occurs, the agency will not be able to ensure that these projects, totaling about $1.3 billion per year, are still continuing to maintain alignment with the FAA's strategic plans and its business goals and objectives. Table 5 shows the rating for each key practice required to implement the critical process for meeting business needs at the Stage 2 level of maturity and summarizes the evidence that supports these ratings. Table 5: Meeting Business Needs: Type of practice: Organizational commitments; Key practice: 1. The organization has documented policies and procedures for ensuring IT projects or systems that support the organization's ongoing and future business needs; Rating: Executed; Summary of evidence: AMS and mission analysis guidance contain documented policies and procedures for identifying the IT projects or systems that support the organization's ongoing and future business needs. Type of practice: Prerequisites; Key practice: 2. The organization has a documented business mission with stated goals and objectives; Rating: Executed; Summary of evidence: The Federal Aviation Administration Flight Plan (Strategic Plan) for fiscal years 2004 through 2008 defines the agency's mission goals and objectives. Type of practice: Prerequisites; Key practice: 3. Adequate resources, including people, funding, and tools, are provided for ensuring that IT projects and systems support the organization's business needs and meet users' needs; Rating: Executed; Summary of evidence: FAA has adequate resources for ensuring that its IT projects and systems support the organization's business needs and meet users' needs. They include service organizations, the Mission Analysis Steering Group, and the Corporate Mission Analysis Organization. FAA also has detailed procedures and associated templates for developing mission need statements. Type of practice: Activities; Key practice: The organization defines and documents business needs for both proposed and ongoing IT projects and systems; Rating: Executed; Summary of evidence: AMS policy calls for business needs for both proposed and ongoing IT projects and systems to be specified in the mission need statement. We verified that business needs were defined and documented in mission need statements for the three projects we reviewed. Type of practice: Activities; Key practice: The organization identifies specific users and other beneficiaries of IT projects and systems; Rating: Executed; Summary of evidence: FAA policy and procedures call for specific users and other beneficiaries of IT projects and systems to be identified. We verified that specific users and other beneficiaries were identified for the three projects we reviewed. Type of practice: Activities; Key practice: 3. Users participate in project management throughout an IT project's or system's life cycle; Rating: Executed; Summary of evidence: FAA policies and procedures call for users to participate in project management throughout an IT project's or system's life cycle. We verified that users participated in project management throughout the life cycle of the three projects we reviewed. Type of practice: Activities; Key practice: 4. The investment board periodically evaluates the alignment of its IT projects and systems with the organization's strategic goals and objectives and takes corrective actions when misalignment occurs; The investment board periodically evaluates the alignment of its IT projects and systems with the organization's strategic goals and objectives and takes corrective actions when misalignment occurs; Rating: Not executed; Summary of evidence: The JRC evaluates the alignment of systems through development and 2 years into operations with the organization's strategic goals and objectives--through the annual budget formulation process--and takes corrective actions when misalignment occurs. However, there is no process for the JRC to periodically evaluate the alignment of investments later in their life cycles. Source: GAO. [End of table] FAA Has a Disciplined Process for Selecting New IT Proposals but Lacks a Similar Process for Reselecting Ongoing Investments: Selecting new IT proposals and reselecting ongoing investments requires a well-defined and disciplined process to provide the agency's investment board, business units, and developers with a common understanding of the process and the cost, benefit, schedule, and risk criteria that will be used both to select new projects and to reselect ongoing projects for continued funding. According to ITIM, this critical process requires, among other things, (1) making funding decisions for new proposals according to an established process; (2) providing adequate resources for investment selection activities; (3) using a defined selection process to select new investments and reselect ongoing investments; (4) establishing criteria for analyzing, prioritizing, and selecting new IT investments and for reselecting ongoing investments; and (5) creating a process for ensuring that the criteria change as organizational objectives change. (The complete list of key practices is provided in table 6.): FAA has executed 7 of the 10 key practices associated with selecting an investment. For example, the AMS establishes two processes--mission analysis and investment analysis--that together constitute a set of policies and procedures, as well as guidance that is designed to enhance the agency's ability to select investments. In addition, FAA has policies and procedures for its annual F&E budget formulation process to reselect ongoing IT projects. Also, FAA's AMS sets forth policies and procedures for reselecting ongoing IT investments by identifying their capability shortfalls and addressing them as new investments. The AMS also integrates funding with the process of selecting an investment by requiring the Systems Engineering/Operational Analysis Team to perform affordability assessments for new proposed investment programs; it may recommend funding reallocations from lower priority programs when an alternative solution cannot be funded within FAA planning and budgeting baselines. This team also supports the JRC to ensure that the executives' funding decisions are aligned with selection decisions during the investment analysis activities. Resources for proposal selection activities include the program director, the Integrated Product Team, and the Investment Analysis Team, as well as detailed procedures and a template that have been defined for developing investment analysis reports. The investment analysis reports identify the evaluation criteria used, the alternatives analyzed, and the ranking of each alternative so that the JRC can select the best overall solution identified in the mission need statement. The criteria that were established during the initial investment analysis phase are used by the Investment Analysis Team to rank each proposed project on the basis of how well it meets the agency's mission needs compared with other projects. FAA uses the processes defined in the AMS for selecting new IT investments. In addition, it uses two processes to reselect ongoing IT investments. Specifically, the FAA uses its annual budget formulation process for projects in development or in the first 2 years of operations. It also uses the AMS process when a system's capability shortfall is identified, and it treats the correction of the shortfall as a new investment. The managers of the three projects we reviewed confirmed that their projects were selected using the AMS process. One project's officials stated that this included market, alternatives, investment, and affordability analyses. The program managers also stated that the annual F&E budget formulation process is used to reselect their projects. These project officials also noted that if a project is scheduled for a hardware replacement, a reselection is done. The AMS process is followed to explore new alternatives and make sure the replacement is in the best interest of the government. Despite these strengths, FAA has not developed similarly strong processes for NAS investments more than 2 years into their operations- -those NAS systems that are in the in-service management phase. For example, while FAA's F&E budget formulation process establishes criteria for analyzing, prioritizing, and reselecting IT investments for systems in development or up until 2 years into operations, neither of the two processes used to reselect IT investments has established criteria for investments beyond 2 years into operations. In addition, while FAA uses its annual budget formulation process to reselect projects that are part of the F&E budget, the agency does not have an analogous reselection process as part of its operations budget formulation. Until FAA establishes consistent criteria for reselecting all of its IT investments, it will not be adequately assured that it is consistently and objectively continuing to fund ongoing projects that still meet the needs and priorities of the agency in a cost-effective and risk-insured manner. Table 6 shows the rating for each key practice required to implement the critical process for selecting an investment at the Stage 2 level of maturity and summarizes the evidence that supports these ratings. Table 6: Selecting an Investment: Type of practice: Organizational commitments; Key practice: 1. The organization has documented policies and procedures for selecting new IT proposals; Rating: Executed; Summary of evidence: FAA's AMS policy, mission analysis, and investment analysis guidance has documented policies and procedures for selecting new IT proposals. Type of practice: Organizational commitments; Key practice: 2. The organization has documented policies and procedures for reselecting ongoing IT investments; Rating: Executed; Summary of evidence: FAA has documented policies and procedures for its annual F&E budget formulation process, which is used to reselect ongoing IT projects. In addition, FAA's AMS policy has documented policies and procedures for reselecting ongoing IT investments throughout the FAA's acquisition life cycle. Type of practice: Organizational commitments; Key practice: 3. The organization has documented policies and procedures for integrating funding with the process of selecting an investment; Rating: Executed; Summary of evidence: FAA's AMS and investment analysis guidance have documented policies and procedures for integrating funding with the process of selecting an investment. Type of practice: Prerequisites; Key practice: 1. Adequate resources, including people, funding, and tools, are provided for identifying and selecting IT projects and systems; Rating: Executed; Summary of evidence: Adequate resources are provided for identifying and selecting IT projects and systems. They include the program director, Integrated Product Teams, and the Investment Analysis Team. FAA also has detailed procedures and associated templates for developing investment analysis reports. Type of practice: Prerequisites; Key practice: 2. Criteria for analyzing, prioritizing, and selecting new IT investment opportunities have been established; Rating: Executed; Summary of evidence: The Investment Analysis Team has established criteria for analyzing, prioritizing, and selecting new IT investment opportunities. The investment analysis report, which is submitted to the JRC, identifies the evaluation criteria, the alternatives analyzed, and the ranking for each alternative. Type of practice: Prerequisites; Key practice: 3. Criteria for analyzing, prioritizing, and reselecting[A] IT investment opportunities have been established; Rating: Not executed; Summary of evidence: While FAA's F&E budget formulation process has established criteria for analyzing, prioritizing, and reselecting IT investments that are part of that budget, neither of the two processes used to reselect IT investment opportunities has established criteria for investments beyond 2 years into operations. Type of practice: Prerequisites; Key practice: 4. A mechanism exists to ensure that the criteria continue to reflect organizational objectives; Rating: Not executed; Summary of evidence: While FAA ensures that the criteria continue to reflect organizational objectives for selecting new IT investments, there are no consistent criteria used by the JRC to reselect investments more than 2 years into operations. Type of practice: Activities; Key practice: 1. The organization uses its defined selection process, including predefined selection criteria, to select new IT investments; Rating: Executed; Summary of evidence: FAA uses the policies and procedures defined in the AMS, mission analysis, and investment analysis guidance to select new IT investments. We verified that the three projects we reviewed were selected using the mission analysis and investment analysis activities defined in the AMS, mission analysis and the investment analysis guidance. Type of practice: Activities; Key practice: 2. The organization uses the defined selection process, including predefined selection criteria, to reselect[A] ongoing IT investments; Rating: Not executed; Summary of evidence: FAA has a process defined in the AMS to reselect ongoing IT investments. It also uses its annual budget formulation process to reselect projects that are part of the F&E budget. According to the project managers of the three projects we reviewed, this budget process is used to reselect their projects. However, FAA does not consistently use these defined processes to reselect IT investments more than 2 years into operations. Type of practice: Activities; Key practice: 3. Executives' funding decisions are aligned with selection decisions; Rating: Executed; Summary of evidence: The Systems Engineering/Operational Analysis Team, which is composed of representatives from FAA's service organizations, supports the JRC in making funding decisions that are aligned with selection decisions as part of FAA's investment analysis activities. Source: GAO. [A] According to the GAO ITIM framework, reselecting is the periodic reconsideration of an investment's continuing value to the organization and the decision to continue funding. It is a recurring process that continues for as long as a project is receiving funding. [End of table] FAA Does Not Have a Process for Effectively Overseeing Investments in All Phases of Their Life Cycles: An organization should provide effective oversight for its IT projects throughout all phases of their life cycles. Its investment board should maintain adequate oversight and observe each project's performance and progress toward predefined cost and schedule expectations as well as each project's anticipated benefits and risk exposure. The investment board should also employ early warning systems that enable it to take corrective action at the first sign of cost, schedule, or performance slippages. This board has ultimate responsibility for the activities within this critical process. According to ITIM, effective project oversight requires, among other things, (1) having written policies and procedures for management oversight; (2) developing and maintaining an approved management plan for each IT project; (3) making up-to-date cost and schedule data for each project available to the oversight boards; (4) having regular reviews by each investment board of each project's performance against stated expectations; and (5) ensuring that corrective actions for each underperforming project are documented, agreed to, implemented, and tracked until the desired outcome is achieved. (The complete list of key practices is provided in table 7.): FAA has in place 4 of the 7 key practices associated with effective project oversight. The agency has developed written policies and procedures for management oversight of its investments. These include (1) AMS; (2) the integrated program plan, which is the detailed planning document for all aspects of a program's implementation, including program control; and (3) the Integrated Baseline Establishment and Management Process document for reporting variances from the performance expectations approved by the JRC in the acquisition program baseline. We verified that cost, schedule, benefit, and risk expectations were documented in the acquisition program baseline and that the integrated program plan contained details for project execution for En Route Communications Gateway and FAA Telecommunications Infrastructure. For the VSCS Control Subsystem Upgrade, performance expectations and details on project execution were both captured in the integrated program plan.[Footnote 20] In addition, the JRC Secretariat Team maintains a tracking system for action items that are assigned during a project's acquisition reviews, including the action to be taken, the responsible FAA organization, and whether the underlying problem has been resolved. FAA has not established processes that bring investments before the JRC for oversight on a regular basis. There is a process for reporting variances from the performance expectations that were approved by the JRC in the investment's acquisition program baseline. However, although this process is carried out as part of the F&E budget formulation for IT investments in development or less than 2 years into operations, it is not being carried out for investments that are part of the operations budget. Investments that are meeting performance expectations may not return to the JRC for several years. FAA also conducts acquisition reviews as a means for program offices to report to agency executives on the status of investments compared to program baselines. However, since program offices may select which investments they wish to bring forward for review, many investments may never come forward. Until FAA develops (1) procedures for reporting on an investment throughout its entire acquisition life cycle and (2) mechanisms for ensuring that all investments are reviewed regularly, the agency is placing itself at risk that underperforming investments will not be reported to the JRC in order for it to take appropriate actions. Table 7 shows the rating for each key practice that is required to implement the critical process for project oversight at the Stage 2 level of maturity and summarizes the evidence that supports these ratings. Table 7: Providing Investment Oversight: Type of practice: Organizational commitment; Key practice: 1. The organization has documented policies and procedures for management oversight of IT projects and systems; Rating: Executed; Summary of evidence: FAA has developed written policies and procedures for management oversight of IT projects and systems. These include AMS, the integrated program plan, and the Integrated Baseline Establishment and Management Process document for reporting variances from the performance expectations approved in the acquisition program baseline for an investment program. Type of practice: Prerequisites; Key practice: 1. Adequate resources, including people, funding, and tools, are provided for IT project oversight; Rating: Executed; Summary of evidence: FAA has adequate resources for providing IT project oversight. The agency has staff for compiling monthly variance reports submitted by the investment program areas, preparing quarterly baseline variance reports for the JRC, and preparing semi-annual baseline variance report for the FAA Administrator. An automated system is used to facilitate the maintenance of information for these reports. Type of practice: Prerequisites; Key practice: 2. IT projects and systems, including those in steady state (operations and maintenance), maintain approved project management plans that include expected cost and schedule milestones and measurable benefit and risk expectations; Rating: Executed; Summary of evidence: AMS policy calls for an acquisition program baseline (APB) document and an integrated program plan to be available at the JRC final investment decision point. The APB document serves as the AMS cost/schedule/ technical performance/benefits/risks control document. The integrated program plan specifies how the APB baselines will be controlled and details the management, contracting, and technical actions and activities to be performed in executing the acquisition. We verified that cost, schedule, benefit, risk, and performance expectations were documented in the APBs for ECG and FTI. For VCSU, these expectations were documented in an integrated program plan. Type of practice: Activities; Key practice: 1. Data on actual performance (including cost, schedule, benefit, and risk performance) are provided to the appropriate IT investment board; Rating: Not executed; Summary of evidence: FAA has established a process for reporting variances to the JRC from the performance expectations that have been approved by the JRC in the APB for an investment. This process is carried out for IT investments that are part of the F&E budget, but it is not being carried out for investments that are managed as part of the operations budget. Type of practice: Activities; Key practice: 2. Using verified data, each investment board regularly reviews the performance of IT projects and systems against stated expectations; Rating: Not executed; Summary of evidence: FAA does not have a process that provides an opportunity for the JRC to regularly review investment performance. It has a process for conducting acquisition reviews where program offices provide status information to agency executives on the progress of investments against their acquisition program baselines. However, the individual program offices choose which investments they want to discuss at these reviews. Also, although the process for reporting variances from the performance expectations approved by the JRC in the acquisition program baseline is carried out for IT investments that are part of the F&E budget, this process is not being carried out for investment programs that are part of the operations budget, and it only results in investments with variances to be reviewed. Type of practice: Activities; Key practice: 3. For each underperforming IT project or system, appropriate actions are taken to correct or terminate the project or system in accordance with defined criteria and the documented policies and procedures for management oversight; Rating: Not executed; Summary of evidence: During acquisition reviews, action items are identified for investment programs discussed, an organization assigned responsibility to carry them out, and the items are tracked until the appropriate action is taken, at which time they are closed out. Similarly, variance reports are prepared quarterly for the JRC identifying investments with a 10 percent or greater variance from the established acquisition program baseline. An investment program is to remain on the quarterly variance report until successful corrective action is taken. However, FAA has no mechanism that provides assurance that every program has an acquisition review regularly, since it is left to the individual program offices to decide which programs they want discussed at the reviews. Also, although variance reports are prepared for IT investments that are part of the F&E budget, reports are not prepared for investments that are part of the operations budget. Type of practice: Activities; Key practice: 4. The investment board regularly tracks the implementation of corrective actions for each underperforming project until the actions are completed; Rating: Executed; Summary of evidence: The JRC's Secretariat Team maintains a tracking system for action items assigned during a project's acquisition reviews. This system identifies the action to be taken, what FAA organization is to perform it, and whether it is open or closed. Source: GAO. [End of table] FAA Has a Structured Process for Capturing Investment Information and Using It to Support Investment Management: To make good IT investment decisions, an organization must be able to acquire pertinent information about each investment and store that information in a retrievable format. During this critical process an organization identifies its IT assets and creates a comprehensive repository of investment information. This repository provides information to investment decision makers to help them evaluate the impacts and opportunities that would be created by proposed or continuing investments. It can provide insights and trends about major IT cost and management drivers. The repository can take many forms and does not have to be centrally located, but the collection method should identify each IT investment and its associated components. This critical process may be satisfied by the information contained in the organization's current enterprise architecture, augmented by additional information--such as financial information and information on risk and benefits--that the investment board may require to ensure that informed decisions are being made. According to ITIM, effectively managing this repository requires, among other things, (1) developing written policies and procedures for identifying and collecting the information, (2) assigning responsibility for ensuring that the information being collected meets the needs of the investment management process, (3) identifying IT projects and systems and collecting relevant information to support decisions about them, and (4) making the information easily accessible to decision makers and others. (The complete list of key practices is provided in table 8.): FAA's AMS guidance identifies specific information that is needed in the investment management process, including information for its investment analysis phase. FAA maintains a number of repositories of relevant information, including its Simplified Program Information Reporting & Evaluation database, which reports variances in cost, schedule, performance, or benefits from an investment's approved acquisition program baseline. The information that is collected is made available to the JRC in several documents, including program plans and the acquisition program baseline document. The JRC Secretariat Team ensures that the investment board has all the relevant information it needs for its decision-making process. Table 8 shows the rating for each key practice required to implement the critical process for capturing investment information at the Stage 2 level of maturity and summarizes the evidence that supports these ratings. Table 8: Capturing Investment Information: Type of practice: Organizational commitments; Key practice: 1. The organization has documented policies and procedures for identifying and collecting information about IT projects and systems to support the investment management process; Rating: Executed; Summary of evidence: FAA has developed policies and procedures for identifying and collecting information to support the investment management process. For example, AMS guidance indicates what information is needed for the investment analysis phase of FAA's investment management process. Type of practice: Organizational commitments; Key practice: 2. An official is assigned responsibility for ensuring that the information collected during project and systems identification meets the needs of the investment management process; Rating: Executed; Summary of evidence: AMS guidance specifies which officials are responsible for approving the completion of reports containing information prepared for the investment management process. The JRC Secretariat Team ensures through its readiness process that all of the necessary information is available to the JRC for its decision making. Type of practice: Prerequisite; Key practice: 1. Adequate resources, including people, funding, and tools, are provided for identifying IT projects and systems and collecting relevant investment information about them; Rating: Executed; Summary of evidence: FAA has adequate resources for meeting this key practice. Several teams, including a mission analysis team and an investment analysis team, collect the relevant investment information needed by the JRC to make its decisions on which investments to approve. The JRC Secretariat Team ensures that the JRC has all the relevant information for its decision making. Type of practice: Activities; Key practice: 1. The organization's IT projects and systems are identified, and specific information is collected to support decisions about them; Rating: Executed; Summary of evidence: FAA maintains data relevant to the investment management process in several sources, including its financial management system, its Simplified Program Information Reporting & Evaluation tool, and its Capital Investment Plan. Also, AMS guidance identifies information to be collected for the investment management process--including for the investment analysis phase--to aid the JRC in its final investment decisions. Type of practice: Activities; Key practice: 2. The information that has been collected is easily accessible and understandable to decision makers and others; Rating: Executed; Summary of evidence: Information collected for the JRC decision-making process is compiled in documents such as detailed program plans, acquisition program baselines, and investment analysis reports. The JRC Secretariat Team ensures that the JRC has all the relevant information for its decision making through its readiness process. Type of practice: Activities; Key practice: 3. The information repository is used by investment decision makers and others to support investment management; Rating: Executed; Summary of evidence: The JRC Secretariat Team, through its JRC readiness process, collects information for the JRC to use in its decision-making process. The JRC also receives an investment analysis report that contains all of the information that has been gathered during investment analysis activities. Source: GAO. [End of table] FAA Does Not Have Structured Processes to Manage Its Non-NAS Investments: FAA does not have a single set of processes for making consistent basic selection and control decisions for its non-NAS investments (Stage 2 capabilities). As previously discussed in the background section of this report, several business units within FAA make decisions about non-NAS investments. We reviewed the investment management processes of seven of these units--Information Services, Region and Center Operations, Regulation and Certification, Financial Services, Research and Acquisition, Air Traffic Services, and Human Resource Management. Appendix II describes the investment management processes we found in these units. The extent to which these processes comply with the ITIM framework for Stage 2 varies considerably by business unit, and FAA currently does not specify non-NAS investment management processes in a coordinated manner. Since the ITIM framework calls for a consistent investment management process, we assessed FAA's non-NAS investment management capability at an aggregate level. That is, we assessed FAA's capability to manage its non-NAS investments, not the capability of each individual business unit. Even though individual business units may have some of these processes in place, FAA as a whole has not yet defined: * an investment management structure that allows the agency to consistently manage its non-NAS investments, * a uniform process for ensuring that non-NAS investments are linked to business needs and meet users' needs, * a process for selecting new IT proposals and reselecting ongoing investments, * a single process for reviewing the progress of investments and taking corrective action when performance expectations are not being met, or: * a comprehensive inventory of project and system information to support investment decisions. According to FAA officials, the agency has not defined a coherent investment management structure and a set of processes for non-NAS investments in the past because many of these investments have not had the agencywide impact of the NAS investments. However, because there is now recognition that a disciplined approach to managing non-NAS investments could help control FAA's IT assets and costs in general, efforts are currently under way to address this weakness. As previously discussed, an IT Executive Board (ITEB) has been chartered with responsibility for, among other things, making decisions about non-NAS IT investments, but it has not yet taken action on developing a standard process. Until FAA fully establishes the consistent practices it needs to make basic project selection and control decisions, executives will be hampered in their ability to effectively manage non- NAS investments and ultimately to find the opportunities to achieve the cost savings they are seeking. FAA Lacks Key Capabilities Needed to Manage All IT Investments as a Portfolio and Does Not Conduct Postimplementation Reviews: During Stage 3, the investment board enhances the IT investment management process by developing a complete investment portfolio and carrying out PIRs. An IT investment portfolio is an integrated, agencywide collection of investments that are assessed and managed collectively on the basis of common criteria. Managing investments within the context of such a portfolio is a conscious, continuous, and proactive approach to expending limited resources on an organization's competing initiatives in light of the relative benefits expected from these investments. Taking an agencywide perspective enables an organization to consider its investments comprehensively, so that collectively the investments optimally address the organization's missions, strategic goals, and objectives. Managing IT investments with a portfolio approach also allows an organization to determine priorities and make decisions about which projects to fund, and continue to fund, based on analyses of the relative organizational value and risks of all projects, including projects that are proposed, under development, and in operation. For an organization to reap the full benefits of the portfolio process, it should collect all of its investments into an enterprise-level portfolio that is overseen by its senior investment board. Although investments may initially be selected into subordinate portfolios--based on, for example, lines of business or life cycle stages--and managed by subordinate investment boards, they should ultimately be aggregated into this enterprise-level portfolio. The purpose of a PIR is to evaluate an investment after its development has been completed (i.e., after its transition from the implementation phase to the in-service management phase) in order to validate actual investment results. This review is conducted to (1) examine differences between estimated and actual investment costs and benefits and their possible ramifications for unplanned funding needs in the future and (2) extract "lessons learned" about the investment selection and control processes that can be used as the basis for management improvements. Similarly, PIRs should be conducted for investment projects that were terminated before completion, to help to readily identify potential management and process improvements. According to ITIM, critical processes performed by Stage 3 organizations include (1) defining the portfolio criteria, (2) creating the portfolio, (3) evaluating the portfolio, and (4) conducting PIRs. Table 9 shows the purpose of each critical process in Stage 3. Table 9: Stage 3 Critical Processes--Developing a Complete Investment Portfolio: Critical process: Defining the portfolio criteria; Purpose: To ensure that the organization develops and maintains IT portfolio selection criteria that support its mission, organizational strategies, and business priorities. Critical process: Creating the portfolio; Purpose: To ensure that IT investments are analyzed according to the organization's portfolio selection criteria and that an optimal IT investment portfolio with manageable risks and returns is selected and funded. Critical process: Evaluating the portfolio; Purpose: To review the performance of the organization's investment portfolio(s) at agreed-upon intervals and to adjust the allocation of resources among investments as necessary. Critical process: Conducting postimplementation reviews; Purpose: To compare the results of recently implemented investments with the expectations that were set for them and to develop a set of lessons learned from these reviews. Source: GAO. [End of table] FAA has executed only 1 of the 27 key practices associated with Stage 3 critical processes: it has a process for distributing portfolio criteria to project management personnel and other stakeholders. The remaining 26 key practices were not executed--primarily because FAA does not involve the JRC in the regular oversight of non-NAS investments or in NAS investments during their in-service management phase, weaknesses that we noted in our assessment of Stage 2 requirements. Since Stage 3 requires an enterprisewide perspective, the lack of oversight of these classes of investments precludes the successful completion of most Stage 3 critical processes. In addition, Stage 3 requires an enterprisewide perspective that FAA has not adopted, which would enable the JRC to oversee all major IT investments, regardless of life cycle phase or business unit. Although it can be appropriate for FAA to manage its NAS, in-service NAS, and non-NAS investments as separate subordinate portfolios--depending on the successful execution of all Stage 2 key practices--its enterprise- level portfolio should contain all major IT investments regardless of life cycle stage or business line. In building this enterprise-level portfolio, the JRC can choose whether to include specific investments based on predetermined criteria, as described by the ITIM framework. Until FAA fully implements the critical processes associated with managing its investments as a complete portfolio, it will not have the data or enterprisewide perspective it needs to make informed decisions about all of its major IT investments. In addition, FAA has not executed the six key practices for conducting PIRs. In June 2004, in response to a recommendation contained in our 1999 report[Footnote 21] that FAA initiate PIRs for projects or programs within 3 to 12 months of deployment or termination, the NAS Configuration Management and Evaluation Staff developed a proposed approach to PIRs, but this approach was not implemented. In November 2003, the life cycle management policy team proposed a change to the AMS that would require conducting these reviews, but there has been no action on the proposal. Although the JRC has recently reaffirmed its commitment to implement PIRs, there is no policy and no established process to carry them out. If PIRs are not conducted on a routine basis, then FAA will not be able to effectively evaluate the results of its IT investments; this will affect the agency's ability to determine whether to continue, modify, or terminate an IT investment in order to meet its stated mission objectives. Table 10 summarizes the status of FAA's critical processes for Stage 3, showing how many associated key practices it has executed. Table 10: Status of Stage 3 Critical Processes: Critical process: Defining the portfolio criteria; Key practices executed: 1; Total required by critical process: 7; Percentage of key practices executed: 14. Critical process: Creating the portfolio; Key practices executed: 0; Total required by critical process: 7; Percentage of key practices executed: 0. Critical process: Evaluating the portfolio; Key practices executed: 0; Total required by critical process: 7; Percentage of key practices executed: 0. Critical process: Conducting postimplementation reviews; Key practices executed: 0; Total required by critical process: 6; Percentage of key practices executed: 0. Totals; Key practices executed: 1; Total required by critical process: 27; Percentage of key practices executed: 4. Source: GAO. [End of table] FAA Has Not Established a Process for Managing the Succession of Its Information Systems: Once an agency has attained Stage 3 maturity, it evaluates its IT investment processes and portfolios to identify opportunities for improvement (Stage 4 capabilities). This entails (1) improving the portfolio's performance and (2) managing systems and technology succession. We did not assess FAA's capability for improving the portfolio's performance, because it did not claim to be executing any of the relevant key practices in its self-assessment. According to ITIM, regarding system and technology succession management includes (1) defining policies and procedures for managing the IT succession process, (2) assigning responsibility for the IT succession process, (3) developing criteria for identifying IT investments that may meet succession status, and (4) periodically analyzing IT investments to determine whether they are ready for succession. This critical process enables an organization to recognize low-value or high-cost IT investments and augments the routine replacement of systems at the end of their useful lives. It also promotes the development of a forward-looking, solution-oriented view of IT investments that anticipates future resource requirements and allows the organization to plan appropriately. This process differs from the reselection activity in Stages 2 and 3 in that it focuses on anticipating and planning for the retirement of legacy systems and on meeting remaining requirements with other, perhaps new, systems. In addition, succession management takes place at the end of a system's life cycle. FAA has not executed any of the nine key practices required to implement this critical process. Although the agency has defined procedures in AMS for retiring investments, it still needs to describe how to regularly review systems that are in operations in order to identify candidates for retirement. According to FAA, decisions on succession are made by the service organizations. However, no individual or group has been assigned responsibility for managing the succession process from an enterprise perspective, which would allow the FAA to better anticipate and plan for future resource requirements. Without an institutionalized process for succession management, the FAA may not be able to identify those IT investments that are eligible for succession in enough time to minimize the effect of the transition on their successors. In addition, by establishing an effective succession management process, the agency can identify systems for retirement, freeing resources for other, superior, investments. FAA Has Initiated Efforts to Improve Its Investment Management Process: We have previously reported that to effectively implement IT investment management processes, organizations need to be guided by a plan that (1) is based on an assessment of strengths and weaknesses; (2) specifies measurable goals, objectives, and milestones; (3) specifies needed resources; (4) assigns clear responsibility and accountability for accomplishing tasks; and (5) is approved by senior management. FAA has begun to take steps to resolve some of the weaknesses identified in this report. For example, at a June 10, 2004, meeting, the JRC decided to incorporate budget justification documents (Exhibit 300s), which are currently prepared for the Office of Management and Budget (OMB) as part of the President's Budget formulation process, into the AMS process for managing NAS investments. The Exhibit 300 will become the board's decision-making document, and essential information from existing AMS-required documents--the investment management report, the acquisition strategy paper, the integrated program plan, and the requirements documents--will be incorporated into the Exhibit 300. The JRC also recently decided to implement PIRs in order to track metrics during program implementation. Finally, at that same meeting, the board decided to collectively determine, at the meeting where the F&E budget is approved, which F&E and OPS programs should be brought forward for review the following year. This decision serves to bring certain investments in the in-service management phase under the JRC's direct purview, although it does not specify that consistent criteria be established, as the ITIM framework requires. FAA has also begun to initiate steps to bring more clarity to the ITEB's responsibilities, although the specifics have yet to be defined. In its charter, the ITEB is charged with making investment decisions about non-NAS IT investments. This action would begin to bring all of the non-NAS investments under a single authority. The charter suggests that the ITEB choose among three options: (1) to send major non-NAS investment decisions to the JRC, (2) to make the decision itself, given an acceptable review process similar to the JRC processes, or (3) have the CIO, Chief Financial Officer, and owning assistant/associate administrator make the decision jointly. This description of the ITEB's roles and responsibilities further alludes to the senior board's evolving responsibility toward major non-NAS IT investments, although it falls short of laying out specific criteria for selecting which investments should be sent forward to the JRC. The ITEB has been given responsibility for four short-term initiatives as well, including establishing an agencywide cost control program for non-NAS expenditures and ensuring that all OMB Exhibit 300s receive a passing grade for the 2006 budget year. The ITEB has been charged with the long-term initiative of clearly delineating the roles it plays and its relationship with the more senior board. The successful completion of this initiative is likely to satisfy the single key practice that FAA has not yet executed in the Instituting the Investment Board critical process of the ITIM. The Chief Operating Officer's recent reorganization of the ATO is intended to make the heads of the service units responsible for IT projects from their inception through the in-service management phase. This new organization is designed to support his expressed intentions to increase accountability for systems in operation in order to manage costs more effectively. According to the Chief Operating Officer, FAA recognizes that good processes are needed for both NAS and non-NAS to improve the way the agency manages its investments. While FAA has initiated these improvement efforts, it has not linked them together in a plan with the characteristics listed above that would help coordinate and guide the efforts. Until FAA develops a plan that would allow for the systematic prioritization, sequencing, and evaluation of improvement efforts, the agency risks not being able to effectively establish mature investment management processes. DOT is Taking Steps to Integrate Oversight of FAA's IT Investments: DOT has recently initiated several efforts that can serve to provide better departmental oversight of FAA investments. This fiscal year DOT and FAA reached an agreement by which DOT reviews FAA's Exhibit 300s as part of the department's annual budget process, in which all departmental components participate. Under this agreement, DOT conducts a review of all FAA Exhibit 300s starting in June of each budget year and culminating in the review of all Exhibit 300s by the Department Investment Review Board in late August, prior to the submission of the budget to OMB in September. As part of this agreement, DOT has outlined a process and schedule for reviewing the fiscal year 2006 budget justifications for major FAA programs and is monitoring FAA's progress in meeting this schedule. In addition, the department has identified about a dozen programs that it plans to monitor regularly and has begun reviewing these programs through its senior investment management decision-making board, on which the FAA Administrator is a voting member. DOT has also requested that FAA set reasonable expectations for cost, schedule, and performance for its major projects and that it then report quarterly on variances to those expectations. FAA submitted its first quarterly report as of June 2004. These regular reports are intended to help DOT maintain oversight of FAA's processes and ensure that they are appropriate and consistent with OMB's requirements. Furthermore, the department is currently planning to issue an investment management guide that specifies minimum expectations that its operating administrations (including FAA) are to follow in managing their investments. According to DOT officials, FAA has been complying with the department's requests for information to facilitate its oversight process. Department officials are attributing their increased oversight--and cooperation from FAA--to the fact that the department has recently reinstituted its own investment management processes. In addition, DOT officials said that FAA now understands the role the department can play in helping it to obtain the funding it needs for its programs. Conclusions: FAA has established most of the project selection and control capabilities needed to manage its NAS investments. This should help provide the executive-level decision-making and oversight capabilities required to establish accountability and guide major IT investments through most of their life cycles. However, weaknesses remain. For example, although business units are involved in the regular review of investments throughout their life cycles, the JRC may not review the performance of operations systems for several years unless they require significant additional funds. Also, FAA has yet to define and implement the practices it needs to select and control its non-NAS investments. Ultimately, because the JRC does not regularly review NAS systems during the in-service management phase and does not regularly review the non-NAS systems in general, significant portions of FAA's approximately $2.5 billion investment in IT go without top-level executive oversight and are not viewed as part of an enterprisewide portfolio. FAA has taken some initial steps to implement PIRs, but it has not yet established a process to carry them out. The agency has begun to take some steps to develop improvements to address some of these weaknesses, such as establishing an Information Technology Executive Board with relevant responsibilities. In addition, the JRC has begun integrating some budgeting and oversight processes, and the Chief Operating Officer has begun to articulate a vision that includes additional accountability for investments in operations. But FAA has not developed a comprehensive plan to guide all improvement efforts. Such a plan would help coordinate and prioritize improvement efforts and help sustain commitment to the efforts under way. The increasing collaboration between FAA and DOT further contributes to the likelihood that the management of FAA's investments will improve as FAA's Exhibit 300s have the benefit of department-level review and the departmental investment review board conducts periodic reviews of selected projects. Recommendations for Executive Action: To strengthen FAA's investment management capability and address the weaknesses discussed in this report, we recommend that the Secretary of the Department of Transportation direct the FAA Administrator to develop and implement a plan for improving FAA's IT investment management processes. The plan should address the weaknesses described in this report, beginning with those we identified in our Stage 2 analysis and continuing with those we identified in our Stage 3. The plan should also draw together ongoing efforts as well as instituting new initiatives where called for. The plan should, at a minimum, provide for accomplishing the following: In Stage 2: * Define procedures for aligning the JRC and the newly established ITEB. * Establish a process for the JRC to periodically reevaluate the alignment of projects in the in-service management phase with strategic goals and objectives. * Establish a process for the JRC to regularly review the performance of IT systems throughout their life cycles and take corrective actions when expected performance is not being met. * Define and implement an IT investment management structure, including an investment management board and a disciplined process for managing all non-NAS investments. In Stage 3: * Define and implement processes for managing major investments as part of an enterprise-level portfolio, including NAS F&E investments, NAS investments in the in-service management phase, and non-NAS investments. * Define and implement processes for carrying out PIRs on investments as they enter the in-service management stage. In developing the plan, the FAA Administrator should ensure that it (1) specifies measurable goals, objectives, and milestones; (2) specifies needed resources; (3) assigns clear responsibility and accountability for accomplishing tasks; and (4) is approved by senior management. In implementing the plan, the FAA Administrator should ensure that the needed resources are provided to carry out the plan and that progress is measured and reported periodically to the Secretary of Transportation. Agency Comments: In commenting on a draft of this report, DOT's Director of Audit Relations stated via e-mail that DOT appreciated the opportunity to review and offer comment on our report and that GAO had done a good job keeping the report balanced and fair, showing where FAA has many capabilities in place and identifying areas that need improvement. The Director also provided a technical comment, which we have incorporated into the report. As agreed with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from the date of this letter. At that time, we will send copies to other interested congressional committees, the Director of the Office of Management and Budget, the Secretary of Transportation, FAA's Administrator and CIO, and other interested parties. We also will make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at [Hyperlink, http://www.gao.gov]. Should you or your offices have questions on matters discussed in this report, please contact me at (202) 512-9286 or Lester P. Diamond, Assistant Director, at (202) 512-7957. We can also be reached by e-mail at [Hyperlink, pownerd@gao.gov], or [Hyperlink, diamondl@gao.gov], respectively. Key contributors to this report were William G. Barrick, Niti Bery, Joanne Fiorino, Michael Giannone, Sabine R. Paul, and Nik Rapelje. Signed by: David A. Powner: Director, IT Management Issues: [End of section] Appendixes: Appendix I: Objectives, Scope, and Methodology: The objectives of our review were to (1) evaluate FAA's capabilities for managing its IT investments, (2) determine what plans the agency might have for improving these capabilities, and (3) describe how DOT oversees FAA's investments and investment process. Because FAA told us that it managed its NAS and non-NAS investments differently, we performed separate assessments for the practices to evaluate FAA's capabilities for managing IT investments. To address the first objective, for the NAS investments we reviewed the results of the agency's self-assessment of Stages 2, 3, and 4 practices using GAO's ITIM framework[Footnote 22] and validated and updated the results of the self-assessment through document reviews and interviews with officials. We reviewed written policies, procedures, and guidance and other documentation providing evidence of executed practices, including FAA's Acquisition Management System guidance, mission analysis and investment analysis guidance, and memorandums. We also reviewed JRC guidance and records of decision, acquisition review guidance and meeting minutes, and variance reporting procedures and reports. We did not assess FAA's progress in establishing the capabilities found in one of the two Stage 4 critical processes, entitled Improving the Portfolio's Performance, or in any of the Stage 5 critical processes, because FAA acknowledged that it had not executed any of the key practices in these critical processes. For the non-NAS investments, we reviewed the results of FAA's self-assessments of Stage 2 practices using GAO's ITIM framework and conducted interviews to clarify and update the results. We did not perform a detailed assessment of these practices because they most likely will be superseded by a new process (when it is defined) for managing non-NAS investments, and non-NAS investments are of lower cost and impact to FAA. As part of our analysis, we selected three IT projects as case studies to verify that the critical processes and key practices were being applied. We selected projects that (1) supported different FAA functional areas, (2) were in different life cycle phases, and (3) required different levels of funding. The three projects are described below: * FAA Telecommunications Infrastructure (FTI)--FTI is a performance- based telecommunications services contract for voice, video, and data point-to-point support for telecommunications for the National Airspace System and its support system. It contributes to both the separation of aircraft (the mission-support network) and other FAA uses (the operational network, e.g., e-mail and phone). FTI will replace the current telecom system. FTI will eliminate the need for other subnetworks, of which there are currently eight or nine, and therefore eliminate the management overhead associated with operating so many networks. The integration of multiple networks and subnetworks will provide a single source and single vehicle for telecom. FTI is in the Technical Operations unit and has estimated life cycle costs of $2 billion. The contract for FTI was awarded in June 2002. * En Route Communications Gateway (ECG)--ECG is a mission critical gateway, or interface, for data from radar sites to Air Route Traffic Control Centers. ECG will serve as a single domain communications gateway and will provide the path for exchanging flight plan data from outside sources and transfer data among systems. ECG provides a commercial-off-the-shelf nondevelopmental item digital gateway using a modern, open and extensible platform consisting of modular scalable hardware components. ECG will incorporate interface capability to support legacy and future systems and will provide the capability to transition to modern network communications and access more surveillance sources. The flexibility provided by the ECG system architecture will facilitate the evolution of the En Route domain modernization. ECG will replace the Peripheral Adapter Module Replacement Item system and provide a modern domain gateway that will support the current and future En Route infrastructure. ECG is in the En Route & Oceanic Service group and has estimated life cycle costs of $442.5 million through September 2015. * Voice Switching and Control System (VSCS)--In our review of the VSCS program, we focused our review on one of VSCS's subcomponents, the VSCS Control Subsystem Upgrade (VCSU). The VCSU program, part of the Technical Operations Communications service group, is designed to maintain overall supportability of VSCS[Footnote 23] by replacing the hardware for the existing control subsystem, associated VSCS operational and application software, required software licenses, and supporting software and hardware documentation. Deliverables for the VCSU program include all hardware, spare parts, software, software licenses, system baseline documentation, training, and other technical documentation necessary to support the product at 21 locations. According to FAA, the VCSU program has a funding baseline of over $59 million and is in the operations and maintenance phase. For these projects, we reviewed project management documentation, such as mission needs statements, acquisition program baselines, and integrated program plans. We also interviewed the project managers for these projects. We compared the evidence collected from our document reviews and interviews to the key practices in ITIM. We rated the key practices as "executed" on the basis of whether the agency demonstrated (by providing evidence of performance) that it had met the criteria of the key practice. A key practice was rated as "not executed" when we found insufficient evidence of a practice during the review or when we determined that there were significant weaknesses in FAA's execution of the key practice. To address our second objective, we obtained and evaluated documents showing what management actions had been taken and what initiatives had been planned by the agency. This documentation included JRC records of decisions, the agency's capital investment guidance, and the recently formed ITEB charter and meeting minutes. We also interviewed the Chief Information Officer, other members of the JRC, and the Chief Operating Officer to determine what efforts FAA had undertaken to improve IT investment management processes. To address our third objective, we reviewed documentation on DOT's process for reviewing FAA's budget proposals and capital planning and investment control reviews. We also conducted interviews with both FAA and DOT officials, including DOT's CIO and Director for Capital Planning and Investment Control to determine DOT's oversight role in FAA's investments and investment management processes. We conducted our work at FAA Headquarters in Washington, D.C., from October 2003 through July 2004, in accordance with generally accepted government auditing standards. [End of section] Appendix II: Investment Management Process Used by Some Organizational Units to Manage Non-NAS Investments: Financial Services (ABA): Instituting the investment board; ABA has an investment board that conducts periodic and monthly program reviews for all IT programs to determine whether a program will be approved as an IT investment. A life cycle process guide is now in place to direct the activities of the investment board along with providing oversight of IT projects within ABA. Meeting business needs; The business needs of a project within ABA, along with the dates for achieving them, need to be aligned with the strategic goals established in the FAA Flight Plan. Projects or systems that are no longer aligned with the Flight Plan will be decommissioned; A project management plan identifies, among other things, the system's users, customers, and types of services to be provided. Selecting an investment; Selecting and reselecting an IT investment within ABA involves both the executive management team and ABA's CIO team. The executive management team reviews the business needs of the investment and compares them against the ABA's IT budget, while ABA's CIO team is involved with the selecting and reselecting processes by analyzing the technical costs associated with the IT investment and comparing those technical costs against the ABA's IT budget. Providing investment oversight; ABA uses its life cycle process guide to help manage its $25 million IT budget, which consists of 22 or 23 financial systems, 5 or 6 of them considered major programs under OMB's definition of a major IT investment. A requirement of the life cycle process guide is for every critical system in ABA to have a detailed project management plan that addresses performance measures such as cost, schedule, benefits, and risks; The day-to-day progress of IT projects is tracked against critical milestones that have been already established through weekly summary reviews with IT staff. For major IT projects, biweekly meetings are conducted that address any concerns with meeting the performance measures. Capturing the investment information; ABA captures its IT asset information using its Information Technology Investment Portfolio System (ITIPS), which is available to all ABA management and system support personnel. The information in ITIPS is used to manage projects that are in production as well as ensuring that the life cycle activities are in alignment with FAA's mission statements. Research and Acquisitions (ARA): Instituting the investment board; ARA uses its Operations Resource Management Team guide to select, control, and evaluate ARA IT investments. The team composed of representatives from ARA service units. ARA investments are controlled and tracked through quarterly reviews. These reviews look at the cost, schedule, and overall performance of the investment. Meeting business needs; The business needs for ARA investments need to be mapped back to the Flight Plan. A monthly status review report is prepared in order to ensure that the business needs are tracking back to the Flight Plan. Selecting an investment; ARA does not have any well-defined selection criteria since each program uses its own configuration management plan. ARA Ops build process guides the establishment of new projects. Providing investment oversight; A project plan does exist, along with established expenditures, which the program managers submit to the ARA CIO on a monthly basis. These monthly status reports occur between the CIO and the program managers to decide if an investment's resources, such as funding, need to be reallocated. Once the CIO and program manager decide that it is necessary for an investment's resources to be reallocated, the CIO will discuss the need further with the Deputy Associate Administrator for ARA, who ultimately will determine whether a program will receive additional resources, such as funding; With respect to the level of interaction that ARA has had with the JRC in the past, only one program from ARA, NextGen, has gone before the JRC. According to the ARA CIO, in order for a program to go to the JRC, there must be justification made to the council that the program is fully operational and is considered to be a benefit and a priority to FAA. The ARA Deputy Associate Administrator will determine if a program should go before the JRC for approval and funding. Capturing investment information; The configuration control board uses a database to capture asset inventory data about the systems that are owned by the ARA CIO. According to the ARA CIO, in order for IT assets to be effectively managed in ARA, there needs to be vision from AIO about what programs to invest in over the next 5 years. Air Traffic Services (ATS): Instituting the investment board; The Information Resource Management Executive Board is responsible for selecting, controlling, and evaluating ATS IT investments. Meeting business needs; Not all services within ATS have defined their business needs. Even though ATS has the NAS Support Integration Process (NSIP) data repository available for capturing IT asset information, including business needs, and for defining system users, there is no consistency in terms of the records being complete because there are systems within ATS that have not registered with NSIP. Selecting an investment; The ATS CIO manages the selection process, which begins with the NSIP registration criteria. Providing investment oversight; Each business unit within ATS has its own project management plan and procedures. The day-to-day tracking of projects as well as the monitoring of whether corrective actions are being executed is also the responsibility of the individual business units. Even though the individual business units are tasked with this level of responsibility, the ATS CIO does play an oversight role by setting the criteria and policies for the investments to be made for the projects. Capturing investment information; ATS uses the NSIP meta data repository to collect any changes to the IT projects and systems by providing a full declaration of the project or system. This includes providing information to help ATS avoid unwanted costs due to systems having redundant functionality and determining whether a system's or a project's functions match the stated mission goals for ATS. NSIP also handles the technical rollover for ATS systems or projects. Information Services (AIO): Instituting the investment board; AIO's investment management process can be characterized as iterative and well managed, but undocumented. The AIO Business Plan and IT Strategy are used to ensure that when funds are appropriated and allocated that they map back to the Flight Plan. Investments are controlled or tracked by the Deputy CIO on a monthly basis to get an indication as to where the program is in the process against the expenditures that have been already established. Weekly meetings are held with the unit's CIO to discuss any issues regarding AIO's investment management process. Meeting business needs; AIO does not have any written policies or procedures for identifying business needs for its IT projects. Only one of its major projects, NAS Adaptation Service and Environment, has documented its requirements, which includes specific users. Selecting an investment; AIO uses an undocumented process for reviewing new IT proposals to reach an agreement on selection. Providing investment oversight; There are no AIO-wide policies or procedures for managing projects or investment oversight. The Information Technology Executive Board (ITEB)[A] has been formed to provide a governing structure for non-NAS programs. One of the targets for ITEB is to look at cost control and cross-cutting IT initiatives by involving the heads of the lines of business. The ITEB is also going to be involved with improving the scores on the Exhibit 300 business cases for OMB. Capturing the investment information; AIO uses ITIPS to track its asset inventory and IT investments. The Deputy CIO of AIO is responsible for ensuring that the inventory located in ITIPS meets the needs of AIO's investment management process. According to AIO, the information within ITIPS is updated at least twice a year. Human Resource Management (AHR): Instituting the investment board; AHR does not have an investment board. Instead, AHR's senior management[B] is responsible for selecting, controlling, and evaluating all IT investments by using established agency acquisition policies and procedures to conduct investment management decisions. Meeting business needs; Business needs and specific users for each project are identified within the project plan and are aligned with the AHR Strategic Plan, the FAA Flight Plan, and the AIO Plan. AHR is also aligning its business needs to the ITEB plans. Business needs are re- evaluated on a quarterly basis to ensure that a project is aligned with FAA's strategic goals and objectives. Selecting an investment; AHR senior management uses its prioritization process to evaluate and select investments for funding. The office and center directors determine their requirements and then a budget request is submitted for proposal funding. AHR receives an allowance amount from the budget office. The first priority is to handle personnel payments. The remaining balance is then redistributed to the business divisions. The "building blocks" process starts at this point. This is when base funding is reviewed to decide if a current investment needs continued funding by asking questions about the importance of continuing the funding of a particular project by looking at the project activity and what the impact will be if this project is no longer funded. Each division will submit a list of prioritized projects with costs to the directorate. This list may exceed the budget level. The directorate will reprioritize the original list. Providing investment oversight; AHR has a Human Resource Management Automation Plan that contains procedures for approving IT projects, and describes the policies and procedures that AHR uses for project management. Despite having project management policies and procedures, not all projects within AHR have a formal project plan. The size and scope of the project are two factors that help determine whether a project has a formal project plan. AHR Division Managers ensure that projects are on time by performing quarterly reviews that assess a project's cost and schedule. AHR uses a color scheme (red, green, and yellow) to indicate the schedule status of major milestones. Capturing the investment information; AHR uses the ITIPS as its inventory for making investment management decisions. AHR projects are listed in ITIPS, along with business cases. Regions and Center Operations (ARC): Instituting the investment board; The IT Configuration Management Board is ARC's investment review board. The board's charter has recently been redone to provide more traceability back to the Flight Plan. The board functions include evaluating potential IT investment options for ARC, making recommendations on IT investment, establishing ARC-wide IT standards, and developing and maintaining investment policies and procedures. The board is led by the unit's CIO and includes four IT managers from the regional offices and aeronautical center and two members from the ARC Management Team. The ARC Management Team makes the final-selection decisions. The IT investment management decisions are then incorporated into the ARC Business Plan. The ARC unit is also involved with cross-organizational investment decisions for FAA through its membership on the FAA CIO Council. Meeting business needs; Business needs are identified through entries made in ITIPS, along with documentation from Exhibit 300s and Exhibit 53s. Selecting an investment; ARC does not have its selection criteria documented. To evaluate and select IT investments, the ARC IT Configuration Management Board considers such things as benefits to ARC across the regions, expected return on investment, technical feasibility, and risk. The ARC business plan and the Flight Plan are the documents that address these priorities. Providing investment oversight; ARC does not have policies or procedures for project management. Instead, ARC uses a weekly teleconference to address expectations and progress of ARC-wide IT initiatives at the IT manager level across ARC. According to the ARC CIO, a second teleconference has been added to discuss portfolio management--schedule, budget, training, and deployment along with whether the project will be integrated with other lines of business. Capturing the investment information; ARC uses ITIPS as its standardized repository for collecting asset information that will be useful for ARC's IT investment management decisions by providing information about what types of systems and functions are available and how they are supporting a specific business issue. Regulation and Certification (AVR): Instituting the investment board; Similar to an IT investment board, AVR has a two-tiered management structure that is composed of the AVR management team and the CIO management team. The AVR management team includes the Associate Administrator and the Service Directors who make the final decisions based upon recommendations and input from the CIO management team and its business partners from each of the service units. According to AVR, its IT investment process guide is still under development and will be completed at the end of Fiscal Year 2004. Meeting business needs; Each line of business within AVR identifies and documents its business needs including project requirements and specific users. Once the business needs have been identified, the IT Management and Resources section prioritizes them for funding. Selecting an investment; Programs in AVR are reviewed quarterly. For major projects, meetings are designed to look at project milestones to see if they are being met. These meetings are carried out biweekly and presented to the AVR management team. Providing investment oversight; The AVR CIO management team is responsible for monitoring projects and reporting to the AVR Management team. Biweekly meetings are held for major projects within AVR. Capturing the investment information; AVR's system inventory is a part of its enterprise architecture. The system inventory is being used primarily in developing the Exhibit 300s. The performance of IT projects in AVR is monitored daily, based upon each project's individual plan, using project management tools such as MS Project. According to AVR, not all projects have a project plan in place, but AVR is trying to make it a requirement. Source: GAO, based on information from FAA. [A] ITEB is a board that can provide a governing structure so that information technology is used as an agency wide strategic asset. [B] Composed of Assistant Administrator; two Deputy Assistant Administrators; three Office Directors; the Director, Center for Management and Development; and the AHR Business Officer. [End of table] (310456): FOOTNOTES [1] See GAO, High-Risk Series: An Overview, GAO/HR-95-1 (Washington, D.C.: February 1995); GAO, High-Risk Series: Information Management and Technology, GAO/HR-97-9 (Washington, D.C.: February 1997); GAO, High- Risk Series: An Update, GAO/HR-99-1 (Washington, D.C.: January 1999); GAO, High-Risk Series: An Update, GAO-01-263 (Washington, D.C.: January 2001); and GAO, High-Risk Series: An Update, GAO-03-119 (Washington, D.C.: January 2003). [2] GAO, Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity, GAO-04-394G (Washington, D.C.: March 2004). [3] The 10 service units that make up the ATO include Safety, Communications, Operations Planning, Finance, Acquisition & Business Services, En Route and Oceanic Services, Terminal Services, Flight Services, System Operations Services, and Technical Operations Services. [4] FAA uses three types of facilities to control traffic: airport towers, terminal radar approach control facilities, and en route centers. Airport towers direct traffic on the ground, before landing, and after takeoff within 5 nautical miles from the airport and about 3,000 feet above the airport. Terminal radar approach control facilities sequence and separate aircraft as they approach and leave airports, beginning about 5 nautical miles and ending about 50 nautical miles from the airport and generally up to 10,000 feet above the ground. Air route traffic control centers, called en route centers, control planes in transit and during approaches to some airports, generally controlling air space that extends above 18,000 feet for commercial aircraft. [5] GAO-03-119. [6] GAO, Air Traffic Control: FAA's Modernization Investment Management Approach Could Be Strengthened, GAO/RCED/AIMD-99-88 (Washington, D.C.: Apr. 30, 1999); GAO, Air Traffic Control: FAA's Modernization Efforts- -Past, Present, and Future, GAO-04-227T (Washington, D.C.: Oct. 30, 2003). [7] 49 U.S.C. 40110(d). [8] May be delegated to an associate administrator. [9] These teams may operate as entities or be organized into subintegrated product teams or product teams to develop, procure, and deliver products and services for users or customers. They are responsible for the acquisition of new or improved capability for services and products throughout their life cycles and for developing cost and schedule baselines for candidate solutions during investment analysis. [10] The investment analysis staff assists and oversees the work of all the investment analysis teams, is responsible for all investment analyses, and is responsible for developing the tools, techniques, and databases to ensure quality performance of investment analysis on behalf of the JRC. [11] A service organization is any organization within FAA that delivers a service, whether it is a business unit, project office, program directorate, or integrated product team, or whether it is engaged in air traffic services, security, regulation, certification, operations, commercial space transportation, or airport development. These organizations are responsible and accountable for managing service delivery throughout the life cycle. Investment decisions are made to support service delivery. Specifically, after the investment decision has been made, the service organization assumes responsibility for the investment program, implements the selected solution, and manages the product throughout the in-service management phase of its life cycle. [12] Non-NAS business units include Information Services (AIO), Region and Center Operations (ARC), Regulation and Certification (AVR), Financial Services (ABA), Research and Acquisition (ARA), Air Traffic Services (ATS), and Human Resource Management (AHR). [13] GAO-04-394G. [14] GAO, Information Technology: DLA Needs to Strengthen Its Investment Management Capability, GAO-02-314 (Washington, D.C.: Mar. 15, 2002); GAO, United States Postal Service: Opportunities to Strengthen IT Investment Management Capabilities, GAO-03-3 (Washington, D.C.: Oct. 15, 2002); GAO, Information Technology: Departmental Leadership Crucial to Success of Investment Reforms at Interior, GAO-03-1028 (Washington, D.C.: Sept. 12, 2003); and GAO, Bureau of Land Management: Plan Needed to Sustain Progress in Establishing IT Investment Management Capabilities, GAO-03-1025 (Washington, D.C.: Sept. 12, 2003). [15] An IT investment board is a decision-making body, made up of senior program, financial, and information managers, that is responsible for making decisions about IT projects and systems based on comparisons and trade-offs among competing projects, with an emphasis on meeting mission goals. [16] 49 U.S.C. 40110(d). [17] An advisory group, composed of representatives from each line of business, that establishes guidelines for conducting mission analysis and developing mission need statements as well as resolving agencywide mission analysis issues. [18] We reviewed the FAA Telecommunications Infrastructure, En Route Communications Gateway, and Voice Switching and Control System (VSCS) Control Subsystem Upgrade (VCSU) projects. The projects are described in appendix I. [19] VCSU is a subcomponent of the VSCS project. We decided to review VCSU because its investment management process was carried out using FAA's AMS, whereas the VSCS project was funded before the AMS became part of the FAA's investment management process. [20] According to FAA, no acquisition program baseline was prepared for VCSU. [21] GAO/RCED/AIMD-99-88. [22] GAO-04-394G. [23] VSCS is FAA's highly distributed, computer-controlled communications and control system for U.S. air traffic management that allows air traffic controllers to establish all air-to-ground and ground-to-ground communications with pilots and other air traffic controllers. GAO's Mission: The Government Accountability Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO's Web site ( www.gao.gov ) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as "Today's Reports," on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select "Subscribe to e-mail alerts" under the "Order GAO Products" heading. Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office 441 G Street NW, Room LM Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov Automated answering system: (800) 424-5454 or (202) 512-7470: Public Affairs: Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, D.C. 20548:

The Justia Government Accountability Office site republishes public reports retrieved from the U.S. GAO These reports should not be considered official, and do not necessarily reflect the views of Justia.