Aviation Safety

A fundamental principle of aviation safety is the need to maintain adequate separation between aircraft and to ensure that aircraft maintain a safe distance from terrain, obstructions, and airspace that is not designated for routine air travel. Air traffic controllers employ separation rules and procedures that define safe separation in the air and on the ground.1 An operational error occurs when the separation rules and procedures are not followed due to equipment or human error. Data maintained by the Federal Aviation Administration (FAA) indicate that a very small number of operational errors occur in any given year--on average about three operational errors per day occurred in fiscal year 2002. However, some of these occurrences can pose safety risks by directing aircraft onto converging courses and, potentially, midair collisions. Congress asked us to provide information on FAA's data on operational errors and whether this data can be used to identify types of air traffic control facilities with greater safety risks. Specifically, we were asked to (1) determine what is known about the reliability and validity of the data that FAA maintains on operational errors and (2) identify whether comparisons of operational errors among air traffic control facilities can be used to determine the facilities' relative safety record.

We identified several potential limitations with FAA's data on operational errors based on our review of issued GAO and DOT reports and application of best methodological practices. First, it is very difficult to determine the completeness of the data. FAA collects data on operational errors from two sources--self-reporting by air traffic controllers and automatic reports of errors detected on the en route portion of a flight. The possibility exists for underreporting by air traffic controllers, since some errors are self-reported and some air traffic controllers may not self-report every incident. Second, due to the way the data are recorded, the severity of many errors cannot be determined or is misleading. Prior to 2001, minor errors, such as establishing a 4.5-mile rather than a 5-mile separation, were counted in the same way as more serious errors, according to DOT. In 2001, DOT began to address this issue by establishing a rating system to identify the severity of, or collision hazard posed by, operational errors. The system uses a 100-point scale to rate and categorize operational errors as high, moderate, or low severity. However, in 2003, DOT's IG reported continuing concerns with FAA's data on operational errors. The IG noted that the new rating system provides misleading information and that FAA needs to modify the system to more accurately identify the most serious operational errors. The DOT IG found that in one instance FAA rated an operational error as moderate that was less than 12 seconds from becoming a midair collision. The IG believed that this operational error should have been rated as high severity. The IG also reported that FAA cannot be sure that air traffic controllers report all operational errors. Comparisons of operational errors among types of air traffic control facilities, such as FAA staffed facilities versus contractor-staffed facilities, cannot be used alone to provide valid conclusions about safety due to three factors that we identified based on standard methodological practices and our understanding of FAA's data. First, such problems as the completeness and specificity of data on operational errors are likely to affect the validity of comparisons among air traffic control facilities because operational errors may not be comparably reported at the types of facilities being compared. Because of these factors, the determination of real differences in the rate of operational errors between different types of air traffic control facilities is difficult, and comparisons of operational error rates alone are not sufficient to draw conclusions about the relative safety records of air traffic control facilities. At a minimum, the additional factors mentioned above would need to be considered and analyzed with a technique that models the occurrence of rare events and looks at these events over time. This approach, however, is not without risk and would depend upon the existence of proper and reliable data on operational error rates, operating conditions at the towers at the time the error occurred, and other factors that may be associated with operational errors. Such an approach would allow for a more meaningful comparison of facilities' operational errors through ascertaining and accounting for the multiple factors that may be associated with such errors.