Aviation Weather
Services at Key Aviation Facilities Lack Performance Measures, but Improvement Efforts Are Under Way Gao ID: GAO-08-491T February 26, 2008The National Weather Service (NWS), an agency under the Department of Commerce's National Oceanic and Atmospheric Administration (NOAA), provides staff on-site at each of the Federal Aviation Administration's (FAA) en route centers--the facilities that control high-altitude flight outside the airport tower and terminal areas. This group of NWS meteorologists provides air traffic managers with forecasts and briefings on regional conditions such as turbulence and icing. Over the last few years, FAA has been exploring options for enhancing the efficiency of the aviation weather services provided by these NWS meteorologists. In late December 2007, FAA delivered revised requirements and associated performance measures to NWS to improve these services. GAO was asked to summarize key segments of its report being released today, including its assessment of NWS and FAA efforts to ensure the quality of aviation weather services at en route centers, and its recommendations to improve these efforts. In addition, GAO was asked to provide an update on FAA's recent efforts to establish aviation weather requirements and performance measures, and NWS's plans for responding to these requirements. To do so, GAO summarized segments of its report, reviewed FAA's recently released requirements, and interviewed the official responsible for NWS's response.
Although interagency agreements between NWS and FAA state that both agencies have responsibilities for assuring and controlling the quality of aviation weather observations, neither agency consistently does so for weather products and services produced at the en route centers. Specifically, neither agency has developed and implemented performance measures and metrics, regularly evaluated weather service unit performance, or provided feedback to improve these aviation weather products and services. Because of this lack of performance tracking and oversight, NWS cannot demonstrate the quality or value of its services, and FAA cannot ensure the quality of the services it funds. Until both agencies are able to measure and ensure the quality of the aviation weather products at the en route centers, FAA may not be getting the information it needs to effectively manage air traffic. In its report being issued today, GAO is making recommendations to the Secretaries of Commerce and Transportation to ensure that NWS and FAA develop performance measures, evaluate the services against those measures, and provide feedback to NWS. Commerce agreed with the recommendations and stated that NOAA would work with FAA to develop methods for performance monitoring and evaluation. Transportation did not agree or disagree with the recommendations, but stated that FAA's revised requirements would establish performance measures and evaluation procedures, and that FAA would negotiate with NWS to implement them. FAA has begun to address GAO's recommendations. In late December 2007, FAA finalized its new requirements, including performance measures and methods for evaluating performance and providing feedback to NWS. In doing so, FAA provides its overall vision for aviation weather services, reiterates its need for existing products and services, provides revisions to existing requirements, and defines a new product. FAA directed NWS to respond by May 2008 and include plans in its response for three operational concepts--in its existing configuration located at the 21 en route centers, through remote services provided by a reduced number of regional facilities, and through remote services provided by a single centralized facility. FAA stated that NWS should assume a transition time of 90 days for the existing configuration, 180 days for regionalized services, and 1 year for a single facility. NWS plans to respond to FAA by the May 2008 deadline, but FAA's estimated time frames for transitioning to a new operational concept may be overly ambitious. Given the importance of accurate and timely weather information in air traffic control, it will be important for NWS to conduct a thorough evaluation before it transitions to a new operational concept in order to ensure that there are no impacts on the continuity of air traffic operations and no degradation of weather service.
GAO-08-491T, Aviation Weather: Services at Key Aviation Facilities Lack Performance Measures, but Improvement Efforts Are Under Way
This is the accessible text file for GAO report number GAO-08-491T
entitled 'Aviation Weather: Services at Key Aviation Facilities Lack
Performance Measures, but Improvement Efforts Are Under Way' which was
released on February 26, 2008.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Testimony:
Before the Subcommittee on Energy and Environment, House Committee on
Science and Technology:
United States Government Accountability Office:
GAO:
For Release on Delivery:
Expected at 1:00 p.m. EST:
Tuesday, February 26, 2008:
Aviation Weather:
Services at Key Aviation Facilities Lack Performance Measures, but
Improvement Efforts Are Under Way:
Statement of David A. Powner, Director: Information Technology
Management Issues:
GAO-08-491T:
GAO Highlights:
Highlights of GAO-08-491T, a testimony before the Subcommittee on
Energy and Environment, House Committee on Science and Technology.
Why GAO Did This Study:
The National Weather Service (NWS), an agency under the Department of
Commerce‘s National Oceanic and Atmospheric Administration (NOAA),
provides staff on-site at each of the Federal Aviation Administration‘s
(FAA) en route centers”the facilities that control high-altitude flight
outside the airport tower and terminal areas. This group of NWS
meteorologists provides air traffic managers with forecasts and
briefings on regional conditions such as turbulence and icing. Over the
last few years, FAA has been exploring options for enhancing the
efficiency of the aviation weather services provided by these NWS
meteorologists. In late December 2007, FAA delivered revised
requirements and associated performance measures to NWS to improve
these services.
GAO was asked to summarize key segments of its report being released
today, including its assessment of NWS and FAA efforts to ensure the
quality of aviation weather services at en route centers, and its
recommendations to improve these efforts. In addition, GAO was asked to
provide an update on FAA‘s recent efforts to establish aviation weather
requirements and performance measures, and NWS‘s plans for responding
to these requirements. To do so, GAO summarized segments of its report,
reviewed FAA‘s recently released requirements, and interviewed the
official responsible for NWS‘s response.
What GAO Found:
Although interagency agreements between NWS and FAA state that both
agencies have responsibilities for assuring and controlling the quality
of aviation weather observations, neither agency consistently does so
for weather products and services produced at the en route centers.
Specifically, neither agency has developed and implemented performance
measures and metrics, regularly evaluated weather service unit
performance, or provided feedback to improve these aviation weather
products and services. Because of this lack of performance tracking and
oversight, NWS cannot demonstrate the quality or value of its services,
and FAA cannot ensure the quality of the services it funds. Until both
agencies are able to measure and ensure the quality of the aviation
weather products at the en route centers, FAA may not be getting the
information it needs to effectively manage air traffic.
In its report being issued today, GAO is making recommendations to the
Secretaries of Commerce and Transportation to ensure that NWS and FAA
develop performance measures, evaluate the services against those
measures, and provide feedback to NWS. Commerce agreed with the
recommendations and stated that NOAA would work with FAA to develop
methods for performance monitoring and evaluation. Transportation did
not agree or disagree with the recommendations, but stated that FAA‘s
revised requirements would establish performance measures and
evaluation procedures, and that FAA would negotiate with NWS to
implement them.
FAA has begun to address GAO‘s recommendations. In late December 2007,
FAA finalized its new requirements, including performance measures and
methods for evaluating performance and providing feedback to NWS. In
doing so, FAA provides its overall vision for aviation weather
services, reiterates its need for existing products and services,
provides revisions to existing requirements, and defines a new product.
FAA directed NWS to respond by May 2008 and include plans in its
response for three operational concepts”in its existing configuration
located at the 21 en route centers, through remote services provided by
a reduced number of regional facilities, and through remote services
provided by a single centralized facility. FAA stated that NWS should
assume a transition time of 90 days for the existing configuration, 180
days for regionalized services, and 1 year for a single facility.
NWS plans to respond to FAA by the May 2008 deadline, but FAA‘s
estimated time frames for transitioning to a new operational concept
may be overly ambitious. Given the importance of accurate and timely
weather information in air traffic control, it will be important for
NWS to conduct a thorough evaluation before it transitions to a new
operational concept in order to ensure that there are no impacts on the
continuity of air traffic operations and no degradation of weather
service.
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.GAO-08-491T]. For more information, contact
David A. Powner at 202-512-9286 or pownerd@gao.gov.
[End of section]
Mr. Chairman and Members of the Subcommittee:
We appreciate the opportunity to participate in today's hearing to
discuss our work on the National Weather Service's (NWS) aviation
weather services. NWS is responsible for providing storm and flood
warnings and weather forecasts for the United States, its territories,
and adjacent oceans and waters. NWS's weather products are also a vital
component of the Federal Aviation Administration's (FAA) air traffic
control program, providing weather information to local, regional, and
national air traffic management, navigation, and surveillance systems.
NWS aviation weather products include forecasts and warnings of
meteorological conditions that could affect air traffic, including
thunderstorms, air turbulence, and icing. In addition to providing
aviation weather products developed at its own facilities, NWS also
provides staff on-site at each of FAA's en route centers--the
facilities responsible for controlling high-altitude air traffic
outside the tower and terminal areas.
Over the last few years, FAA has been exploring options for enhancing
the efficiency of the aviation weather services provided at its en
route centers. In September 2005, the agency asked NWS to restructure
its services to be more efficient; in response, NWS conducted a
prototype and proposed restructuring its offices to provide services
remotely. FAA declined this proposal in favor of making its existing
requirements more precise. In late December 2007, FAA delivered its
revised requirements to NWS to improve these services.
As requested, our testimony summarizes our report being released today
on NWS's aviation weather services[Footnote 1] and provides an update
on recent efforts to develop aviation weather requirements and
performance measures. Specifically, we discuss both agencies' efforts
to ensure the consistency and quality of these services, our
recommendations to improve these services, FAA's recent efforts to
establish requirements and performance measures, and NWS's plans for
responding to these requirements.
The information in this statement is based largely on our work
supporting the report being released today. In addition, to provide an
update on the agencies' recent efforts, we reviewed key documents
completed in December 2007, including a new interagency agreement,
FAA's requirements, and the accompanying quality assurance plan. We
compared NWS's tentative next steps with best practices for validating
requirements and interviewed the NWS official responsible for
responding to the new requirements. We conducted our work on the report
and testimony between May 2007 and February 2008, in accordance with
generally accepted government auditing standards. Those standards
require that we plan and perform the audit to obtain sufficient,
appropriate evidence to provide a reasonable basis for our findings and
conclusions based on our audit objectives. We believe that the evidence
obtained provides a reasonable basis for our findings and conclusions
based on our audit objectives. Additional details on our objectives,
scope, and methodology are provided in appendix I.
Results in Brief:
Although interagency agreements between NWS and FAA state that both
agencies have responsibilities for assuring and controlling the quality
of aviation weather observations, neither agency consistently does so
for weather products and services produced at the en route centers.
Specifically, neither has implemented performance measures and metrics,
regularly evaluated weather service unit performance, or provided
feedback to improve these aviation weather products and services.
Because of this lack of performance tracking and oversight, NWS cannot
demonstrate the quality or value of its services, and FAA cannot ensure
the quality of the services it funds. Until both agencies are able to
measure and ensure the quality of the aviation weather products and
services at the en route centers, FAA may not be getting the
information it needs to effectively manage air traffic.
In our report being released today, we are making recommendations to
the Secretaries of Commerce and Transportation to ensure that NWS and
FAA develop performance measures for aviation weather services provided
at en route centers, evaluate the services against those measures, and
provide feedback to the NWS staff on how to improve services. The
Secretary of Commerce agreed with our recommendations and stated that
the National Oceanic and Atmospheric Administration (NOAA) would work
with FAA to develop methods for performance monitoring and evaluation.
The Department of Transportation did not agree or disagree with our
recommendations, but stated that FAA's revised requirements would
establish performance measures and evaluation procedures, and that FAA
would negotiate with NWS to implement them.
FAA has already begun to address the recommendations noted in our
report; specifically, in late December 2007, FAA finalized its new
aviation weather requirements, which include proposed performance
measures and methods for evaluation. In its requirements, FAA provides
NWS with an overall vision for aviation weather services that are
performance-based, standardized, continuous, and have a national scope.
FAA reiterates its need for existing products and services (such as
twice-daily briefings), provides revisions to some existing
requirements, and defines a new graphical forecast product for terminal
radar approach control facilities. In addition, FAA identifies
performance measures--such as customer satisfaction and forecast
accuracy--and processes for evaluating performance and providing
feedback to NWS. FAA expects NWS to respond as to whether it is able to
meet the requirements by early May 2008, and has directed NWS to
include plans for three operational concepts to fulfill the
requirements--in its existing configuration located at the 21 en route
centers, through remote services provided by a reduced number of
regional facilities, and through remote services provided by a single
centralized facility. FAA plans to select one of the operational
concepts and NWS will immediately begin to transition to the new
concept. FAA required that NWS assume a transition time of 90 days if
it selects the existing configuration, 180 days if it selects the
regionalized remote services concept, and 1 year if it selects the
single facility concept.
NWS plans to respond to FAA by the May 2008 deadline, but FAA's
estimated time frames for providing the revised services may be overly
ambitious. NWS created a team to analyze FAA's requirements and to
develop a response package for all three operational concepts. The NWS
official responsible for aviation services reported that the agency is
on track to respond by FAA's deadline of May 2008. However, FAA's
estimated time frames for transitioning to a new operational concept
may be overly ambitious. Given the importance of accurate and timely
weather information in air traffic control, it will be important for
NWS to conduct a thorough evaluation before it transitions to a new
operational concept in order to ensure that there are no impacts on the
continuity of air traffic operations and no degradation of weather
service.
Background:
FAA is responsible for ensuring safe, orderly, and efficient air travel
in the national airspace system. NWS supports FAA by providing aviation-
related forecasts and warnings at air traffic facilities across the
country. Among other support and services, NWS provides four
meteorologists at each of FAA's 21 en route centers to provide on-site
aviation weather services. This arrangement is defined and funded under
an interagency agreement.
FAA's Mission and Organizational Structure:
FAA's primary mission is to ensure safe, orderly, and efficient air
travel in the national airspace system. The agency's ability to fulfill
its mission depends on the adequacy and reliability of its air traffic
control systems, as well as weather forecasts made available by NWS and
automated systems. These resources reside at, or are associated with,
several types of facilities: air traffic control towers, terminal radar
approach control facilities, air route traffic control centers (en
route centers), and the Air Traffic Control System Command Center. The
number and functions of these facilities are as follows:
* 517 air traffic control towers manage and control the airspace within
about 5 miles of an airport. They control departures and landings, as
well as ground operations on airport taxiways and runways.
* 170 terminal radar approach control facilities provide air traffic
control services for airspace within approximately 40 miles of an
airport and generally up to 10,000 feet above the airport, where en
route centers' control begins. Terminal controllers establish and
maintain the sequence and separation of aircraft.
* 21 en route centers control planes over the United States--in transit
and during approaches to some airports. Each center handles a different
region of airspace. En route centers operate the computer suite that
processes radar surveillance and flight planning data, reformats them
for presentation purposes, and sends them to display equipment that is
used by controllers to track aircraft. The centers control the
switching of voice communications between aircraft and the center, as
well as between the center and other air traffic control facilities.
Two en route centers also control air traffic over the oceans.
* The Air Traffic Control System Command Center manages the flow of air
traffic within the United States. This facility regulates air traffic
when weather, equipment, runway closures, or other conditions place
stress on the national airspace system. In these instances, traffic
management specialists at the command center take action to modify
traffic demands in order to keep traffic within system capacity.
See figure 1 for a visual summary of the facilities that control and
manage air traffic over the United States.
Figure 1: FAA Facilities Involved in Air Traffic Control:
[See PDF for image]
This figure is an illustration of FAA Facilities Involved in Air
Traffic Control. The following data is depicted:
Air Traffic Control System Command Center:
Preflight: Air traffic control tower;
Takeoff: Air traffic control tower;
Departure: Terminal radar approach control;
En route: Air route traffic control center;
Descent: Air route traffic control center; Terminal radar approach
control;
Approach: Terminal radar approach control;
Landing: Air traffic control tower.
Source: GAO analysis of FAA data.
[End of figure]
NWS's Mission and Organizational Structure:
The mission of NWS--an agency within the Department of Commerce's NOAA--
is to provide weather, water, and climate forecasts and warnings for
the United States, its territories, and its adjacent waters and oceans
to protect life and property and to enhance the national economy. In
addition, NWS is the official source of aviation- and marine-related
weather forecasts and warnings, as well as warnings about life-
threatening weather situations.
The coordinated activities of weather facilities throughout the United
States allow NWS to deliver a broad spectrum of climate, weather,
water, and space weather services in support of its mission. These
facilities include 122 weather forecast offices located across the
country that provide a wide variety of weather, water, and climate
services for their local county warning areas, including advisories,
warnings, and forecasts; 9 national prediction centers[Footnote 2] that
provide nationwide computer modeling to all NWS field offices; and 21
center weather service units that are located at FAA en route centers
across the nation and provide meteorological support to air traffic
controllers.
NWS Provides Aviation Weather Services to FAA:
As an official source of aviation weather forecasts and warnings,
several NWS facilities provide aviation weather products and services
to the FAA and aviation sector. These facilities include the aviation
weather center, weather forecast offices located across the country,
and center weather service units located at FAA en route centers. See
table 1.
Table 1: NWS Offices That Provide Aviation Weather Products and
Services to FAA:
Office: Aviation weather center;
Description: The aviation weather center, located in Kansas City,
Missouri, issues warnings, forecasts, and analyses of hazardous weather
for aviation. Staffed by 65 personnel, the center develops warnings of
hazardous weather for aircraft in flight and forecasts of weather
conditions for the next 2 days that could affect both domestic and
international aviation. The center also leads a collaborative effort to
develop a forecast of expected convective events for the entire country
every 2 hours. This is used by FAA to manage aviation traffic flow
across the country.
Office: Weather forecast offices;
Description: NWS's 122 weather forecast offices issue terminal area
forecasts for approximately 625 locations every 6 hours or when
conditions change. These forecasts consist of the expected weather
conditions significant to a given airport or terminal and are primarily
used by commercial and general aviation pilots.
Office: Center weather service units;
Description: NWS's center weather service units are located at each of
FAA's 21 en route centers and operate 16 hours a day, 7 days a week
(see fig. 2). Each weather service unit usually consists of three
meteorologists and a meteorologist-in-charge who provide strategic
advice and aviation weather forecasts to FAA traffic management
personnel. Governed by an interagency agreement, FAA currently
reimburses NWS approximately $12 million annually for this support.
Source: GAO analysis of NWS and FAA data.
[End of table]
Figure 2: Center Weather Service Unit Locations and Service Areas:
[See PDF for image]
This figure is a map of the United States, illustrating Center Weather
Service Unit Locations and Service Areas. The following locations are
depicted:
Boston;
New York;
Washington, DC;
Cleveland;
Indianapolis;
Memphis;
Atlanta;
Jacksonville;
Miami;
Houston;
Fort Worth;
Kansas City;
Chicago;
Minneapolis;
Denver;
Albuquerque;
Salt Lake City;
Los Angeles;
Oakland;
Seattle;
Anchorage.
Source: NWS (data). Map Resources (map).
[End of figure]
Center Weather Service Units: An Overview of Current Requirements:
FAA's existing requirements for the center weather service units are
broadly outlined in an interagency agreement that is updated every few
years. The interagency agreement specifies that NWS is to provide
meteorological advice and consultation to en route center operations
personnel and other designated FAA air traffic facilities within the en
route area of responsibility. This agreement establishes specific terms
that govern the number of NWS staff, their working hours, and cost
reimbursement details. It does not specify the contents, quality, or
frequency of weather products.
An NWS directive, signed in May 2006 and intended for NWS's weather
forecast offices and center weather service units, provides more
specific information regarding the content of weather products and
services, including center weather advisories, daily briefings, on-
demand consultations, and meteorological impact statements. These
products and services are described in table 2. In addition, center
weather service unit meteorologists can provide input every 2 hours to
the Aviation Weather Center's creation of the Collaborative Convective
Forecast Product; train FAA personnel on how to interpret weather
information; and, if warranted, provide weather briefings to nearby
terminal radar approach control facilities.
Table 2: Key Products and Services Currently Provided by Center Weather
Service Units:
Product or service: Center weather advisory;
Description: A short-term, unscheduled warning of hazardous weather
conditions used primarily by air crews to anticipate and avoid adverse
weather conditions in the en route and terminal environments. It
describes current weather conditions or adverse weather conditions--
such as moderate to severe icing or turbulence, thunderstorms, and low
ceilings and visibility-- beginning within the next 2 hours.
Product or service: Briefings;
Description: Short updates provided by en route center meteorologists
to FAA supervisors twice a day; they include current weather
advisories, a summary of the predicted weather in the en route area,
terminal forecasts, and jet stream and freezing information.
Product or service: On-demand consultation;
Description: Unscheduled verbal presentations provided to traffic
management controllers, supervisors, and other FAA facilities within
the en route center area. Consultations may be about the expected
weather conditions or interpretations of weather information from the
satellite images.
Product or service: Meteorological impact statement;
Description: An unscheduled forecast of weather conditions that are
expected to adversely impact the flow of air traffic in the en route
center's area of responsibility within 4 to 12 hours. These statements
detail weather conditions expected to adversely impact air traffic flow
in the service unit area of responsibility and should include the
location, height, extent, and movement of the weather conditions.
Source: GAO analysis of FAA and NWS data.
[End of table]
FAA Sought to Improve Aviation Weather Services Provided at En Route
Centers:
In recent years, FAA has sought to assess and improve the performance
of the center weather service units.[Footnote 3] For example, FAA
performed multiple studies on the current services provided by the
center weather service units that noted the lack of standardization of
products and services. In addition, FAA conducted a study to determine
if remote operations were feasible, and requested that NWS restructure
its aviation weather services to provide improved services more
efficiently. In response to this request, NWS conducted a prototype of
remote operations in which center weather service unit products and
services were prepared by the closest weather forecast office. NWS
proposed expanding this prototype to FAA, but the agency declined this
proposal. Instead, FAA stated that it would redefine its requirements
for the functions provided by center weather service units. Table 3
provides more information about the agency's assessment and improvement
efforts.
Table 3: Recent Assessment and Improvement Efforts by FAA and NWS:
Time frame: November 2003;
Activity: FAA performed a functional audit of center weather service
units and found that the services provided at different en route
locations were inconsistent, the products were not standardized, and
there was little communication and collaboration between neighboring
service units.
Time frame: September 2005;
Activity: FAA requested that NWS restructure its aviation weather
services to provide improved services more efficiently.
Time frame: January 2006;
Activity: FAA initiated an analysis of the value of different
activities performed by the center weather service units. Similar to
the 2003 study, the results of this analysis noted the lack of
standardization of products, services, tools, and procedures. In
addition, the report found that quality assurance was provided on an
informal basis, there was no formal feedback process for products and
services, and meteorological training was not standardized.
Time frame: August 2006;
Activity: NWS conducted a prototype in which center weather service
unit products and services were completed and delivered remotely from
the closest weather forecast office. This prototype showed that remote
operations were possible and effective, but that they would be
difficult to implement because of the need for cultural change,
technology upgrades, and communication stability. Specifically,
forecasters in the prototype were not able to provide dedicated support
for the aviation mission because their other duties-
-including forecasting severe weather at the weather forecast office--
took precedence. In addition, a collaboration technology used during
the prototype was not operationally ready to use, servers were
unstable, critical radar data were inconsistent with weather forecast
office data, and communications lines were unstable throughout the
prototype.
Time frame: September 2006;
Activity: An FAA study confirmed that it is possible to deliver weather
information, products, and services from one or many remote locations
with currently available state-of-the-art technology platforms.
Time frame: October 2006;
Activity: FAA administered a market survey to determine whether the
private sector could provide remote weather services at a lower cost
than currently provided. Ten organizations, including private sector
firms and government-funded laboratories, responded that they could
provide the services that FAA wanted; NWS presented its proposal for
restructuring its aviation weather services to FAA. In this proposal,
NWS suggested moving meteorologists from the en route centers to
weather forecast offices, and providing remote aviation weather
services from the weather forecast offices.
Time frame: April 2007;
Activity: FAA declined NWS's proposal. Instead, FAA reported that it
would redefine its requirements for the functions provided by the
center weather service units.
Source: GAO analysis of NWS and FAA data.
[End of table]
FAA Found Its Requirements Were Not Sufficiently Precise and Worked to
Develop New Ones:
When FAA declined NWS's proposal for restructured aviation weather
services, it did so in part because it considered its existing
requirements governing NWS's center weather service units to be too
broad to ensure the efficiency and cost-effectiveness of the services.
FAA then worked for several months to redefine these requirements. In
April 2007, FAA's Air Traffic Organization began refining its
requirements for aviation weather services at the en route centers. To
do this, FAA collected relevant NWS and FAA orders and directives and
developed a list of over 100 products and services that the different
service units provide. FAA then sent this list to traffic managers in
each of the en route centers, asking them to specify the products and
services that they need, the ones they do not need, and any new
products or services that they would like. Traffic managers were also
able to determine if they would want some of the more customized
weather products that are currently available at selected en route
centers. Using results from this survey, FAA developed a revised list
of requirements and performance measures, which it provided to NWS in
late December 2007.
Neither NWS nor FAA Currently Ensures the Quality of Aviation Weather
Services at En Route Centers:
While interagency agreements between NWS and FAA state that both
agencies have responsibilities for assuring and controlling the quality
of aviation weather observations, neither NWS nor FAA consistently does
so for weather products produced at the en route centers. Leading
organizations use quality assurance to provide staff and management
with objective insights into processes and associated work products
[Footnote 4]. Generally, quality assurance includes objectively
evaluating performed processes, work products, and services against
applicable process descriptions, standards, and procedures; identifying
and documenting noncompliance issues; providing feedback to project
staff and managers on the results of quality assurance activities; and
ensuring that noncompliance issues are addressed. However, neither NWS
nor FAA has developed and implemented performance measures and metrics,
regularly evaluated weather service unit performance, or provided
feedback to improve these aviation weather products and services.
Because of this lack of performance tracking and oversight, NWS cannot
demonstrate the quality or value of its services, and FAA cannot ensure
the value of the services it funds. As a result, it is not clear that
FAA is getting the information it needs to effectively manage air
travel. FAA officials stated that they intend to establish performance
measures for their redefined requirements and to improve their
oversight against these measures. However, FAA has not worked with NWS
to define a comprehensive set of measures for its requirements, and it
is unclear how the agency would develop a performance baseline for
comparison to actual performance because many of the products and
services have not previously been measured.
NWS Does Not Measure or Evaluate Aviation Weather Products and Services
at En Route Centers:
NWS does not measure or evaluate the aviation weather services it
provides at en route centers. Under existing interagency agreements,
NWS is responsible for controlling the quality of its aviation weather
observations. Specifically, the agency is responsible for monitoring
and evaluating the quality and effectiveness of its aviation weather
services, including the services provided at the weather forecast
offices, the Aviation Weather Center, and the en route centers.
While NWS has developed and continues to monitor performance measures
for aviation weather forecasts provided by its weather forecast offices
and the Aviation Weather Center, the agency has not done so for the
weather products and services provided at the en route centers.
Specifically, NWS has not developed performance measures for aviation
weather products and services at en route centers, evaluated the
aviation weather products and services developed at the en route
centers, or provided feedback for those services. NOAA and NWS
officials declined to explain why the agency does not have performance
measures for aviation weather products or services at en route centers,
but they noted that neither FAA nor NWS has required or funded such an
effort. Further, the aviation services branch chief told us that he had
planned to begin evaluations for weather unit services at the en route
centers but decided to wait because of the potential for large-scale
changes to the services.
Until NWS establishes performance measures and evaluates the quality
and effectiveness of its products against these measures, the agency
will remain unable to ensure that it provides consistent quality
products and to effectively demonstrate the value it provides to FAA.
FAA Does Not Consistently Evaluate or Provide Feedback on Aviation
Weather Services at En Route Centers:
FAA has not consistently evaluated NWS services at its en route centers
or adequately provided feedback on the results of its few evaluations.
Under interagency agreements, FAA is responsible for ensuring that
aviation weather services meet its requirements. In addition, it
requires the traffic management officer within each traffic management
unit to evaluate the aviation weather services at the en route centers
annually and to provide feedback to the resident meteorologist-in-
charge.
FAA has not consistently ensured the quality of aviation weather
services at en route centers. Specifically, it currently does not have
any quantitative and objective performance measures--such as
timeliness, accuracy, or false alarm rate--by which to evaluate these
services. Agreements between the agencies broadly specify the types of
aviation weather products to be developed at the en route centers but
do not provide criteria by which these products can be evaluated. In
addition, FAA has not consistently performed its annual evaluations of
these products and services. According to the contracting officer's
technical representative responsible for the evaluations, the last
evaluation was performed in 2006, and its results were largely
anecdotal. Specifically, the evaluation called for the traffic
management officer to rate the weather unit on a scale of 0 to 4 in
different categories, including quality and timeliness of products and
services, knowledge of air traffic control, and participation in
training. The technical representative could not find any evaluations
in 2005, evaluations of only three service units in 2004, and
evaluations of a similarly small number of service units in 2003.
Further, FAA is not consistently providing feedback to weather staff at
the en route centers. According to the technical representative, the
evaluations from 2006 were not compiled or analyzed because the
evaluations contained no glaring problems or issues that needed
additional attention. In addition, the NWS aviation services chief told
us that FAA had sent him copies of the evaluations from 2006 but did
not offer analysis of these evaluations, express concerns about the
services, or send the results to the individual center weather service
units. This official also stated that he was not aware that FAA had
performed any annual evaluations of the center weather service units
prior to 2006.
Because FAA has not established performance requirements or
consistently and thoroughly evaluated the aviation weather services at
en route centers, the agency cannot be sure that the products and
services provided by the weather unit meteorologists are adding value,
and it cannot provide feedback to NWS meteorologists in order to
improve the services. To address this shortfall, FAA officials stated
that they intend to establish performance measures for aviation weather
services at en route centers when they revise their requirements and to
improve their oversight of NWS against these measures. However, FAA has
not worked with NWS to develop measures for the products and services
it will require from NWS, and it is unclear how the agency would
develop a performance baseline for comparison to actual performance
because many of the products and services have not previously been
measured.
Implementation of GAO Recommendations Should Help Address Performance
Measurement Shortfalls:
While many steps remain in defining the future of aviation weather
services at en route centers--including negotiations between FAA and
NWS on the provision of these services and FAA's subsequent decision on
whether to obtain selected services from alternative sources--there are
steps both agencies can take to ensure that the quality of future
aviation weather products and services is measured and evaluated. In
our accompanying report released today,[Footnote 5] we made two
recommendations to the Secretary of Commerce and three recommendations
to the Secretary of Transportation to improve the quality of aviation
weather products and services at en route centers.
We recommended that the Secretary of Commerce direct the Assistant
Administrator for the National Weather Service to:
* assist FAA in developing performance measures and metrics for the
products and services to be provided by center weather service units,
and:
* perform annual evaluations of aviation weather services provided at
en route centers and provide feedback to the center weather service
units.
Further, we recommended that the Secretary of the Department of
Transportation direct the FAA Administrator to:
* work with NWS to define performance measures and metrics for aviation
weather services provided by meteorologists,
* evaluate the services it receives against those measures and metrics,
and:
* ensure that results of these evaluations are provided to staff
stationed at center weather service units so that they can improve
performance, where applicable.
In written comments on a draft of our report, the Secretary of Commerce
agreed with our recommendations to assist FAA in developing performance
measures and metrics, and to perform annual evaluations of aviation
weather services and provide feedback to the center weather service
units. The department stated that after FAA provides its revised
requirements, it would work with FAA to develop methods for performance
monitoring and evaluation. Subsequently, on December 19, 2007, FAA
provided its revised requirements to NWS.
The Department of Transportation's Director of Audit Relations provided
comments on a draft of the report via e-mail. In those comments, the
department did not agree or disagree with our recommendations. The
department stated that FAA's revised requirements are consistent with
our recommendations in that they establish performance measures and
evaluation procedures, and that FAA would begin to negotiate with NWS
to implement them. In addition, in late December 2007, after reviewing
our draft report, FAA and NWS signed a new interagency agreement that
requires FAA to develop performance standards and measures for the
assessment of center weather service units, and requires NWS to develop
and track metrics to support FAA's performance measures.
FAA Identified New Aviation Weather Requirements and Performance
Measures:
FAA has already begun to address our recommendations; specifically, in
late December 2007, FAA finalized its new requirements for the aviation
weather services to be provided by center weather service units, which
include proposed performance measures and methods for evaluation. In
its requirements, FAA provides NWS with its overall vision for aviation
weather services, revises existing requirements, and defines a
requirement for a new product for terminal radar approach control
facilities. In addition, FAA identifies performance measures and
processes for evaluating performance and providing feedback to NWS.
FAA envisions services that are performance-based, standardized,
continuous, and have a national scope. Specifically, FAA requires
performance-based services that are measurable and allow for
identifying both successful performance and any performance problems.
In addition, FAA requests that the center weather service units provide
standardized services to all en route centers and increase their
service coverage from 16 hours a day to 24 hours a day. Finally, FAA
calls for transitioning the scope of the center weather service units
to monitor the entire national airspace system, rather than the
respective en route center regions. This national scope is expected to
allow more integrated decision making at the national level while
continuing to provide specialized products at the regional and local
levels.
In its new requirements, FAA also reiterates its need for existing
products and services and provides revisions to some of these
requirements. Specifically, FAA continues to require products such as
twice-daily briefings, center weather advisories, and the Collaborative
Convective Forecast Product. In addition, center weather service units
will continue to provide forecast coordination with other NWS offices,
on-demand advice and consultation, emergency planning, training, and
dissemination of a number of weather advisories into both NWS and FAA
systems. Daily briefings, however, will now be recorded, verified, and
disseminated to other facilities that do not receive an in-person
briefing. In addition, on-demand consultation will be provided to en
route centers, terminal radar approach control facilities, towers, and
the Air Traffic System Command Center. According to the aviation
services branch chief, this consultation is currently provided only to
en route centers, selected terminal radar approach control facilities,
and a small number of towers.
Further, the revised requirements define a new product: the terminal
radar approach control forecast product. This product is based in part
on decision aids currently used in select center weather service units,
and on requirements developed by a team consisting of aviation
meteorological stakeholders from industry and FAA. This forecast, which
describes the next 6 hours and is updated at least every 2 hours, will
be presented in a graphical format and include convection, winds,
ceilings, and visibilities for the area around terminal radar approach
control facilities. FAA also expects this product to include methods
for verification and the systematic collection of user feedback.
In addition to these requirements, FAA identifies performance measures
as well as processes for evaluating performance and providing feedback
to the forecasters. These performance measures include customer
satisfaction, forecast accuracy, and the aggregate of aircraft
incidents attributed to inaccurate aviation weather forecasts.
Baselines for all of these measures have not yet been developed.
According to the chief of the aviation services branch, NWS will
propose additional performance measures and develop baselines as it is
able. To measure against these performance measures, FAA has identified
methods by which to evaluate NWS. For example, to determine customer
satisfaction, FAA plans to develop a questionnaire for traffic
management unit staff to be filled out quarterly; to determine the
aggregate of aircraft incidents attributed to inaccurate aviation
weather forecasts, FAA is to use safety statistics currently tracked by
the Air Traffic Organization. In addition, FAA is planning to draw on
NWS's subject matter expertise to record and analyze information to
determine the accuracy of forecasts.
FAA expects NWS to respond as to whether it is able to meet the
requirements by early May 2008. In addition, FAA directed NWS to
include plans for three operational concepts (including technical and
cost information) for fulfilling the requirements--in its existing
configuration located at the 21 en route centers, through remote
services provided by a reduced number of regional facilities, and
through remote services provided by a single centralized facility.
According to the requirements, NWS's response should assume a
transition time of 90 days for the existing configuration, 180 days for
regionalized remote services, and 1 year for a single facility. For
each of these concepts, FAA has requested that NWS include a technical
plan, a facilities plan, a quality management plan, and a plan for
transitioning to the new approach. In addition, FAA asked that NWS
include cost plans for each of the concepts with the assumption of 1
base year and four 1-year options thereafter. The cost plan is also
expected to include estimated annual cost savings over this 5-year
period. FAA plans to select one of these operational concepts.
NWS Plans to Respond to FAA's Requirements, but Proposed Transition
Time Frames May Be Overly Ambitious:
NWS plans to respond to FAA by the May 2008 deadline, but FAA's
estimated time frames for providing the revised services may be overly
ambitious. NWS plans to submit its proposals for the three operational
concepts--the existing configuration located at the 21 en route
centers, remote services provided by a reduced number of regional
facilities, and remote services provided by a single centralized
facility. FAA directed NWS to assume a transition time of 90 days for
the existing configuration, 180 days for regionalized remote services,
and 1 year for a single facility. To respond, NWS management created a
team to analyze the requirements, gain clarification on the
requirements from FAA, and develop the response. The NWS aviation
services branch chief reported that the agency is on track to respond
by May 2008.
However, FAA's proposed time frames for transitioning to a new
operational concept may be overly ambitious because of the activities
that NWS should perform before any transition. In accordance with best
practices and the opinion of the National Transportation Safety Board,
NWS intends to validate that the organization can provide these new
requirements--through a prototype or similar demonstration--before
transitioning to a new approach. Leading organizations validate
requirements to determine what impact the intended operational
environment will have on the ability to satisfy the stakeholders'
needs, expectations, constraints, and interfaces. As part of this
validation, organizations explore the adequacy and completeness of
requirements by developing prototypes or simulations and by obtaining
feedback about them from relevant stakeholders. Given the importance of
accurate and timely weather information in air traffic control, it will
be important for NWS to conduct a thorough evaluation before it
transitions to a new operational concept in order to ensure that there
are no impacts on the continuity of air traffic operations and no
degradation of weather service.
In addition, NWS has agreed to negotiate with its employees' union, the
National Weather Service Employees Organization, whenever
organizational changes could affect working conditions--unless the
union has sufficient predecisional involvement. NWS's employees' union
has a representative on the team that is responding to FAA's
requirements, so that later negotiations may be unnecessary. However,
it is too soon to determine whether negotiations will be needed and how
long they will take.
NWS's aviation services branch chief agreed that FAA's transition time
lines would be challenging. This official estimated that it could take
over 2 years to transition to a new operational concept. To address
this disconnect between NWS's capabilities and FAA's expectations, NWS
plans to propose more feasible time frames in its response to FAA. FAA
officials reported that the agency will be open to NWS's proposal.
In summary, even though center weather service units have been in
operation for over two decades, neither FAA nor NWS has implemented
performance measures and metrics, regularly evaluated weather service
unit performance, or provided feedback to improve these aviation
weather products and services. Until the agencies establish a system of
performance tracking and oversight, NWS will not be able to demonstrate
the quality or value of its services, and FAA will not be able to
ensure the value of the services it funds. To address these shortfalls,
FAA has defined more precise requirements that include performance
measures and evaluation methods, and NWS is working to respond to these
requirements. In its response, NWS is expected to prepare plans for
three alternative operational concepts for fulfilling these
requirements. FAA will choose one of the concepts. However, FAA's
proposed time frames for transitioning to a new operational concept may
be overly ambitious. Given the potential for major changes to services
and structure, NWS may require additional time to properly validate the
requirements, plan for any necessary operational transitions, and
ensure that aviation weather services are not degraded by any potential
changes.
Moving forward, it is important that FAA and NWS work together to
ensure that NWS's aviation weather services address requirements and
that effective performance measures and evaluation methods are
established. This collaboration will help both agencies ensure the
quality and consistency of these services, and ensure that FAA has the
information it needs to effectively manage air traffic.
Mr. Chairman, this concludes my statement. I would be happy to answer
any questions that you or members of the subcommittee may have at this
time.
If you have any questions on matters discussed in this testimony,
please contact me at (202) 512-9286 or by e-mail at pownerd@gao.gov.
Other key contributors to this testimony include Colleen Phillips
(Assistant Director), Kate Agatone, Amos Tevelow, and Jessica Waselkow.
[End of section]
Appendix I: Objectives, Scope, and Methodology:
The objectives of this statement were to summarize selected sections of
our report[Footnote 6] being released today, including National Weather
Service (NWS) and Federal Aviation Administration (FAA) efforts to
ensure the consistency and quality of aviation weather services at en
route centers, and our recommendations to improve these services. In
addition, we were asked to provide an update on FAA's recent efforts to
develop aviation weather requirements and performance measures, and
NWS's plans for responding to these requirements.
To summarize selected sections of our report, we relied on the work
supporting that report. That report contains a detailed overview of our
scope and methodology.
In addition, to provide an update on FAA's recent efforts to establish
requirements, we reviewed the new interagency agreement signed by both
agencies in late December 2007; FAA's requirements sent to NWS on
December 19, 2007; and the accompanying quality assurance plan. We also
interviewed NWS's aviation services branch chief to clarify which of
these requirements were new and which were revisions of current
requirements.
To determine NWS's plans for responding to these requirements, we
reviewed the new interagency agreement, FAA's requirements, and the
accompanying quality assurance plan. We also interviewed the aviation
services branch chief, who is serving in an oversight role for NWS's
response to FAA. We compared NWS's tentative next steps with best
practices for validating requirements from the Capability Maturity
Model® Integration for Development.
We performed our work on the report and testimony from May 2007 to
February 2008. All of the work on which this testimony is based was
performed in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objectives.
We believe that the evidence obtained provides a reasonable basis for
our findings and conclusions based on our audit objectives.
[End of section]
Footnotes:
[1] GAO, Aviation Weather: FAA Is Reevaluating Services at Key Centers;
Both FAA and the National Weather Service Need to Better Ensure Product
Quality, GAO-08-258 (Washington, D.C.: Jan. 11, 2008).
[2] These centers include the National Centers for Environmental
Prediction Central Operations, Aviation Weather Center, Environmental
Modeling Center, Hydrometeorological Prediction Center, Ocean
Prediction Center, Storm Prediction Center, Tropical Prediction Center/
National Hurricane Center, Climate Prediction Center, and Space
Environment Center.
[3] FAA is also involved in a longer-term initiative to increase the
efficiency of the national airspace system and to improve its overall
safety. This initiative, called the Next Generation Air Transportation
System, is a joint effort of the Department of Transportation, the
National Aeronautics and Space Administration, the White House Office
of Science and Technology Policy, and the Departments of Homeland
Security, Defense, and Commerce. FAA anticipates that this initiative
may lead to major changes in the aviation weather program that would
supersede its current efforts.
[4] The Carnegie Mellon University's Software Engineering Institute is
recognized for its expertise in software and system processes. See
Carnegie Mellon University Software Engineering Institute, Capability
Maturity Model® Integration for Development Version 1.2 (Pittsburgh,
PA: August 2006).
[5] GAO-08-258.
[6] GAO, Aviation Weather: FAA Is Reevaluating Services at Key Centers;
Both FAA and the National Weather Service Need to Better Ensure Product
Quality, GAO-08-258 (Washington, D.C.: Jan. 11, 2008).
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "Subscribe to Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
