Aviation Weather
Agencies Need to Improve Performance Measurement and Fully Address Key Challenges Gao ID: GAO-10-843 September 9, 2010The National Weather Service's (NWS) weather products are a vital component of the Federal Aviation Administration's (FAA) air traffic control system. In addition to providing aviation weather products developed at its own facilities, NWS also provides on-site staff at each of FAA's en route centers--the facilities that control high-altitude flight outside the airport tower and terminal areas. NWS's on-site staff is called a center weather service unit. For several years, NWS and FAA have been exploring options for improving the aviation weather services provided at en route centers. GAO agreed to (1) determine the status of the agencies' efforts to restructure aviation weather services, (2) assess the agencies' progress in establishing performance baselines in order to measure the effect of any changes, and (3) evaluate plans to address key challenges. To do so, GAO evaluated agency progress and plans and compared agency efforts with leading practices.
After developing and shelving four proposals for restructuring the center weather service units over the last 5 years, in July 2010, senior NWS and FAA officials agreed to continue the current center weather service units at each of the 21 en route centers through September 2011 and to take immediate steps to improve aviation weather services by (1) having the service units provide forecasts for 10 key FAA terminal radar facilities and (2) having nearby weather forecast offices support FAA's en route centers when the service units are closed for the night. In addition, the agencies agreed to establish a joint team to baseline current capabilities and develop firm requirements for aviation weather services supporting air traffic flow management. While this agreement is important, the details have not been fully defined. Thus, it is not yet clear what will happen to the 21 service units after September 2011, when the immediate improvements in services will be in place, whether there are any costs associated with these steps, and who will pay for them. Until the two agencies further define their plans, the risk remains that the agencies will misjudge their responsibilities and not fulfill their agreements. FAA and NWS have made progress in identifying performance measures for the weather service units located at FAA en route centers, and NWS is beginning to track its service units' performance. However, NWS has not yet tracked, established baselines for, and reported to FAA on all applicable performance measures. Specifically, of seven possible performance measures, NWS is tracking performance for three of the measures and partially tracking a fourth measure. Of these four measures, the agency has established a sound baseline and reported on two of these measures and has made partial progress on two others. The agency is not tracking performance, documenting baselines, or reporting on three of the measures because it has not yet determined how to track them. Without an understanding of the current level of performance of the identified measures, the agencies will be limited in their ability to evaluate what progress has been made. In addition, until NWS regularly reports on its performance, the agencies lack the information they need to determine what is working well and what needs to be improved. In September 2009, GAO identified three challenges in modifying NWS's aviation weather services provided at FAA's en route centers: achieving interagency collaboration, defining requirements, and aligning changes with the Next Generation Air Transportation System (NextGen)--a long-term initiative to increase the efficiency of the national airspace system. The agencies have not yet fully addressed these challenges. Specifically, while senior agency officials recently agreed on how to proceed, work remains to be done to refine requirements, develop and execute an implementation plan, and to ensure that improvements are aligned with the long-term vision for NextGen. Until these fundamental challenges are addressed, the agencies are unlikely to achieve significant improvements in the aviation weather services provided at en route centers. GAO recommends that the Departments of Commerce and Transportation define their agreements, refine performance management processes, and address key challenges. In commenting on a draft of this report, Commerce agreed with GAO's recommendations and identified plans to address them; Transportation agreed to consider the recommendations.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: David A. Powner Team: Government Accountability Office: Information Technology Phone: (202) 512-9286GAO-10-843, Aviation Weather: Agencies Need to Improve Performance Measurement and Fully Address Key Challenges
This is the accessible text file for GAO report number GAO-10-843
entitled 'Aviation Weather: Agencies Need to Improve Performance
Measurement and Fully Address Key Challenges' which was released on
October 12, 2010.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
September 2010:
Aviation Weather:
Agencies Need to Improve Performance Measurement and Fully Address Key
Challenges:
GAO-10-843:
GAO Highlights:
Highlights of GAO-10-843, a report to congressional requesters.
Why GAO Did This Study:
The National Weather Service‘s (NWS) weather products are a vital
component of the Federal Aviation Administration‘s (FAA) air traffic
control system. In addition to providing aviation weather products
developed at its own facilities, NWS also provides on-site staff at
each of FAA‘s en route centers”the facilities that control high-
altitude flight outside the airport tower and terminal areas. NWS‘s on-
site staff is called a center weather service unit. For several years,
NWS and FAA have been exploring options for improving the aviation
weather services provided at en route centers.
GAO agreed to (1) determine the status of the agencies‘ efforts to
restructure aviation weather services, (2) assess the agencies‘
progress in establishing performance baselines in order to measure the
effect of any changes, and (3) evaluate plans to address key
challenges. To do so, GAO evaluated agency progress and plans and
compared agency efforts with leading practices.
What GAO Found:
After developing and shelving four proposals for restructuring the
center weather service units over the last 5 years, in July 2010,
senior NWS and FAA officials agreed to continue the current center
weather service units at each of the 21 en route centers through
September 2011 and to take immediate steps to improve aviation weather
services by (1) having the service units provide forecasts for 10 key
FAA terminal radar facilities and (2) having nearby weather forecast
offices support FAA‘s en route centers when the service units are
closed for the night. In addition, the agencies agreed to establish a
joint team to baseline current capabilities and develop firm
requirements for aviation weather services supporting air traffic flow
management. While this agreement is important, the details have not
been fully defined. Thus, it is not yet clear what will happen to the
21 service units after September 2011, when the immediate improvements
in services will be in place, whether there are any costs associated
with these steps, and who will pay for them. Until the two agencies
further define their plans, the risk remains that the agencies will
misjudge their responsibilities and not fulfill their agreements.
FAA and NWS have made progress in identifying performance measures for
the weather service units located at FAA en route centers, and NWS is
beginning to track its service units‘ performance. However, NWS has
not yet tracked, established baselines for, and reported to FAA on all
applicable performance measures. Specifically, of seven possible
performance measures, NWS is tracking performance for three of the
measures and partially tracking a fourth measure. Of these four
measures, the agency has established a sound baseline and reported on
two of these measures and has made partial progress on two others. The
agency is not tracking performance, documenting baselines, or
reporting on three of the measures because it has not yet determined
how to track them. Without an understanding of the current level of
performance of the identified measures, the agencies will be limited
in their ability to evaluate what progress has been made. In addition,
until NWS regularly reports on its performance, the agencies lack the
information they need to determine what is working well and what needs
to be improved.
In September 2009, GAO identified three challenges in modifying NWS‘s
aviation weather services provided at FAA‘s en route centers:
achieving interagency collaboration, defining requirements, and
aligning changes with the Next Generation Air Transportation System
(NextGen)-”a long-term initiative to increase the efficiency of the
national airspace system. The agencies have not yet fully addressed
these challenges. Specifically, while senior agency officials recently
agreed on how to proceed, work remains to be done to refine
requirements, develop and execute an implementation plan, and to
ensure that improvements are aligned with the long-term vision for
NextGen. Until these fundamental challenges are addressed, the
agencies are unlikely to achieve significant improvements in the
aviation weather services provided at en route centers.
What GAO Recommends:
GAO recommends that the Departments of Commerce and Transportation
define their agreements, refine performance management processes, and
address key challenges. In commenting on a draft of this report,
Commerce agreed with GAO‘s recommendations and identified plans to
address them; Transportation agreed to consider the recommendations.
View [hyperlink, http://www.gao.gov/products/GAO-10-843] or key
components. For more information, contact David A. Powner at (202) 512-
9286 or pownerd@gao.gov.
[End of section]
Contents:
Letter:
Background:
NWS and FAA Officials Agreed to Improve Services, but This Agreement
Is Not Fully Defined:
NWS Has Made Progress in Measuring Center Weather Service Unit
Performance but Has Not Yet Established Performance Baselines or
Reported on Key Measures:
NWS and FAA Have Not Fully Addressed Key Challenges:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Key FAA Concerns and NWS Steps to Address Them:
Appendix III: Comments from the Department of Commerce:
Appendix IV: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Key Weather Products Provided by the Aviation Weather Center:
Table 2: Key Systems Used in the Center Weather Service Units:
Table 3: Key Products and Services Provided by Center Weather Service
Units:
Table 4: Chronology of Efforts by FAA and NWS:
Table 5: Performance Measures for Center Weather Service Units:
Table 6: Key FAA Concerns and NWS Actions:
Figures:
Figure 1: FAA Facilities Involved in Air Traffic Control:
Figure 2: Center Weather Service Unit Locations and Service Areas:
Figure 3: Analysis of NWS Performance Measurement:
Abbreviations:
FAA: Federal Aviation Administration:
NextGen: Next Generation Air Transportation System:
NOAA: National Oceanic and Atmospheric Administration:
NWS: National Weather Service:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
September 9, 2010:
The Honorable Bart Gordon:
Chairman:
Committee on Science and Technology:
House of Representatives:
The Honorable Brad Miller:
Chairman:
Subcommittee on Investigations and Oversight:
Committee on Science and
Technology: House of Representatives:
The National Weather Service (NWS) plays a significant role in
providing weather services to the aviation community. NWS's weather
products and data are vital components of the Federal Aviation
Administration's (FAA) air traffic control system, providing weather
information to local, regional, and national air traffic management,
navigation, and surveillance systems. NWS aviation weather products
include forecasts and warnings of meteorological conditions that could
affect air traffic, including thunderstorms, air turbulence, and
icing. In addition to providing aviation weather products that are
developed at its own facilities, NWS also provides staff on site at
each of FAA's en route centers--the facilities that control high-
altitude flight outside the airport tower and terminal areas. This
group of NWS meteorologists--called a center weather service unit--
provides air traffic staff with forecasts, advisories, and periodic
weather briefings on regional conditions.
Over the last few years, FAA and NWS have been exploring options for
enhancing the efficiency of the aviation weather services provided at
en route centers. In September 2005, FAA asked NWS to restructure its
services to be more efficient. Since then, at FAA's request, NWS has
submitted five different proposals for restructuring its services.
Because of your interest in possible changes to NWS's aviation weather
services, we agreed to (1) determine the status of the agencies'
efforts to restructure aviation weather services and products, (2)
assess the agencies' progress in establishing performance baselines in
order to measure the effect of any changes, and (3) evaluate plans to
address key challenges.
To address our objectives, we analyzed agency documentation, including
FAA's requirements for aviation weather services and NWS's proposals.
We evaluated the agencies' progress in establishing a baseline of
performance and compared NWS's performance measurement practices with
government and industry guidance and best practices. To evaluate plans
to address challenges, we compared agency efforts with leading
practices in industry and government on interagency collaboration and
system development. We also interviewed officials of both agencies.
We conducted our work at NOAA and FAA facilities in the Washington,
D.C., metropolitan area. We conducted this performance audit from
October 2009 to September 2010, in accordance with generally accepted
government auditing standards. Those standards require that we plan
and perform the audit to obtain sufficient, appropriate evidence to
provide a reasonable basis for our findings and conclusions based on
our audit objectives. We believe that the evidence obtained provides a
reasonable basis for our findings and conclusions based on our audit
objectives. Additional details on our objectives, scope, and
methodology are provided in appendix I.
Background:
FAA is responsible for ensuring safe, orderly, and efficient air
travel in and around the United States. NWS supports FAA by providing
aviation-related forecasts and warnings at air traffic facilities
across the country. Among other support and services, NWS provides
four meteorologists at each of FAA's 21 en route centers to provide on-
site aviation weather services. This arrangement is defined and funded
under an interagency agreement.
FAA's Mission and Organizational Structure:
In performing its primary mission to ensure safe air travel, FAA
reported that air traffic in the national airspace system exceeded 43
million flights and 745 million passengers in 2008. In addition, at
any one time, as many as 7,000 aircraft--both civilian and military--
could be aloft over the United States. In 2004, FAA's Air Traffic
Organization was formed to, among other responsibilities, improve the
provision of air traffic services. More than 34,000 employees within
FAA's Air Traffic Organization support the operations that help move
aircraft through the national airspace system. The agency's ability to
fulfill its mission depends on the adequacy and reliability of its air
traffic control systems, as well as weather forecasts made available
by NWS and automated systems. These resources reside at, or are
associated with, several types of facilities: air traffic control
towers, terminal radar approach control facilities, air route traffic
control centers (en route centers), and the Air Traffic Control System
Command Center. The number and functions of these facilities are as
follows:
* 510 air traffic control towers manage and control the airspace
within about 5 miles of an airport. They control departures and
landings, as well as ground operations on airport taxiways and runways.
* 163 terminal radar approach control facilities provide air traffic
control services for airspace within approximately 40 miles of an
airport and generally up to 10,000 feet above the airport, where en
route centers' control begins. Terminal controllers establish and
maintain the sequence and separation of aircraft.
* 21 en route centers control planes over the United States--in
transit and during approaches to some airports. Each center handles a
different region of airspace. En route centers operate the computer
suite that processes radar surveillance and flight planning data,
reformats the data for presentation purposes, and sends it to display
equipment used by controllers to track aircraft. The centers control
the switching of voice communications between aircraft and the center,
as well as between the center and other air traffic control
facilities. Four of these en route centers also control air traffic
over the oceans.
* The Air Traffic Control System Command Center manages the flow of
air traffic within the United States. This facility regulates air
traffic when weather, equipment, runway closures, or other conditions
place stress on the national airspace system. In these instances,
traffic management specialists at the command center take action to
modify traffic demands in order to keep traffic within system capacity.
See figure 1 for a visual summary of the facilities that control and
manage air traffic over the United States.
Figure 1: FAA Facilities Involved in Air Traffic Control:
[Refer to PDF for image: illustration]
Air Traffic Control Tower:
Preflight;
Takeoff.
Terminal Radar Approach Control:
Departure.
Air Route Traffic Control Center:
Departure;
En route;
descent.
Terminal Radar Approach Control:
Descent;
Approach.
Air Traffic Control Tower:
Landing.
Source: GAO analysis of FAA data.
[End of figure]
NWS's Mission and Organizational Structure:
The mission of NWS--an agency within the Department of Commerce's
National Oceanic and Atmospheric Administration (NOAA)--is to provide
weather, water, and climate forecasts and warnings for the United
States, its territories, and its adjacent waters and oceans to protect
life and property and to enhance the national economy. In addition,
NWS is the official source of aviation-and marine-related weather
forecasts and warnings, as well as warnings about life-threatening
weather situations.
The coordinated activities of weather facilities throughout the United
States allow NWS to deliver a broad spectrum of climate, weather,
water, and space weather services in support of its mission. These
facilities include 122 weather forecast offices located across the
country that provide a wide variety of weather, water, and climate
services for their local county warning areas, including advisories,
warnings, and forecasts; 9 national prediction centers[Footnote 1]
that provide nationwide computer modeling to all NWS field offices;
and 21 center weather service units that are located at FAA en route
centers across the nation and provide meteorological support to air
traffic controllers.
NWS Provides Aviation Weather Services to FAA:
As an official source of aviation weather forecasts and warnings,
several NWS facilities provide aviation weather products and services
to FAA and the aviation sector. These facilities include the Aviation
Weather Center, weather forecast offices located across the country,
and 21 center weather service units located at FAA en route centers
across the country.
Aviation Weather Center:
The Aviation Weather Center located in Kansas City, Missouri, issues
warnings, forecasts, and analyses of hazardous weather for aviation.
Staffed by 55 personnel, the center develops warnings of hazardous
weather for aircraft in flight and forecasts of weather conditions for
the next 2 days that could affect both domestic and international
aviation.
The center also produces a Collaborative Convective Forecast Product,
a graphical representation of expected thunderstorms or related
conditions at 2, 4, and 6 hours. This is used by FAA to manage
aviation traffic flow across the country. The Aviation Weather
Center's key products are described in table 1.
Table 1: Key Weather Products Provided by the Aviation Weather Center:
Weather product: Significant Meteorological Information;
Description: An advisory concerning the occurrence or expected
occurrence of potentially hazardous weather conditions that may affect
the safety of aircraft operations in the en route environment.
Weather product: Convective Significant Meteorological Information;
Description: A text product describing the occurrence or expected
occurrence of thunderstorms and related weather conditions over the
contiguous United States within 2 hours of issuance time.
Weather product: Airman's Meteorological Information;
Description: An advisory concerning the occurrence or expected
occurrence of certain weather conditions that may affect the safety of
aircraft in the en route environment, but at intensities that do not
meet the criteria to develop a Significant Meteorological Information
product.
Weather product: Collaborative Convective Forecast Product;
Description: A graphical convection forecast developed for strategic
planning and management of en route air traffic. It is produced every
2 hours through collaboration--by way of an online chat room--among
the Aviation Weather Center, the Meteorological Services of Canada,
industry meteorologists, and the center weather service units. These
collaborative forecasts are produced between March 1 and October 31
every year.
Source: GAO analysis of NWS data.
[End of table]
Weather Forecast Offices:
NWS's 122 weather forecast offices issue terminal area forecasts for
approximately 632 locations every 6 hours or when conditions change,
consisting of the expected weather conditions significant to a given
airport or terminal area, and are primarily used by commercial and
general aviation pilots. The terminal area forecasts are updated every
3 hours for 35 key airports and every 2 hours for the airports in New
York, Atlanta, and Chicago.
Center Weather Service Units:
NWS's center weather service units are located at each of FAA's 21 en
route centers and operate 16 hours a day, 7 days a week (see figure
2). Each center weather service unit usually consists of three
meteorologists and a meteorologist-in-charge who provide strategic
advice and aviation weather forecasts to FAA traffic management
personnel. Governed by an interagency agreement, FAA currently
reimburses NWS approximately $13 million annually for this support.
Figure 2: Center Weather Service Unit Locations and Service Areas:
[Refer to PDF for image: illustrated map]
Center Weather Service Unit Locations:
Albuquerque;
Anchorage;
Atlanta;
Boston;
Chicago;
Cleveland;
Denver;
Fort Worth;
Houston;
Indianapolis;
Jacksonville;
Kansas City;
Los Angeles;
Memphis;
Miami;
Minneapolis;
New York;
Oakland;
Salt Lake City;
Seattle.
Washington, DC.
Sources: NWS (data); Map Resources (map).
Note: Four center weather service units (Anchorage, Oakland, Miami,
and New York) have oceanic responsibilities that are not pictured here.
[End of figure]
Center Weather Service Units: An Overview of Systems and Operations:
The meteorologists at the center weather service units use a variety
of systems to gather and analyze information compiled from NWS and FAA
weather sensors. Key systems used to compile weather information
include FAA's Weather and Radar Processor, FAA's Integrated Terminal
Weather System, FAA's Corridor Integrated Weather System, and a remote
display of NWS's Advanced Weather Interactive Processing System.
Meteorologists at several center weather service units also use NWS's
National Centers--Advanced Weather Interactive Processing System.
Table 2 provides a description of key systems.
Table 2: Key Systems Used in the Center Weather Service Units:
System: Weather and Radar Processor;
Description: This FAA system is used in en route centers and receives
NWS products and data, information from automated weather sensors
located at airports, and data from other sources such as weather
satellites and radars. It compiles the information and provides
current weather and forecasts to air traffic supervisors, traffic flow
managers, and the center weather service unit meteorologists.
System: Advanced Weather Interactive Processing System--Remote Display;
Description: This NWS system integrates hydrometeorological data from
a variety of sources and produces graphical displays at NWS weather
forecast offices, river forecast centers, and national centers. This
system aids forecaster analysis and decision making. Meteorologists at
the en route centers have access to this system through a Remote
Display system, which provides a dedicated connection to the
supporting weather forecast office. The Remote Display is funded by
FAA, and maintenance is provided by NWS.
System: Integrated Terminal Weather System;
Description: This FAA system furnishes air traffic controllers and
meteorologists with full-color graphic displays of weather information
concerning airport terminal airspace within a 60-mile radius. The
system also projects movement of severe weather systems up to 1 hour
in the future and has been installed at 39 airports.
System: Corridor Integrated Weather System;
Description: This FAA system is a prototype decision support tool that
gathers weather information to help controllers select the most
efficient routes for diverting traffic to avoid severe weather
conditions. This system provides traffic flow managers with
comprehensive convective weather data needed for tactical
modifications occurring within 2 hours to the operational plan. These
tactical modifications to the operational plan may include weather
impacts on air traffic control capacity, a need to modify the
mitigation plan, and the execution of a modified mitigation plan.
System: National Centers--Advanced Weather Interactive Processing
System;
Description: This NWS system is a meteorological data visualization
and integrated product generation system that provides a national
scope of weather information. It is composed of software that ingests,
analyzes, displays, and integrates various types of
hydrometeorological data including numerical model, surface, upper-
air, satellite, radar, and text data. This system is used in only a
few center weather service units.
Sources: GAO analysis of FAA and NWS data.
[End of table]
NWS meteorologists at the en route centers provide several products
and services to the FAA staff, including meteorological impact
statements, center weather advisories, periodic briefings, and on-
demand consultations. These products and services are described in
table 3. In addition, center weather service unit meteorologists
receive and disseminate pilot reports, provide input every 2 hours to
the Aviation Weather Center's creation of the Collaborative Convective
Forecast Product, train FAA personnel on how to interpret weather
information, and provide weather briefings to nearby terminal radar
approach control facilities and air traffic control towers.
Table 3: Key Products and Services Provided by Center Weather Service
Units:
Product or service: Meteorological impact statement;
Description: An unscheduled forecast of weather conditions that are
expected to adversely impact the flow of air traffic in the en route
center's area of responsibility within 4 to 12 hours.
Product or service: Center weather advisory;
Description: A short-term, unscheduled warning of hazardous weather
conditions used primarily by air crews to anticipate and avoid adverse
weather conditions in the en route and terminal environments. It
describes current weather conditions or adverse weather conditions--
such as moderate to severe icing or turbulence, thunderstorms, low-
level wind shear, and low ceilings and visibility--beginning within
the next 2 hours.
Product or service: Briefings;
Description: Short updates provided by NWS meteorologists to FAA
supervisors twice a day: these briefings include current weather
warnings and advisories, a summary of forecasted weather across the
national airspace, terminal forecasts, and other pertinent
meteorological information.
Product or service: On-demand consultation;
Description: Unscheduled verbal presentations regarding ongoing or
expected weather conditions provided to FAA traffic control personnel,
supervisors, and other FAA facilities.
Sources: GAO analysis of FAA and NWS data.
[End of table]
FAA Seeks to Improve Aviation Weather Services Provided at En Route
Centers:
In recent years, FAA has undertaken multiple initiatives to assess and
improve the performance of the center weather service units.[Footnote
2] Studies conducted in 2003 and 2006 highlighted concerns with the
lack of standardization of products and services at NWS's center
weather service units. To address these concerns, the agency sponsored
studies that determined that weather data could be provided remotely
using current technologies, and that private sector vendors could
provide these services. In 2005, the agency requested that NWS
restructure its aviation weather services by consolidating its center
weather service units to a smaller number of sites, reducing personnel
costs, and providing products and services 24 hours a day, 7 days a
week. NWS subsequently submitted a proposal for restructuring its
services, but FAA declined the proposal citing the need to refine its
requirements.
In December 2007, FAA issued revised requirements and asked NWS to
respond with proposals defining the technical and cost implications of
three operational concepts. The three concepts involved (1) on-site
services provided within the existing configuration of offices located
at the 21 en route centers, (2) remote services provided by a reduced
number of regional facilities, and (3) remote services provided by a
single centralized facility. NWS responded with three proposals, but
FAA rejected these proposals in September 2008, noting that while
elements of each proposal had merit, the proposed costs were too high.
FAA requested that NWS revise its proposal to bring costs down while
stating a preference to move toward a single center weather service
unit with a backup site.
As a separate initiative, NWS began a series of improvements in order
to address FAA's key concerns. Specifically, in April 2008, the agency
initiated a program to improve the consistency of the center weather
service units' products and services. This program involved
standardizing the technology, collaboration, and training for all 21
center weather service units and conducting site visits to evaluate
and provide feedback to each unit. NWS reported that it completed
these efforts in 2009. A summary of FAA's key concerns and NWS's
efforts to address them is included in appendix II.
After two requests for deadline extensions on a new proposal, NWS
provided FAA with an updated proposal in June 2009 based on the two-
site approach FAA had requested in September 2008. FAA responded to
NWS's proposal by requesting more information and stated that the
agencies would work together to resolve issues. From September through
November 2009, the agencies completed a series of meetings to address
issues from the proposal and agreed that NWS would resubmit its
proposal in December 2009 to consolidate the service units. In
December 2009, FAA revised its requirements to reflect the agencies'
efforts aimed at improving center weather service operations. However,
NWS did not submit its proposal in December 2009 to consolidate the
center weather service units. According to NWS officials, they decided
not to submit the proposal because (1) the NWS labor union and others
raised concerns about consolidating offices, (2) NWS could implement
technical improvements more quickly under the current organizational
structure, and (3) the agency wanted to focus its efforts and
resources on future weather system development rather than
restructuring existing operations. Table 4 provides a chronology of
the agencies' assessment and improvement efforts.
Table 4: Chronology of Efforts by FAA and NWS:
Time frame: November 2003;
Activity: FAA performed a functional audit of center weather service
units and found that the services provided at different en route
locations were inconsistent, that products were not standardized, and
there was little communication and collaboration between neighboring
service units.
Time frame: September 2005;
Activity: FAA requested that NWS restructure its aviation weather
services to provide improved services more efficiently. Specifically,
FAA requested that NWS consolidate 20 of the center weather service
units (excluding the unit in Alaska) to a smaller number of sites,
reduce NWS personnel costs by 20 percent, and deliver forecast
products and services 24 hours a day, 7 days a week.
Time frame: January 2006;
Activity: FAA initiated an analysis of the value of different
activities performed by the center weather service units. Similar to
the 2003 study, the results of this analysis noted the lack of
standardization of products, services, tools, and procedures. In
addition, the report found that quality assurance was provided on an
informal basis, there was no formal feedback process for products and
services, and meteorological training was not standardized.
Time frame: August 2006;
Activity: NWS conducted a prototype in which center weather service
unit products and services were completed and delivered remotely from
the closest weather forecast office. This prototype showed that remote
operations were possible and effective, but that they would be
difficult to implement because of the need for cultural change,
technology upgrades, and communication stability.
Time frame: September 2006;
Activity: An FAA study reported that it was possible to deliver
weather information, products, and services from one or many locations
with currently available state-of-the-art technology platforms.
Time frame: October 2006;
Activity: FAA administered a market survey to determine whether the
private sector could provide remote weather services at a lower cost
than currently provided. Ten organizations, including private sector
firms and government-funded laboratories, responded that they could
provide the services that FAA wanted; Separately, NWS presented its
first proposal for restructuring its aviation weather services to FAA.
In this proposal, NWS suggested moving meteorologists from the en
route centers to regional weather forecast offices and providing
remote aviation weather services from the weather forecast offices.
Time frame: April 2007;
Activity: FAA declined NWS's proposal. Instead, FAA reported that it
would redefine its requirements for the functions provided by the
center weather service units.
Time frame: December 2007;
Activity: FAA transmitted its redefined requirements to NWS and
requested a written response detailing three different operational
concepts.
Time frame: April 2008;
Activity: NWS initiated a short-term improvement program for the
center weather service units. The goal of this program was to
standardize the technology and training for the units to improve the
consistency of products and services.
Time frame: May 2008;
Activity: In response to the new requirements, NWS provided FAA with a
second series of proposals, which identified three alternatives for
restructuring the center weather service units.
Time frame: September 2008;
Activity: FAA rejected NWS's second series of proposals, stating that
while elements of each alternative had merit, the agency could not
accept them because the costs were too high. FAA requested that NWS
deliver a revised proposal by December 2008, stating a preference to
move toward a single center weather service unit with a back-up site.
Time frame: December 2008;
Activity: NWS requested and FAA approved a 60-day extension on the
deadline for NWS's third proposal to allow time to address any public
misperceptions regarding the proposed changes.
Time frame: February 2009;
Activity: NWS requested another 60-day extension on the deadline for
NWS's proposal to allow the new NOAA Administrator time to work with
the then-unnamed FAA Administrator on the consolidation. FAA approved
a 30-day extension.
Time frame: June 2009;
Activity: NWS provided FAA with its third proposal, which was to
consolidate 20 of the 21 center weather service units into two
locations.
Time frame: August 2009;
Activity: FAA responded to NWS's proposal by requesting more
information and stating that the agencies would work together to
resolve issues.
Time frame: November 2009;
Activity: The agencies completed a series of meetings to resolve
issues regarding the proposal and agreed that NWS would submit its
fourth proposal to consolidate the center weather service units in
December 2009.
Time frame: December 2009;
Activity: FAA revised its prior set of requirements to reflect the
agencies' collaboration during meetings held earlier in the year; NWS
did not provide its fourth proposal to consolidate center weather
service units to FAA because (1) the NWS labor union and others raised
concerns about consolidating offices, (2) NWS could implement
technical improvements more quickly under the current organizational
structure, and (3) the agency wanted to focus its efforts and
resources on future weather system development rather than
restructuring operations. NWS officials decided to consider other
options.
Sources: GAO analysis of NWS and FAA data.
[End of table]
Prior GAO Reports Identified Concerns with Center Weather Service
Units; Recommended Steps to Improve Quality Assurance:
In January 2008, we reported on concerns about inconsistencies in
products and quality among center weather service units.[Footnote 3]
We noted that while both NWS and FAA have responsibilities for
assuring and controlling the quality of aviation weather observations,
neither agency monitored the accuracy and quality of the aviation
weather products provided at center weather service units, performed
annual evaluations of aviation weather services provided at en route
centers, and provided feedback to the center weather service units. We
recommended they do so. The Department of Commerce agreed with our
recommendations, and the Department of Transportation stated that FAA
planned to revise its requirements and that these would establish
performance measures and evaluation procedures.
In September 2009, we reported that the agencies were considering
plans to consolidate 20 of the 21 existing center weather service
units to two locations, but it was not clear whether and how the
changes would be implemented.[Footnote 4] Moreover, we reported that
NWS and FAA faced challenges in their efforts to improve the aviation
weather structure, including achieving interagency collaboration,
defining FAA's requirements, and aligning any changes with the Next
Generation Air Transportation System. We also identified three
challenges the agencies would face in implementing their plans--
developing a feasible schedule that includes adequate time for
stakeholder involvement, undertaking a comprehensive demonstration to
ensure no services are degraded, and effectively reconfiguring the
infrastructure and technologies. We recommended that the agencies
address these challenges, and NOAA and the Department of
Transportation agreed with our recommendations.
NWS and FAA Officials Agreed to Improve Services, but This Agreement
Is Not Fully Defined:
After developing and shelving four different proposals for
restructuring the center weather service units over the last 5 years,
NWS and FAA have reached agreement on how to improve aviation weather
services. In March 2010, NWS proposed maintaining the current 21
center weather service units collocated at en route centers,
increasing staffing at the Aviation Weather Center in order to provide
remote services during the service units' off-hours, and developing a
new collaborative weather product. NWS estimated that these
improvements would cost FAA about $3 million per year. This is in
addition to the annual cost of maintaining the existing 21 centers.
[Footnote 5] NWS also estimated that it would be able to implement the
proposal within 21 months. FAA responded that it was not prepared to
accept the proposal because of the increased costs.
Subsequently, in July 2010, FAA and NWS reached an agreement on the
steps the two agencies would take to improve aviation weather
services. Specifically, FAA proposed and NWS agreed to continue the
current center weather service units at each of the 21 en route
centers through September 2011 and to take immediate steps to improve
aviation weather services by (1) having the service units provide
forecasts at 10 key FAA terminal radar approach control facilities
[Footnote 6] and (2) providing around-the-clock coverage at all of the
en route centers by having the local weather forecast office support
the en route centers when the center weather service units are closed
for the night--a practice that currently is used at selected en route
centers. In addition, the agencies agreed to establish a joint team to
baseline current capabilities and develop firm requirements for NWS
products and services supporting FAA's air traffic flow management out
through 2015. The agencies expect that the joint team will establish
an implementation plan by November 2010 and then begin to implement it.
However, the agencies' documentation of this agreement does not
address the future locations of the center weather service units, or
provide details and a schedule for the proposed improvements to
services. As a result, it is not clear what will happen to the 21
service units after September 2011, when the immediate improvements in
services will be in place, whether there are any costs associated with
these steps, whether the benefits outweigh the costs, and who will pay
for them. Until this agreement is further defined in writing and
formalized between the two agencies, the risks remain that the
agencies will misjudge their responsibilities and not fulfill their
agreements.
NWS Has Made Progress in Measuring Center Weather Service Unit
Performance but Has Not Yet Established Performance Baselines or
Reported on Key Measures:
According to best practices in the federal government and in industry,
organizations should measure performance in order to evaluate the
success or failure of their activities and programs.[Footnote 7]
Performance measurement involves identifying performance goals and
measures, establishing performance baselines by tracking performance
over time, identifying targets for improving performance, and
measuring progress against those targets. In January 2008, we
recommended that NWS and FAA develop performance measures and track
metrics for the products and services provided by center weather
service units and that they provide feedback to the center weather
service units so that they could improve their performance.[Footnote
8] Further, in September 2009, we recommended that the agencies
approve their draft performance measures and establish performance
baselines so that they could understand the effects of any changes
from restructuring aviation weather services.[Footnote 9]
Over the past year, NWS has made progress in identifying performance
measures, tracking performance on selected measures, and reporting on
the selected measures; however, the agency is not yet tracking or
reporting on all applicable performance measures. In December 2008,
FAA provided NWS five performance measures of center weather service
unit performance. Under the current interagency agreement, NWS is
required to track and report to FAA on these measures. In addition, in
its last two proposals, NWS proposed additional measures, two of which
could be tracked under the current organizational structure and using
current products.[Footnote 10] We previously recommended that NWS
immediately identify the current level of performance of the proposed
measures that could be identified under the current organizational
structure, so that they will have a performance baseline to compare to
should they decide to implement operational changes. The agency agreed
with this recommendation. Table 5 describes the performance areas
applicable to the current center weather service unit structure.
Table 5: Performance Measures for Center Weather Service Units:
Performance measure: Product participation;
Description: 90 percent participation in the Collaborative Convective
Forecast Product when convection is expected to occur in an en route
center's domain;
Source: Required by interagency agreement.
Performance measure: Format consistency;
Description: Consistency of two products (the meteorological impact
statement and the center weather advisory)--including the products'
formats, information content, and procedures for issuance across all
en route centers;
Source: Required by interagency agreement.
Performance measure: Service provision (briefings);
Description: Daily briefings delivered twice a day are to be provided
100 percent of the time;
Source: Required by interagency agreement.
Performance measure: Service provision (organizational);
Description: Center weather service unit services are to be provided
on-site or in backup mode, 16 hours a day, 7 days a week 100 percent
of the time;
Source: Required by interagency agreement.
Performance measure: Forecast accuracy;
Description: A determination of the accuracy of forecasts used in
decisions for traffic management initiatives;
Source: Required by interagency agreement.
Performance measure: Customer satisfaction;
Description: A measure of satisfaction with product quality,
timeliness, accuracy, and customer service, as well as the number of
complaints received;
Source: Proposed by NWS.
Performance measure: Training completion;
Description: A measure of completion of standardized training;
Source: Proposed by NWS.
Sources: GAO analysis of FAA and NWS information.
[End of table]
NWS has started tracking performance for three of the seven measures
and is partially tracking a fourth. Specifically, NWS has tracked data
on each center weather service unit's (1) participation in the
Collaborative Convective Forecast Product, (2) organizational service
provision, and (3) customer satisfaction. Further, it has partially
tracked data on format consistency, by collecting data on one of two
required products. However, the agency has not tracked data on the
other measures for a number of reasons. For example, the agency did
not track the format consistency for the second of the two required
products because, until recently, the briefing has not had a
consistent format. Also, the agency is not tracking training
completion because it has not yet determined what standardized
training will be provided. For the forecast accuracy measure, agency
officials stated that they do not currently have the means to track
this measure, but that they are developing a tool to do so.
Of the measures it is tracking, NWS has established baselines and
reported its results on two measures and has partially done so for two
other measures. Specifically, NWS has established baselines on each
center weather service unit's participation in the Collaborative
Convective Forecast Product and its organizational service provision.
In addition, NWS has partially established a baseline on the format
consistency measure in that it has historical data for one of the two
required products.
However, because it has not tracked the format consistency of the
second product, NWS has not established a complete baseline for that
measure. Further, while NWS has calculated customer satisfaction
scores from its 2009 site evaluations, it does not yet have a reliable
baseline because it has not yet matured its approach to documenting
this measure. Specifically, NWS changed its approach during its 2010
site evaluations, which will make it harder to compare scores from
year to year. Moreover, the agency mixed positive and negative
findings to come up with its rating scores for some sites, thereby
rendering the 2009 scores at selected sites ineffective at measuring a
site's performance. Figure 3 identifies NWS's efforts to track data,
develop baselines, and report on the performance measurement areas.
Figure 3: Analysis of NWS Performance Measurement:
[Refer to PDF for image: table]
Performance measure: Product participation;
Currently tracking? Yes;
Baseline established? Yes;
Results reported? Yes;
Status and evaluation: NWS has collected data on this measure for each
center weather service unit since 2009. The agency provided feedback
to the service units and has reported its results to FAA.
Performance measure: Format consistency;
Currently tracking? Partial;
Baseline established? Partial;
Results reported? Partial;
Status and evaluation: NWS has collected data on one of two products
since 2009 and reported its results for the first three months of 2010
to FAA in its quarterly progress reports. NWS recently developed a
standardized format for the second product, but has not yet begun
tracking it.
Performance measure: Service provision (briefings);
Currently tracking? No;
Baseline established? No;
Results reported? No;
Status and evaluation: NWS has not proactively monitored this measure.
NWS currently relies on center weather service units to report when
there is an issue with briefings not being delivered. As of June 2010,
the agency did not have plans to track this measure because it had not
identified how to gather this data. In late August 2010, while this
report was in final processing, NWS proposed an approach for tracking
this measure.
Performance measure: Service provision (organizational);
Currently tracking? Yes;
Baseline established? Yes;
Results reported? Yes;
Status and evaluation: NWS collected data on the hours of operation of
each center weather service unit since 2009, and has reported its
results for the first three months of 2010 to FAA in its quarterly
progress report.
Performance measure: Forecast accuracy;
Currently tracking? No;
Baseline established? No;
Results reported? No;
Status and evaluation: NWS is developing a tool that will capture
forecast accuracy and plans to implement it later this year.
Performance measure: Customer satisfaction;
Currently tracking? Yes;
Baseline established? Partial;
Results reported? Partial;
Status and evaluation: NWS officials are measuring customer
satisfaction using a rating score derived from annual site
evaluations. The rating score is based on the number of negative
findings at each site. NWS conducted site evaluations of each service
unit in 2009 and reported its results to FAA. NWS recently began a new
round of site evaluations in 2010. However, the agency‘s approach is
not yet mature enough to form a reliable baseline of performance.
Specifically, NWS modified its approach to the site visits in 2010 to
factor in lessons learned from the 2009 visits, which will make it
harder to compare scores from year to year and identify performance
trends. Moreover, the agency mixed positive and negative findings to
come up with its rating scores for some sites, thereby rendering the
2009 scores at selected sites ineffective at measuring a site‘s
performance and not valuable as a feedback mechanism.
Performance measure: Training completion;
Currently tracking? No;
Baseline established? No;
Results reported? No;
Status and evaluation: While NWS reported it tracked the service units‘
completion of customer service training in 2008, the agency noted that
it has not yet determined what standardized training it will require.
Source: GAO analysis based on NWS information.
It is important for NWS and FAA to track performance in the identified
measures in order to understand the value currently provided and to
assess the impact of any changes they make to operations. Reporting
also helps improve performance. For example, after reporting on its
performance in product participation and organizational service
provision for 2009, NWS noted significant improvements in 2010.
Until the agencies track and develop a performance baseline for all
applicable measures, they will be limited in their ability to evaluate
progress that has been made and whether or not they are achieving
their goals. In addition, until NWS regularly reports on its
performance, the agencies lack the information they need to determine
what is working well and what needs to be improved. Moreover, as the
agencies refine their approach to performance measurement, it will be
important to revisit and refine the performance measures to ensure an
appropriate mix of process-and outcome-oriented measures. For example,
NWS could consider measuring the number of aircraft incidents
attributed to inaccurate aviation weather forecasts or the number of
weather-related delays as a percentage of all delays.
NWS and FAA Have Not Fully Addressed Key Challenges:
In September 2009, we identified three challenges that FAA and NWS
faced in modifying the current aviation weather structure: (1)
achieving interagency collaboration, (2) defining requirements, (3)
aligning changes with the Next Generation Air Transportation System
(NextGen)--a long-term initiative to increase the efficiency of the
national airspace system. The agencies have taken initial steps to
collaborate, refine requirements, and look for ways to align their
plans with NextGen, but they have not yet fully addressed the
challenges. Until these fundamental challenges are addressed, the
agencies are unlikely to achieve significant improvements in the
aviation weather services provided at en route centers.
Achieving Interagency Collaboration:
We have previously reported on key practices that can help enhance and
sustain interagency collaboration.[Footnote 11] The practices
generally consist of two or more agencies defining a common outcome,
establishing joint strategies to achieve the outcome, agreeing upon
agency roles and responsibilities, establishing compatible policies
and procedures to operate across agency boundaries, and developing
mechanisms to monitor, evaluate, and report the results of
collaborative efforts.
In September 2009, we reported that NWS and FAA had not defined a
common outcome for modifying the aviation weather services provided at
en route centers, established joint strategies, or agreed upon their
respective responsibilities.[Footnote 12] We recommended that the
agencies complete these activities. NOAA and the Department of
Transportation agreed with our recommendation.
Since September 2009, NWS and FAA have made progress in defining a
common outcome, but have not yet established joint strategies to
achieve the outcome or agreed upon agency responsibilities.
Specifically, in July 2010, the two agencies defined a common outcome
when they reached an agreement to continue the current center weather
service unit configuration at each of the 21 en route centers and to
take immediate steps to improve aviation weather services. The two
agencies also plan to form a team that will develop an implementation
plan by November 2010. However, the agreement does not provide the
details needed to establish joint strategies and only provides general
agency responsibilities. Until the agencies establish joint strategies
and agree on respective agency responsibilities, it may prove
difficult to move forward in efforts to improve aviation weather
services.
Defining Requirements:
According to best practices of leading organizations, requirements
describe the functionality needed to meet user needs and perform as
intended in the operational environment.[Footnote 13] A disciplined
process for developing and managing requirements can help reduce the
risks associated with developing or acquiring a system or product.
In September 2009, we reported that FAA's requirements were unstable
and recommended that the agencies establish and finalize requirements
for aviation weather services at en route centers.[Footnote 14] NOAA
and the Department of Transportation agreed with our recommendation.
FAA updated its requirements in December 2009 based on the work that
the two agencies did in the fall of 2009. However, these changes were
nullified by the more recent decision to continue with 21 center
weather service units. In July 2010, the two agencies agreed to
establish a joint team to develop firm requirements for NWS products
and services supporting FAA's air traffic flow management out to 2015,
including those provided by the center weather service units. While
this is an important step, significant work remains to be done to
revise these requirements. Until the requirements are in place, the
agencies may find it difficult to move forward in efforts to improve
aviation weather services.
Alignment with the Next Generation Air Transportation System:
In September 2009, we reported that neither FAA nor NWS had ensured
that their restructuring plans fit with the national vision for
NextGen--a long-term initiative to transition FAA from the current
radar-based system to an aircraft-centered, satellite-based system. We
recommended that the agencies ensure that any proposed organizational
changes are aligned by seeking a review by the Joint Planning and
Development Office,[Footnote 15] the office responsible for planning
and coordinating NextGen. NOAA and the Department of Transportation
agreed with our recommendation.
Among other agreements in July 2010, the two agencies plan to work
together to develop requirements and an implementation plan that
extends through 2015--the NextGen Midterm Operating Capability date--
by November 2010. However, because this plan has not been developed or
approved, it is not clear that future actions will be aligned with
NextGen. As NWS and FAA discuss the current proposal and plan
improvements to aviation weather services, it will be important for
the agencies to continue to ensure alignment with the long-term goals
of NextGen.
Conclusions:
After many years of proposals and counter proposals for improving the
center weather service units, NWS and FAA recently agreed to continue
the current center weather service unit configuration at each of the
21 en route centers through September 2011 and to take immediate steps
to improve aviation weather services. However, many questions remain
about what will happen, when, and at what cost. Given the long history
of unresolved issues between FAA and NWS regarding the center weather
service units, it is more important than ever that the two agencies be
extremely clear on what their commitments entail.
An important component of any effort to improve operations is a solid
understanding of current performance. While NWS has made progress in
measuring the performance of the center weather service units, is not
adequately documenting performance baselines or reporting on several
of its performance measures. Further, the agency has begun efforts to
measure customer satisfaction, but the process is immature, and the
results are unreliable. Specifically, NWS has changed its approach to
the annual evaluations making it difficult to compare performance from
year to year, and its scoring process mixes positive and negative
findings for several sites. As a result, the scores may not accurately
reflect each center's performance. Until NWS has a solid understanding
of the current level of performance, it will be limited in its ability
to evaluate what progress has been made and whether or not it is
achieving its goals.
As the agencies move forward with plans to make aviation weather
services more efficient, they continue to face challenges, including a
record of false starts on interagency collaboration, unstable
requirements, and a lack of assurance that operational changes will
align with the future vision of NextGen. Until these challenges are
fully addressed, the agencies will likely find it difficult to make
meaningful changes in aviation weather services.
Recommendations for Executive Action:
To improve the aviation weather products and services provided at
FAA's en route centers, we are making three recommendations to the
Secretaries of Commerce and Transportation. Specifically, we recommend
that the Secretaries direct the NWS and FAA Administrators to:
* define, document, and sign the agencies' recent agreements on (1)
the locations of the center weather service units, (2) immediate
improvements in aviation weather services and operating hours, and (3)
the development of an implementation plan for improvements through
2015;
* ensure that NWS regularly tracks progress, documents performance
baselines, and reports on its format consistency, forecast accuracy,
and training performance measures; and:
* ensure that NWS develops a reliable customer satisfaction baseline
by refining the questions used during annual evaluations, so that
comparable information is collected from year to year, and revising
the scoring process to ensure that scores accurately reflect each
center's performance.
* In addition, we are reiterating our prior recommendations to the two
agencies to address key challenges in achieving interagency
collaboration, defining requirements, and aligning any organizational
changes with plans for NextGen.
Agency Comments and Our Evaluation:
We received written comments on a draft of this report from the
Secretary of Commerce, who transmitted NOAA's comments (see appendix
III). In its comments, NOAA stated that our report is generally
representative of challenges facing NWS and FAA in the execution of
aviation weather services provided by the center weather service
units. The agency agreed with our recommendations and identified plans
to implement selected parts of the recommendations. Specifically, NOAA
reiterated its plan to form a joint NWS/FAA team to determine weather
requirements for traffic flow management and to implement products and
services through the year 2015. NOAA stated that this team's results
will serve as additional documentation of the agreements. In addition,
NOAA reported that it plans to begin measuring format consistency in
September 2010 and forecast accuracy in December 2010.
NOAA also noted that it was using 2009 site evaluations as the basis
for its scoring and that the 2009 results would serve as a baseline
for comparison to 2010 and subsequent results. However, as we discuss
in the report, our analysis of the 2009 site evaluations and scoring
process found that the results were not reliable because the process
for collecting information on customer service was inconsistent, and
the scores did not always accurately reflect the centers' performance.
As a result, the 2009 scores are not useful as a baseline or as a
feedback tool. Moving forward, as NOAA analyzes the results from its
ongoing 2010 site evaluations, it will be important to ensure that the
scores accurately reflect each center's performance. Further, in
future years, it will be important to ensure that comparable
information is collected from year to year so that a reliable
performance baseline can be established.
The Department of Transportation's Director of Audit Relations
provided comments on a draft of this report via e-mail. In those
comments, he noted that the department agreed to consider our
recommendations.
Both departments also provided technical comments that we incorporated
as appropriate.
As agreed with your offices, unless you publicly announce the contents
of this report earlier, we plan no further distribution until 30 days
from the report date. At that time, we will send copies of this report
to the appropriate congressional committees, the Secretary of
Commerce, the Secretary of Transportation, the Director of the Office
of Management and Budget, and other interested parties. The report
also will be available at no charge on the GAO Web site at [hyperlink,
http://www.gao.gov].
If you or your staff members have any questions about this report,
please contact me at (202) 512-9286 or by e-mail at pownerd@gao.gov.
Contact points for our Offices of Congressional Relations and Public
Affairs may be found on the last page of this report. GAO staff who
made major contributions to this report are listed in appendix IV.
Signed by:
David A. Powner:
Director, Information Technology:
Management Issues:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
Our objectives were to (1) determine the status of the agencies'
efforts to restructure aviation weather services and products, (2)
assess the agencies' progress in establishing performance baselines in
order to measure the effect of any changes, and (3) evaluate plans to
address key challenges.
To determine the status of the agencies' efforts to restructure
aviation weather services and products, we analyzed Federal Aviation
Administration (FAA) and National Weather Service (NWS) documentation,
including FAA's requirements for center weather service units, the
interagency agreement between FAA and NWS, and NWS's proposals to meet
FAA needs for center weather service units. We also interviewed
officials from both agencies to discuss their plans and status in
reaching a decision on proposed changes.
To assess the agencies' progress in establishing performance
baselines, we identified the agencies' previous efforts to establish
baselines and evaluated the extent to which they have made progress in
doing so. We analyzed NWS's approach to measuring center weather
service unit performance and compared its performance measurement
practices with guidance and best practices in performance management
identified by government and industry. Specifically, we assessed the
agencies' actions taken to identify performance measures, track them,
establish baselines of performance, and report on those baselines. We
also assessed the reliability of the performance data that NWS
reported. Specifically, for the customer satisfaction measurements, we
analyzed supporting data and calculated customer satisfaction scores
using NWS's guidance for developing scores. We then compared the
scores we calculated with NWS's scores. In instances where our scores
did not match NWS's, we interviewed agency officials in order to
determine why NWS's scores did not match our own, focusing on four
sites with the largest number of findings. We found that the agency's
customer satisfaction data was not reliable. For the other reported
measures, we evaluated supporting data and interviewed responsible
agency officials to determine the agency's processes for validating
the data. We found that the data reported for these performance
measures was sufficient to meet our reporting purposes.
To evaluate plans to address key challenges identified in our prior
report, we reviewed agency documents including FAA requirements, an
NWS proposal, plans for the Next Generation Air Transportation System
(NextGen), and FAA's response to NWS's proposal. We compared agency
efforts with leading practices in industry and government on
interagency collaboration and system development. In addition, we
interviewed the FAA contracting officer's technical representative for
the center weather service units to discuss the challenges the agency
would have in implementing NWS's proposal, as well as the agency's
plans to ensure requirements were stabilized. We also interviewed NWS
officials to discuss their plans for aligning their system development
initiatives with NextGen. We also interviewed the co-chair of the
weather working group of the Joint Planning and Development Office to
determine whether the office had reviewed NWS's proposal and if the
office had concerns about the proposal's impact on NextGen.
We conducted our work at National Oceanic and Atmospheric
Administration (NOAA) and FAA facilities in the Washington, D.C.,
metropolitan area. We conducted this performance audit from October
2009 to September 2010, in accordance with generally accepted
government auditing standards. Those standards require that we plan
and perform the audit to obtain sufficient, appropriate evidence to
provide a reasonable basis for our findings and conclusions based on
our audit objectives. We believe that the evidence obtained provides a
reasonable basis for our findings and conclusions based on our audit
objectives.
[End of section]
Appendix II: Key FAA Concerns and NWS Steps to Address Them:
Table 6 lists concerns that FAA identified in a series of studies
between 2003 and 2006, as well as the steps that NWS has taken to
address these concerns.
Table 6: Key FAA Concerns and NWS Actions:
Category: Products and services;
FAA concern: Lack of standardized products and services;
NWS actions taken:
* Conducted a terminal forecast improvement program;
* Established a Web presence for the center weather service units;
* Inserted aviation products in NWS operational system;
* Created enhanced terminal forecast monitoring tools;
* Initiated performance measurement of consistency in products and
services.
Category: Products and services;
FAA concern: Dissatisfaction with the Collaborative Convective
Forecast Product (particularly with accuracy and the time it takes to
produce the product);
NWS actions taken:
* Initiated performance measurement of participation in this product;
* Supported FAA research and development of tools that enhance NWS
convective forecasts.
Category: Products and services;
FAA concern: Confusion on priority of center weather service unit
services;
NWS actions taken: No actions taken.
Category: Products and services;
FAA concern: Need for a forecast for terminal radar approach control
facilities;
NWS actions taken:
* Established these forecasts in certain locations.
Category: Procedures;
FAA concern: Lack of standardized procedures;
NWS actions taken:
* Established a Web presence for the center weather service units.
Category: Procedures;
FAA concern: Lack of collaboration with nearby NWS offices;
NWS actions taken:
* Conducted a terminal forecast improvement program;
* Created tools for enhanced terminal forecast monitoring.
Category: Procedures;
FAA concern: Category: Inadequate oversight of the center weather
service units;
NWS actions taken:
* Conducted center weather service unit site reviews.
Category: Procedures;
FAA concern: Lack of routine evaluation and feedback;
NWS actions taken:
* Initiated annual site evaluations;
* Established a new quality assurance manager position.
Category: Training;
FAA concern: Inadequate training;
NWS actions taken:
* Conducted NWS meteorology training.
Category: Training;
FAA concern: Lack of knowledge of center weather service unit
customers;
NWS actions taken:
* Conducted customer service training;
* Sent meteorologists to FAA's air traffic training.
Category: Organizational support;
FAA concern: Need for services 24 hours a day, 7 days a week;
NWS actions taken: No actions taken.
Category: Organizational support;
FAA concern: Need for a reduction in the cost of services;
NWS actions taken: No actions taken.
Category: Organizational support;
FAA concern: Need for restructuring to a smaller number of sites;
NWS actions taken: No actions taken.
Category: Organizational support;
FAA concern: Desire for on-site meteorologists[A];
NWS actions taken: No actions taken.
Category: Organizational support;
FAA concern: Need for performance-based service;
NWS actions taken:
* Initiated a performance measurement program (identified performance
measures and began measuring performance).
Sources: GAO analysis of NWS and FAA data.
[A] An FAA official stated that the concerns expressed over the years
have evolved and that the agency does not currently consider the
desire for on-site meteorologists as a concern.
[End of table]
[End of section]
Appendix III: Comments from the Department of Commerce:
The Secretary Of Commerce:
Washington, D.C. 20230:
August 23, 2010:
Mr. David A. Powner:
Director:
Information Technology Management Issues:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Mr. Powner:
Thank you for the opportunity to review and comment on the Government
Accountability Office's draft report entitled, "Aviation Weather:
Agencies Need to Improve Performance Measurement and Fully Address Key
Challenges" (GA0-10-843). On behalf of the Department of Commerce, I
have enclosed the National Oceanic and Atmospheric Administration's
programmatic comments to the draft report.
Sincerely,
Signed by:
Gary Locke:
Enclosure:
[End of letter]
Department of Commerce:
National Oceanic and Atmospheric Administration:
Comments to the Draft GAO Report Entitled "Aviation Weather: Agencies
Need to Improve Performance Measurement and Fully Address Key
Challenges" GAO-10-843 (September 2010):
General Comments:
The Department of Commerce's National Oceanic and Atmospheric
Administration (NOAA) appreciates the opportunity to review the
Government Accountability Office's (GAO) report on aviation weather.
The report is generally representative of challenges facing NOAA's
National Weather Service (NWS) and the Department of Transportation's
Federal Aviation Administration (FAA) in the execution of the aviation
weather services provided by Center Weather Services Units (CWSU). The
report contains three new recommendations; however, the text makes
reference to "four recommendations," the fourth being a reiteration or
reference to prior GAO recommendations to both NOAA and FAA. Since
NOAA has already responded to GAO's prior recommendations, NOAA's
response below addresses the three current recommendations only.
NOAA Response to GAO Recommendations:
The draft GAO report states, "To improve the aviation weather products
and services provided at FAA's en route centers, we are making four
recommendations to the Secretaries of Commerce and Transportation.
Specifically, we are recommending that the Secretaries direct the NWS
and FAA Administrators to..."
Recommendation 1: "define, document and sign the agencies' recent
agreements on (1) the locations of the center weather service units,
(2) immediate improvements in aviation weather services and operating
hours, and (3) the development of an implementation plan for
improvements through 2015;"
NOAA Response:
NOAA agrees with this recommendation. The Assistant Administrator for
Weather Services responded to the FAA in a letter dated July 29, 2010,
confirming the NWS' commitment to establish an NWS-FAA team to work
collaboratively to determine weather requirements for Traffic Flow
Management and implementation of products and services through 2015.
The results from the NWS/FAA Team will serve as additional
documentation of the agreement between NWS and FAA on the path forward
in providing aviation weather services to Traffic Flow Management.
Recommendation 2: "ensure that NWS regularly tracks progress,
documents performance baselines, and reports on its forecast
consistency, forecast accuracy, and training performance measures;"
NOAA Response:
NOAA agrees with this recommendation. We continue to track performance
of our CWSUs and will work with the FAA to improve our measures and
move them into the FAA's Aviation Weather Quality Management System.
Our tracking for forecast consistency will be in place for CWSU
products in September 2010, and will be provided to the FAA in our
Quarterly Reports. Tracking of forecast accuracy will begin in
December 2010.
Recommendation 3: "ensure that NWS develops a reliable customer
satisfaction baseline by refining the questions used during annual
evaluations so that comparable information is collected from year to
year and revising the scoring process to ensure that scores accurately
reflect each center's performance."
NOAA Response:
NOAA agrees with this recommendation. After conducting the initial
2009 Site Review program, NOAA established a baseline for customer
service based upon findings from the reviews. Specifically, the
Quality Verification Rating (QVR) scores calculated from the 2009 site
reviews will serve as the baseline measure of customer satisfaction,
and will be used to compare QVR scores from the 2010 and subsequent
site reviews.
[End of section]
Appendix IV: GAO Contact and Staff Acknowledgments:
GAO Contact:
David A. Powner, (202) 512-9286, or pownerd@gao.gov:
Staff Acknowledgments:
In addition to the individual named above, Colleen Phillips, Assistant
Director; Neil Doherty; Rebecca Eyler; Joshua Leiling; and Jessica
Waselkow made key contributions to this report.
[End of section]
Footnotes:
[1] These centers include the National Centers for Environmental
Prediction Central Operations, Aviation Weather Center, Environmental
Modeling Center, Hydrometeorological Prediction Center, Ocean
Prediction Center, Storm Prediction Center, Tropical Prediction
Center/National Hurricane Center, Climate Prediction Center, and the
Space Weather Prediction Center.
[2] FAA is also involved in a longer-term initiative to increase the
efficiency of the national airspace system and to improve its overall
safety. This initiative, called the Next Generation Air Transportation
System, is a joint effort of the National Aeronautics and Space
Administration, the White House Office of Science and Technology
Policy, and the Departments of Transportation, Homeland Security,
Defense, and Commerce. FAA anticipates that this initiative may lead
to major changes in the aviation weather program that would supersede
any short-term changes to aviation services.
[3] GAO, Aviation Weather: FAA Is Restructuring Services at Key
Centers; Both FAA and the National Weather Service Need to Better
Ensure Product Quality, [hyperlink,
http://www.gao.gov/products/GAO-08-258] (Washington, D.C.: Jan. 11,
2008).
[4] GAO, Aviation Weather: FAA and the National Weather Service are
Considering Plans to Consolidate Weather Service Offices, but Face
Significant Challenges, [hyperlink,
http://www.gao.gov/products/GAO-09-761] (Washington, D.C.: Sept. 9,
2009).
[5] NWS estimated that the cost of maintaining its 21 center weather
service units will grow over the next 5 years from roughly $13.2 to
$14.8 million.
[6] The 10 facilities are terminal radar approach control sites in
Northern California, Southern California, New York, Potomac, Chicago,
Dallas/Fort Worth, Houston, Atlanta, Miami, and Denver.
[7] Department of the Navy, Office of the Chief Information Officer,
Guide for Developing and Using Information Technology (IT) Performance
Measurements (Washington, D.C.: October 2001); and General Services
Administration, Office of Governmentwide Policy, Performance-Based
Management: Eight Steps To Develop and Use Information Technology
Performance Measures Effectively (Washington, D.C.: 1996).
[8] [hyperlink, http://www.gao.gov/products/GAO-08-258].
[9] [hyperlink, http://www.gao.gov/products/GAO-09-761].
[10] The other six performance measures cannot be measured now because
they involve new products or a consolidated number of office locations.
[11] GAO, Results-Oriented Government: Practices That Can Help Enhance
and Sustain Collaboration among Federal Agencies, [hyperlink,
http://www.gao.gov/products/GAO-06-15] (Washington, D.C.: Oct. 21,
2005).
[12] [hyperlink, http://www.gao.gov/products/GAO-09-761].
[13] Carnegie Mellon University Software Engineering Institute,
Capability Maturity Model® Integration for Development, Version 1.2
(Pittsburgh, Pa.: August 2006). Capability Maturity Model® and
Capability Maturity Modeling are registered in the U.S. Patent and
Trademark Office. CMM is a service mark of Carnegie Mellon University.
[14] [hyperlink, http://www.gao.gov/products/GAO-09-761].
[15] The Joint Planning and Development Office has multiple federal
partners, including FAA; the Departments of Transportation, Commerce,
Defense, and Homeland Security; the National Aeronautics and Space
Administration; and the White House Office of Science and Technology
Policy. As part of NextGen, the Joint Planning and Development Office
envisions restructuring air traffic facilities, including en route
centers, across the country as well as transitioning to new
technologies.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
