Weather Forecasting
National Weather Service's Operations Prototype Needs More Rigorous Planning Gao ID: GAO-07-650 June 8, 2007Using advanced systems and trained specialists located in 122 weather forecast offices throughout the country, the National Weather Service (NWS) provides storm and flood warnings and weather forecasts to protect life and property and to enhance the national economy. To improve the efficiency of its operations, in November 2006, NWS approved an effort to develop a prototype of an alternative way of operating. Under this prototype, weather forecasting offices would share selected responsibilities. GAO (1) determined the status of and plans for the prototype, (2) evaluated whether the prototype's justification was sufficient, (3) determined whether NWS's plans to evaluate the prototype are adequate, (4) evaluated whether NWS is sufficiently involving stakeholders in its prototype plans, and (5) determined how NWS plans to ensure that there will be no degradation of service during and after the prototype. To do so, GAO analyzed agency documentation and interviewed program officials and stakeholders.
NWS's prototype is currently on hold pending a reevaluation of the agency's approach. The agency had just begun the first phase of its three-phased prototype to demonstrate a new concept of operations over a 2-year period when, in late March 2007, the Department of Commerce's Under Secretary suspended the prototype because of concerns about the agency's approach. In the first phase, NWS established a program manager and began planning for the next two phases. During the remaining phases, NWS planned to have 20 weather forecasting offices share responsibilities in 2-office pairs and then in 4-office clusters. NWS then planned to decide whether to implement the new concept of operations on a national basis. The justification for the prototype was not sufficient. Before the prototype was suspended, the agency had approved moving forward with its prototype without conducting a cost-benefit analysis. NWS estimated that the prototype would cost approximately $9.3 million and would offer qualitative benefits, such as increased efficiency and an improved ability to focus on severe weather events, but did not quantify benefits or the expected return on its investment. If NWS were to proceed with the prototype without a cost-benefit analysis, it would lack assurance that its approach would be a cost-effective investment for the agency. NWS identified goals and selected measures to evaluate during its prototype activities, but it did not establish a rigorous evaluation plan. Specifically, NWS did not define a full set of needed measures, how it planned to compare prototype results with baseline performance, or how its selected measures supported the prototype goals. If NWS were to proceed without a rigorous evaluation plan, the agency would run an increased risk of not sufficiently measuring the impact of changes on its performance and could make decisions affecting the nation's weather on the basis of incomplete or flawed data. Although NWS involved internal stakeholders in planning its prototype, it did not (1) involve external stakeholders or (2) establish a plan that identified key stakeholders, the stakeholders' responsibilities, and a time line for involving stakeholders and addressing their comments. If NWS were to proceed with its prototype without such a plan, it could not ensure that stakeholder interests would be identified and addressed. NWS planned to mitigate the risk of degradation during and after the prototype by conducting laboratory exercises to understand the impact of the prototype, using a dedicated workstation in each office so that the office could switch to original systems if warranted, and monitoring its systems and products during the prototype. NWS officials stated that should the agency decide to implement the prototype on a national basis, it has standard procedures for testing and validating systems and software to avoid any degradation of service.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-07-650, Weather Forecasting: National Weather Service's Operations Prototype Needs More Rigorous Planning
This is the accessible text file for GAO report number GAO-07-650
entitled 'Weather Forecasting: National Weather Service's Operations
Prototype Needs More Rigorous Planning' which was released on July 9,
2007.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
June 2007:
Weather Forecasting:
National Weather Service's Operations Prototype Needs More Rigorous
Planning:
GAO-07-650:
GAO Highlights:
Highlights of GAO-07-650, a report to congressional requesters
Why GAO Did This Study:
Using advanced systems and trained specialists located in 122 weather
forecast offices throughout the country, the National Weather Service
(NWS) provides storm and flood warnings and weather forecasts to
protect life and property and to enhance the national economy. To
improve the efficiency of its operations, in November 2006, NWS
approved an effort to develop a prototype of an alternative way of
operating. Under this prototype, weather forecasting offices would
share selected responsibilities.
GAO (1) determined the status of and plans for the prototype, (2)
evaluated whether the prototype‘s justification was sufficient, (3)
determined whether NWS‘s plans to evaluate the prototype are adequate,
(4) evaluated whether NWS is sufficiently involving stakeholders in its
prototype plans, and (5) determined how NWS plans to ensure that there
will be no degradation of service during and after the prototype. To do
so, GAO analyzed agency documentation and interviewed program officials
and stakeholders.
What GAO Found:
NWS‘s prototype is currently on hold pending a reevaluation of the
agency‘s approach. The agency had just begun the first phase of its
three-phased prototype to demonstrate a new concept of operations over
a 2-year period when, in late March 2007, the Department of Commerce‘s
Under Secretary suspended the prototype because of concerns about the
agency‘s approach. In the first phase, NWS established a program
manager and began planning for the next two phases. During the
remaining phases, NWS planned to have 20 weather forecasting offices
share responsibilities in 2-office pairs and then in 4-office clusters.
NWS then planned to decide whether to implement the new concept of
operations on a national basis.
The justification for the prototype was not sufficient. Before the
prototype was suspended, the agency had approved moving forward with
its prototype without conducting a cost-benefit analysis. NWS estimated
that the prototype would cost approximately $9.3 million and would
offer qualitative benefits, such as increased efficiency and an
improved ability to focus on severe weather events, but did not
quantify benefits or the expected return on its investment. If NWS were
to proceed with the prototype without a cost-benefit analysis, it would
lack assurance that its approach would be a cost-effective investment
for the agency.
NWS identified goals and selected measures to evaluate during its
prototype activities, but it did not establish a rigorous evaluation
plan. Specifically, NWS did not define a full set of needed measures,
how it planned to compare prototype results with baseline performance,
or how its selected measures supported the prototype goals. If NWS were
to proceed without a rigorous evaluation plan, the agency would run an
increased risk of not sufficiently measuring the impact of changes on
its performance and could make decisions affecting the nation‘s weather
on the basis of incomplete or flawed data.
Although NWS involved internal stakeholders in planning its prototype,
it did not (1) involve external stakeholders or (2) establish a plan
that identified key stakeholders, the stakeholders‘ responsibilities,
and a time line for involving stakeholders and addressing their
comments. If NWS were to proceed with its prototype without such a
plan, it could not ensure that stakeholder interests would be
identified and addressed.
NWS planned to mitigate the risk of degradation during and after the
prototype by conducting laboratory exercises to understand the impact
of the prototype, using a dedicated workstation in each office so that
the office could switch to original systems if warranted, and
monitoring its systems and products during the prototype. NWS officials
stated that should the agency decide to implement the prototype on a
national basis, it has standard procedures for testing and validating
systems and software to avoid any degradation of service.
What GAO Recommends:
GAO is recommending that if NWS decides to proceed with its prototype,
it undertake a more rigorous approach to justifying, evaluating, and
involving stake-holders in the prototype. In written comments, Commerce
agreed with GAO‘s recommendations.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-650].
To view the full product, including the scope and methodology, click on
the link above. For more information, contact David Powner at (202) 512-
9286 or pownerd@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
NWS Initiated a 2-Year Prototype and Then Suspended It:
NWS Has Not Yet Sufficiently Justified Its Prototype:
NWS Lacks a Rigorous Evaluation Plan for Assessing Its Prototype:
NWS Involved Internal Stakeholders in Planning the Prototype, but Did
Not Involve External Stakeholders or Establish a Plan for Stakeholder
Involvement:
NWS Planned to Ensure That the Prototype Would Not Degrade Its Service:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: NWS Prototype Plans, as of March 2007:
Appendix III: Comments from the Department of Commerce:
Appendix IV: GAO Contact and Staff Acknowledgments:
Table:
Table 1: Clustered Peer Concept of Operations Prototype Costs, as of
March 2007:
Figures:
Figure 1: NWS's 122 Weather Forecast Offices:
Figure 2: Overview of Key Systems and Technologies Supporting NWS
Forecasts:
Figure 3: An AWIPS Workstation:
Figure 4: NWS Offices Involved in the Clustered Peer Concept of
Operations Prototype:
Figure 5: Preliminary Schedule for NWS's Clustered Peer Concept of
Operations Prototype, as of November 14, 2006:
Abbreviations:
AWIPS: Advanced Weather Interactive Processing System:
FAA: Federal Aviation Administration:
NOAA: National Oceanic and Atmospheric Administration:
NWS: National Weather Service:
United States Government Accountability Office:
Washington, DC 20548:
June 8, 2007:
The Honorable Nick Lampson:
Chairman:
The Honorable Bob Inglis:
Ranking Member:
Subcommittee on Energy and Environment:
Committee on Science and Technology:
House of Representatives:
The Honorable David Wu:
House of Representatives:
The Honorable Vernon J. Ehlers:
House of Representatives:
The National Weather Service's (NWS) ability to forecast the weather
affects the life and property of every American. The agency's basic
mission is to provide storm and flood warnings and weather forecasts
for the United States, its territories, and adjacent oceans and waters,
in order to protect life and property and to enhance the national
economy. NWS also supports other federal agencies' operations by
providing aviation-and marine-related weather forecasts and warnings.
To fulfill its mission, NWS operates 122 weather forecasting offices
and other specialized weather centers across the country. In these
offices, trained meteorologists use advanced systems to provide
forecasts and warnings for local geographic areas.
In an effort to improve its operational efficiency, in August 2005, NWS
chartered a working group to evaluate the roles, responsibilities,
functions, and supporting technology of weather forecast offices
nationwide and to make a proposal for a more efficient concept of
operations. In December 2005, the working group proposed an alternative
way of operating in which weather offices could share some weather
forecasting functions--particularly during high-intensity weather
events. After receiving this proposal, NWS chartered another team to
develop plans for a prototype of this concept to more fully evaluate
this approach.
Because of your interest in weather service operations, we (1)
determined the current status of and plans for the prototype, (2)
evaluated whether the justification for the prototype was sufficient,
(3) determined whether NWS's plans to evaluate the prototype are
adequate, (4) evaluated whether NWS is sufficiently involving
stakeholders in its prototype plans, and (5) determined how NWS plans
to ensure that there will be no degradation of service during and after
the prototype.
In late March 2007, after our audit work was nearly completed, the
Department of Commerce's Under Secretary for Oceans and Atmosphere
directed the Director of the National Weather Service to place all
activities associated with the agency's concept of operations on hold
pending a review by the Deputy Under Secretary. NWS officials stated
that the prototype was suspended because of concerns raised about the
agency's approach by some Members of Congress and by National Oceanic
and Atmospheric Administration (NOAA) leadership, the private sector,
NWS's workforce, and its critical emergency management partners.
Shortly after this decision was made, we met with your staffs and
agreed to continue with plans to issue this report to offer insight and
recommendations to NWS, should it decide to proceed with the prototype.
To address our objectives, we reviewed prototype plans and
presentations. We compared program documents with best practices for
justifying and evaluating prototypes and for involving stakeholders. We
also interviewed internal and external stakeholders and agency
officials. We conducted our work at NWS headquarters in the Washington,
D.C., metropolitan area; NWS offices in Tampa, Florida, Detroit,
Michigan, and Kansas City, Missouri; and a NOAA office in Boulder,
Colorado, because of the relevance of these sites to the prototype. We
performed our work from October 2006 to April 2007, in accordance with
generally accepted government auditing standards. Additional details on
our objectives, scope, and methodology are provided in appendix I.
Results in Brief:
NWS had begun the first phase of its three-phased prototype to
demonstrate a new concept of operations in which weather offices would
share selected forecasting responsibilities, when the Under Secretary
for Oceans and Atmosphere suspended the prototype because of concerns
about the agency's approach. As of March 2007, NWS had established a
program manager and begun to document how the prototype would be
executed and to develop training for employees who would participate.
During the second and third phases, NWS planned to have 20 weather
forecasting offices share forecasting responsibilities in 2-office
pairs and then in 4-office clusters. At the conclusion of the 2-year
prototype effort, NWS planned to make a decision on whether to proceed
to implement the new concept of operations on a national basis.
However, these efforts were halted when the agency suspended the
prototype pending a reevaluation on how best to proceed.
Before the prototype was suspended, NWS had approved moving forward
with its prototype plan without conducting an analysis of the relative
costs and benefits of the initiative. NWS estimated that the prototype
would cost approximately $9.3 million and identified expected
qualitative benefits--including increased efficiency, improved ability
to focus on high-impact weather events, and the ability to switch to a
backup site more quickly when a weather forecast office is disabled due
to loss of power or communications. However, the agency has not
conducted a cost-benefit analysis or determined whether there would be
a return on investment for the prototype. If NWS were to proceed with
the prototype without a cost-benefit analysis, it would lack assurance
that its approach would be a cost-effective investment for the agency.
NWS identified goals and selected measures to evaluate during its
prototype activities, but it did not establish a rigorous evaluation
plan. NWS's prototype plan identified evaluation goals, such as to
validate that a group of weather forecast offices, operating as one
unit, could be more effective and efficient than each office on its
own. Also, the prototype plan identified quantitative and qualitative
measures to be used in evaluating the prototype, including the number
of times that work was transferred between offices to focus on high-
impact events, the number of additional hours made available during
benign weather for nonforecast work, and customer perceptions of
service quality and timeliness. However, NWS did not establish an
evaluation plan that identified a comprehensive set of applicable
measures, what baseline performance its prototype would be compared
with, and how its measures supported the goals of the prototype. If NWS
were to proceed with its prototype without a rigorous evaluation plan,
it would run an increased risk of not sufficiently measuring the impact
of planned changes on its performance, and of subsequently making
decisions affecting the nation's weather service on the basis of
incomplete and flawed data.
NWS involved internal stakeholders in planning its prototype, including
having selected internal stakeholders review plans and provide feedback
on work products and laboratory activities. However, NWS did not
involve external stakeholders in its planning for the prototype or
establish a plan for stakeholder involvement during the prototype.
Specifically, NWS did not request input from external stakeholders,
including emergency managers, in the development and planning of the
prototype. In addition, the agency did not establish a plan that
identified key external stakeholders or stakeholder representatives,
determined the responsibilities of internal or external stakeholders,
and established a time line for involving stakeholders and addressing
their comments. If NWS were to proceed with its prototype without such
a plan, the agency would be unable to ensure that employee and customer
interests are anticipated, identified, and addressed.
Prior to suspending the prototype, NWS stated that it would ensure that
there would be no degradation of service resulting from the prototype,
and that it had plans in place to mitigate the risk of degradation. To
these ends, NWS planned to conduct multiple laboratory exercises to
understand the impact of the prototype. Also, NWS planned to conduct
the prototype on a dedicated workstation in each cluster office that
was separate from existing systems, so that it could revert to normal
operations should the need arise. In addition, NWS planned to monitor
its systems and products under the prototype, so that the agency could
revert to baseline systems if warranted. NWS officials stated that,
should the agency decide to implement the prototype on a national
basis, it has standard procedures for testing and validating systems
and software to avoid any degradation of service before they are put
into use.
We are making recommendations to the Secretary of Commerce to ensure
that if NWS decides to proceed with its prototype, it undertake a more
rigorous approach to justifying, evaluating, and involving stakeholders
in its prototype effort. We also are recommending that the agency
evaluate moving forward with technology upgrades to allow offices to
switch to backup service more quickly during high-impact weather
events.
Commerce provided written comments on a draft of this report in which
it agreed with our recommendations, and stated that our findings were
included in the Under Secretary's recent review of the concept of
operations prototype. Based on that review, the NOAA Administrator
directed NWS to cease all activities associated with the concept of
operations prototype, and to undertake a comprehensive and analytical
review to determine new and revised requirements for NWS products and
services. A reprint of the department's comments is provided in
appendix III.
Background:
The mission of NWS--an agency within Commerce's NOAA--is to provide
weather, water, and climate forecasts and warnings for the United
States, its territories, and its adjacent waters and oceans to protect
life and property and to enhance the national economy. NWS is the
official source of aviation-and marine-related weather forecasts and
warnings as well as warnings about life-threatening weather situations.
NWS Office Structure: An Overview:
The coordinated activities of weather facilities throughout the United
States allow NWS to deliver a broad spectrum of climate, weather,
water, and space weather services. These facilities include weather
forecast offices, river forecast centers, national centers, and
aviation center weather service units. The functions of these
facilities are described in the following text:
* 122 weather forecast offices are responsible for providing a wide
variety of weather, water, and climate services for their local county
warning areas, including advisories, warnings, and forecasts. See
figure 1 for a map of the locations of these offices.
* 13 river forecast centers provide river, stream, and reservoir
information to a wide variety of government and commercial users as
well as to local weather forecast offices for use in flood forecasts
and warnings.
* 9 national centers constitute the National Centers for Environmental
Prediction, which provide nationwide computer model output and manual
forecast information to all NWS field offices and to a wide variety of
government and commercial users. These centers include the
Environmental Modeling Center, Storm Prediction Center, Tropical
Prediction Center/National Hurricane Center, Climate Prediction Center,
Aviation Weather Center, and Space Environment Center, among others.
* 21 aviation center weather service units, which are colocated with
key Federal Aviation Administration (FAA) air traffic control centers
across the nation, provide meteorological support to air traffic
controllers.
Figure 1: NWS's 122 Weather Forecast Offices:
[See PDF for image]
Sources: NWS (data) and Map Resources (map).
[End of figure]
NWS Relies on Key Systems and Technologies to Fulfill Its Mission:
To fulfill its mission, NWS relies on a national infrastructure of
systems and technologies that gather and process data from the land,
sea, and air. NWS collects data from many sources, including ground-
based Automated Surface Observing Systems, the Next Generation Weather
Radars, and operational environmental satellites. These data are
integrated by advanced data processing workstations--called an Advanced
Weather Interactive Processing System (AWIPS)--and used by
meteorologists to issue local forecasts and warnings. Also, these data
are fed into sophisticated computer models running on high-speed
supercomputers, which are then used to help develop forecasts and
warnings. Figure 2 depicts the integration of the various systems and
technologies and is followed by a description of each.
Figure 2: Overview of Key Systems and Technologies Supporting NWS
Forecasts:
[See PDF for image]
Source: GAO.
[End of figure]
Automated Surface Observing System:
The Automated Surface Observing System is a system of sensors,
computers, display units, and communications equipment that automates
the ground-based observation and dissemination of weather information
nationwide. This system collects data on temperature, dew point,
visibility, wind direction and speed, pressure, cloud height and
amount, and types and amounts of precipitation. There are currently
1,001 units deployed across the United States, with NWS, FAA, and the
Department of Defense operating 312, 571, and 118 units, respectively.
Next Generation Weather Radar:
The Next Generation Weather Radar is a Doppler radar system that
detects, tracks, and determines the intensity of storms and other areas
of precipitation; determines wind velocities in and around detected
storm events; and generates data and imagery to help forecasters
distinguish hazards, such as severe thunderstorms and tornadoes. The
radar system also provides information about heavy precipitation that
leads to warnings about flash floods and heavy snow. A network of these
radars is made up of 158 operational radar systems and 8 nonoperational
systems that are used for training and testing. Of these, NWS, the
United States Air Force, and FAA sponsor 120, 26, and 12 radars,
respectively.
Operational Environmental Satellites:
Although NWS does not own or operate satellites, geostationary and
polar-orbiting environmental satellites[Footnote 1] are key sources of
data for its operations. These satellite systems continuously collect
environmental data about Earth's atmosphere, surface, and cloud cover
and the electromagnetic environment. These data are used by
meteorologists to develop weather forecasts and other services, and are
critical to the early and reliable prediction of severe storms, such as
tornadoes and hurricanes.
Advanced Weather Interactive Processing System:
AWIPS is a data processing workstation that integrates and displays
hydrometeorological data at NWS weather forecast offices, river
forecast centers, and national centers. This system integrates data
from a variety of sources (including the ground systems, radars, and
satellites previously listed) to produce rich graphical displays to aid
forecaster analysis and decision making. AWIPS includes a graphical
forecast editing tool that allows forecasters to display and manipulate
detailed graphical depictions of expected weather, and to use these
depictions to generate text and graphical forecasts. AWIPS is used to
disseminate weather information to the national centers; weather
offices; the media; and other federal, state, and local government
agencies. Figure 3 shows a standard AWIPS workstation.
Figure 3: An AWIPS Workstation:
[See PDF for image]
Source: NOAA.
[End of figure]
Numerical Weather Models:
Numerical weather models are advanced software programs that assimilate
data from satellites and ground-based observing systems and provide
short-and long-term weather pattern predictions. Meteorologists
typically use a combination of models and their own experience to
develop local forecasts and warnings. In addition, numerical weather
models are a critical source for forecasting weather up to 2 weeks in
advance and forecasting long-term climate changes.
Supercomputers:
NWS leases high-performance supercomputers to execute numerical
calculations supporting weather prediction and climate modeling. In
2002, NWS awarded a $227 million contract to lease high-performance
supercomputers to run its environmental models from 2002 through
September 2011. Included in this contract are an operational
supercomputer used to run numerical weather models, an identical backup
supercomputer located at a different site, and a research and
development supercomputer on which researchers can test new analyses
and models.
Weather Forecast Office Operations: An Overview:
NWS delivers a broad spectrum of climate, weather, water, and space
weather services through the coordinated activities of the weather
forecast offices; river forecast centers; and national centers, which
comprise the National Centers for Environmental Prediction, aviation
center weather service units, and other support offices around the
country. Each weather forecast office currently has a fixed geographic
area for which it provides local warnings and forecasts. These offices
are staffed by meteorologists who produce a suite of weather products,
including short-term and 3-to 5-day extended forecasts as well as
aviation and marine weather products, and monitor for severe weather
events. Predictive weather services (outlooks, forecasts, watches,
warnings, advisories, and other supporting information) are produced
through a process that incorporates observational data, numerical model
output and statistical guidance, and human expertise. These services
are delivered in many forms, including textual, digital, and graphical
formats. Local offices and regional and national centers coordinate as
needed in the preparation of these services.
When a severe weather event occurs, forecast office managers often ask
their staffs to work overtime so that there are enough personnel
available to do both the normal forecasting work and the watches and
warnings required by the severe event. If a weather forecast office is
unable to provide forecast and warning functions due to an interruption
in communications or power, an adjacent office will temporarily assume
those duties. For example, during Hurricane Katrina, several weather
forecast offices in Louisiana, Mississippi, and Alabama were damaged
and inoperable. These offices' responsibilities were transferred to
backup sites in Texas, Florida, Alabama, and Tennessee. During this
transfer of operations, called service backup, the backup office gains
access to the original office's system data and calls in extra
personnel to handle the workload of both offices.
According to NWS officials, due to software limitations in AWIPS,
switching to a backup service arrangement can take 60 to 90 minutes,
and, when invoked, the backup office must assume all products and
services for the defunct office. In addition, the current backup
arrangement can cause duplicate work. For example, one office could
produce a forecast for its own geographic area, and then would need to
produce a separate forecast for its backup partner.
NWS Explored Options for a New Concept of Operations and Proposed One
Option for Further Study:
Over the past few years, NWS has evaluated options for an alternate
concept of operations. According to agency officials, in 2005, NWS's
board of executive advisors (called the Corporate Board)[Footnote 2]
noted that a constrained budget, high labor costs, difficulty in
training and developing its employees, and a lack of flexibility in how
the agency was operating were making it more difficult for the agency
to continue to perform its mission. In August 2005, the board chartered
a working group to evaluate the roles, responsibilities, and functions
of weather offices nationwide and to make a proposal for a more
efficient concept of operations. The group was given a set of guiding
principles, including that the proposed concept should (1) be cost-
effective, (2) ensure that there would be no degradation of service,
(3) ensure that weather services nationwide were equitable, and (4) not
reduce the number of forecast offices nationwide.
The group gathered input from various agency stakeholders and other
partners within NOAA and considered multiple alternatives. They
dismissed all but one of the alternative concepts because they were not
consistent with the guiding principles. In its December 2005 proposal,
the working group proposed the remaining concept, in which several
offices with the same type of weather and warning responsibilities,
climate, and customers would be grouped together to share
responsibilities, particularly when there is a high-impact weather
event affecting one of the offices. NWS called this a "clustered peer
concept of operations."
Under the clustered peer concept of operations, offices would share the
workload associated with routine services, such as 7-day forecasts.
During a high-impact weather event--such as a severe storm, flood, or
wildfire--the offices would redistribute the workload to allow the
impacted office to focus solely on the event, while the other offices
in the cluster would pick up the impacted office's routine services. In
addition, offices in a cluster could occasionally allocate the work so
that certain offices would take on forecasting responsibilities while
other offices focused on other tasks, such as training, customer
outreach, and research. In this manner, peer offices could help
supplement staffing needs, and the workload across multiple offices
could be more efficiently balanced.
Some of these benefits were demonstrated in recent years by a pair of
weather forecasting offices. The Detroit and Grand Rapids offices
established a technology workaround to allow them to more efficiently
switch to backup operations between the two offices. This technology
improvement allows them to switch to backup mode instantaneously,
rather than taking 60 to 90 minutes using current AWIPS technology. To
date, the two offices have used this backup capability to share
workload during high-impact weather events; during system upgrades; and
while conducting officewide training, research, and public outreach.
NWS Developed a Prototype Plan for the Alternative Concept of
Operations:
In February 2006, NWS's Corporate Board chartered a team to develop a
prototype plan for the clustered peer concept of operations. Between
February and November 2006, the team established a preliminary concept
for what the prototype would entail, conducted a laboratory exercise to
determine AWIPS's ability to support the prototype, and--on the basis
of the findings from the laboratory exercise--developed a prototype
plan.
The team's preliminary concept was for the prototype to consist of four
clusters ranging in size from four to eight weather forecast offices.
However, in September 2006, a laboratory exercise conducted between
NOAA's Global Systems Division in Boulder and NWS's Central Region
Headquarters in Kansas City showed that AWIPS was not capable of
supporting clusters of more than four offices. The laboratory results
showed that with minor modifications, AWIPS was capable of allowing a
more efficient and timely switch of operations between two weather
forecast offices for backup purposes, and that, in some cases,
functions could be shared among four offices. Officials in charge of
the laboratory exercise estimated that larger clusters would not be
possible until the next version of AWIPS is deployed--a milestone
currently planned for 2010. The laboratory exercise also identified
several problems with system performance when weather forecast offices
work as a cluster of two or more offices. Specifically, each weather
forecast office has its own set of customized tools supporting each
office's graphical forecast editor. The laboratory exercise found that
running these tools when sharing functions among multiple offices
caused AWIPS to slow down or crash.
On the basis of the findings from the laboratory exercise, the
prototype team developed a plan to demonstrate opportunities for
improved efficiency. The goal of the prototype was to validate that a
group of weather forecast offices, operating as one unit, could be more
efficient than each office operating on its own. The plan entailed
sharing forecasting responsibilities between 2, and subsequently among
4, weather forecast offices using a single workstation in each office.
The prototype was to be a hands-on demonstration conducted at 20
weather forecast offices, clustered in 4 different areas of the
country: the Pacific Northwest, Great Plains, Great Lakes, and
Southeast (see fig. 4). These areas were selected because they present
different forecasting challenges, involve all of the regions in the
contiguous United States, and engage a variety of field offices. In
November 2006, the Corporate Board approved moving forward with the
prototype plan.
Figure 4: NWS Offices Involved in the Clustered Peer Concept of
Operations Prototype:
[See PDF for image]
Sources: NWS (data) and Map Resources (map).
[End of figure]
NWS Initiated a 2-Year Prototype and Then Suspended It:
Following the Corporate Board's decision to proceed, NWS initiated the
first of three phases of its prototype, which it planned to continue
over the next 2 years. The first phase was expected to take place
between November 2006 and June 2007 and was to include planning
activities. As of March 2007, NWS had established a program manager and
initiated efforts to plan for the prototype, including documenting how
the prototype would be executed and developing training for employees
who would participate in the prototype.
During the second phase, which NWS expected to take place between
August 2007 and February 2008, the agency planned to have 20 weather
offices share forecasting responsibilities in 2-office pairs. During
the third phase, which NWS expected to take place between April 2008
and April 2009, the agency planned to have the same 20 weather offices
share forecasting responsibilities in 4-office clusters. Each phase was
to be preceded by both a laboratory exercise to test the new expanded
functions and a board decision on whether to move ahead, based on the
results of the laboratory exercise. NWS planned to conclude prototype
operations and issue a report on the prototype by July 2009. By August
2009, NWS intended to make a decision on whether to implement the new
concept of operations on a national basis. NWS's schedule for the
prototype is provided in figure 5. A more detailed description of the
phases of the prototype is provided in appendix II.
Figure 5: Preliminary Schedule for NWS's Clustered Peer Concept of
Operations Prototype, as of November 14, 2006:
[See PDF for image]
Source: GAO analysis of NWS data.
[End of figure]
These efforts were halted when the prototype was suspended. On March
23, 2007, Commerce's Under Secretary for Oceans and Atmosphere directed
NWS's director to place all activities associated with NWS's concept of
operations on hold pending a review by the Deputy Under Secretary.
NWS's director subsequently issued a memorandum noting that the
prototype was being suspended because of concerns about the agency's
approach expressed by some Members of Congress and by NOAA leadership,
the private sector, the agency's workforce, and the agency's emergency
management partners. The director noted that the prototype would be put
on hold until a reevaluation could be undertaken.
NWS Has Not Yet Sufficiently Justified Its Prototype:
Before the prototype was suspended, NWS developed and approved a 2-year
plan for its prototype without conducting an analysis of the relative
costs and benefits of the initiative. Leading organizations have
developed best practices for justifying prototyping efforts--including
analyzing the expected costs and quantified benefits of alternatives
before beginning prototyping activities--and identifying plans,
schedules, and risks.[Footnote 3] Furthermore, federal
guidance[Footnote 4] notes that in conducting a cost-benefit
assessment, tangible and intangible benefits and costs should be
identified, assessed, and reported.
The prototype plan approved by the Corporate Board in November 2006
identifies the schedules, risks, and expected benefits of the
prototype. The prototype was estimated to cost $9.3 million, and the
Corporate Board's financial investment review committee approved
spending $1.5 million on the prototype in the first half of fiscal year
2007. As of March 2007, NWS had spent approximately $724,000 on the
prototype (see table 1). Moreover, the prototype plan identifies
qualitative benefits of the new concept of operations--including
increased efficiency and time for other activities (such as outreach
and research), the ability to switch to a backup site more quickly when
a weather forecast office is disabled due to loss of power or
communications, and an improved ability to focus on high-impact events.
Table 1: Clustered Peer Concept of Operations Prototype Costs, as of
March 2007:
Fiscal year: 2005;
Approved expenditures: $4,125;
Actual expenditures: $4,125;
Estimated expenditures[A]: --;
Total: $4,125.
Fiscal year: 2006;
Approved expenditures: 430,879;
Actual expenditures: 430,879;
Estimated expenditures[A]: --;
Total: 430,879.
Fiscal year: 2007;
Approved expenditures: 1,500,000;
Actual expenditures: 288,907;
Estimated expenditures[A]: $2,911,093;
Total: 3,200,000.
Fiscal year: 2008;
Approved expenditures: [Empty];
Actual expenditures: --;
Estimated expenditures[A]: 3,000,000;
Total: 3,000,000.
Fiscal year: 2009;
Approved expenditures: [Empty];
Actual expenditures: --;
Estimated expenditures[A]: 2,700,000;
Total: 2,700,000.
Fiscal year: Total;
Approved expenditures: $1,935,004;
Actual expenditures: $723,911;
Estimated expenditures[A]: $8,611,093;
Total: $9,335,004.
Source: GAO analysis of NWS data.
[A] NWS has not yet approved these estimates.
[End of table]
However, NWS did not conduct a cost-benefit analysis to justify its
plans for the prototype or the $1.9 million that has already been
approved for the prototype.
By not conducting a cost-benefit analysis before beginning the
prototype, NWS lacked assurance that its approach was cost-effective.
If NWS decides to proceed with the prototype without conducting a cost-
benefit analysis, the agency will have little basis to ensure that its
investment is sound or that the options it is undertaking provide the
greatest return on investment. Specifically, a cost-benefit analysis
could help ascertain which aspects of the prototype promise the most
benefits and enable NWS to adjust its plans accordingly. For example, a
cost-benefit assessment might show that more benefits could be gained
for the least cost by moving to a 2-office cluster, rather than a 4-
office cluster.
NWS Lacks a Rigorous Evaluation Plan for Assessing Its Prototype:
In its prototype plan, NWS identified goals and selected measures to
evaluate during its prototype activities, but it did not establish a
rigorous evaluation plan. Specifically, the agency had not defined a
comprehensive set of measures, how it planned to compare prototype
results with baseline performance, or how its measures supported the
goals of the prototype.
According to best practices in leading organizations, well-developed
evaluation plans increase the likelihood that evaluations will yield
sound results, thereby supporting effective policy decisions. Key
features of a well-developed evaluation plan include well-defined,
clear, and measurable objectives or goals; quantitative measures for
determining program performance as compared with baseline performance;
and a linkage between the measures and the program goals. Such plans
are essential to ensuring, among other things, that programs are
executed properly and that funding requests are developed
appropriately. In July 2006, we reported that NWS had not yet
established plans, time lines, and measures for evaluating its
prototype, and we recommended that NWS do so before beginning the
prototype.[Footnote 5]
NWS's prototype plan identifies evaluation goals and measures, and
notes that NWS plans to contract with an independent entity to perform
the evaluation. The prototype goals were to (1) validate that a group
of weather forecast offices, operating as one unit, could be more
effective and efficient than each office on its own; (2) increase focus
during high-impact events through more efficient resource allocation;
(3) gain efficiencies through the use of resource allocation during
benign weather that can be applied to training, outreach, and
professional development; and (4) use some of the efficiencies gained
through the prototype to further the NOAA mission. Quantitative and
qualitative measures that were to be used in evaluating the prototype
included the number of times that work was transferred to another
forecast office to focus on high-impact events, the number of
additional hours made available during benign weather for nonforecast
work, and customer perceptions of service quality and timeliness. The
agency stated that system performance and hardware/software
functionality would have been evaluated using staff feedback,
performance monitoring software, and usage logs, while customer
perceptions would have been evaluated using customer satisfaction
surveys.
However, NWS did not establish an evaluation plan that identifies a
full set of applicable measures, what baseline performance its
prototype would be compared with, and how its measures would support
the goals of the prototype. Specifically, while the agency listed
various items that it planned to measure in its prototype plan, it did
not include quantitative measures of product quality, timeliness, the
impact on the efficiency of the office that is receiving the increased
workload, or plans to assess the sensitivity of its results to weather-
related situations. In addition, the agency did not establish the
baseline performance against which it would compare its prototype, so
it was not clear how prototype results would be assessed. Furthermore,
the measures that NWS identified did not directly support the goals.
For example, the number of times functions are transferred to another
site does not demonstrate increased efficiency. For this measure to
support the desired goal of increased efficiency, the measure would
need to be supplemented with an assessment of how the office that
transferred its workload used the newly available time and how the
receiving site handled its increased workload.
If NWS were to proceed with its prototype without a rigorous evaluation
plan, the agency would run an increased risk that it could not
sufficiently measure the impact of planned changes on its performance,
and would subsequently make decisions affecting the nation's weather
service on the basis of incomplete and flawed data.
NWS Involved Internal Stakeholders in Planning the Prototype, but Did
Not Involve External Stakeholders or Establish a Plan for Stakeholder
Involvement:
NWS involved internal stakeholders in planning its prototype, including
having selected stakeholders review plans and provide feedback on work
products and laboratory activities. However, NWS did not involve
external stakeholders or establish a plan for stakeholder involvement
throughout the prototype. Leading organizations routinely involve
relevant stakeholders when considering operational or process
changes.[Footnote 6] In addition, NWS has agreed to negotiate with its
employees' union, the National Weather Service Employees Organization,
whenever organizational changes could affect working
conditions.[Footnote 7] Moreover, best practices call for developing a
plan for stakeholder involvement that includes identifying relevant
stakeholders, the roles and responsibilities of the relevant
stakeholders, and a schedule for stakeholder involvement throughout the
project.
In planning its prototype, NWS involved internal stakeholders, who were
identified by the NWS Director of Strategic Plans as forecasters, other
field and headquarters staff, and union representatives. For example,
when deciding which offices to group into clusters, meteorologists-in-
charge were asked for their input on cluster criteria and preferences.
Furthermore, one meteorologist from each of eight potential prototype
weather forecast offices participated in the laboratory test. Their
feedback on the new concept of operations was acquired through
questionnaires, electronic logs, and debriefings at the end of each
week. In addition, the employees' union provided input before the
decision was made to begin the prototype. Both the initial working
group and the prototype team included union representatives. Also, the
union president participated in Corporate Board meetings that included
discussions of the clustered peer prototype, and a union representative
observed the final week of the laboratory test.
NWS also intended to work with external stakeholders (including
emergency managers and other public safety officials, the media and
other users of NWS information services, and the general public) after
the prototype was complete to assess the results. Specifically, NWS
planned to identify customer perceptions of service as one basis on
which to evaluate the prototype. Furthermore, in an effort to make its
plans available to employees and customers, NWS developed a public Web
site that contains the plans and reports for the prototype.[Footnote 8]
However, NWS did not request input from external stakeholders,
including emergency managers, during the past 20 months that it was
planning and initiating the prototype. NWS officials reported that they
did not involve external stakeholders because changes in the concept of
operations during the prototype should not be noticeable to outside
stakeholders; that is, external stakeholders should not notice any
difference in service. Although this goal has merit, external
stakeholders need to be more involved in the prototype for it to gain
acceptance and succeed. Specifically, external stakeholders need to
understand what is being tested, have input into the tests, be alert
for differences under the prototype, and have formal mechanisms for
sharing their observations. In addition, NWS did not have a
comprehensive plan for involving stakeholders in its prototyping
activities. Specifically, it did not identify key external stakeholders
or stakeholder representatives, determine the responsibilities of
internal or external stakeholders, or develop a time line for involving
stakeholders and addressing their comments.
If NWS were to proceed with its prototype without a plan for involving
internal and external stakeholders, it would not be able to ensure that
employee and customer interests are anticipated, identified, and
addressed.
NWS Planned to Ensure That the Prototype Would Not Degrade Its Service:
NWS stated that it would ensure that there would be no degradation of
service during or after its prototype, and that it had plans in place
to mitigate the risk of any service degradation. Specifically, the
agency conducted and planned to continue to conduct laboratory
exercises to assess the impact of the prototype before moving into each
phase. In addition, NWS planned to run the prototype software on a
single, isolated workstation in each office, while maintaining the
existing AWIPS software on all other workstations. Established
procedures would allow offices to return to the baseline software and
operational practices should problems have developed. Furthermore, NWS
planned to monitor the performance of all activities and the quality
and timeliness of products.
Looking forward, an NWS official stated that they also would ensure
that there would be no degradation of service if the Corporate Board
decided to implement this alternative concept of operations on a
national basis after the prototype was completed. Agency officials
explained that the agency would use its standard procedures for testing
and validating systems and software to ensure that there would be no
degradation of service before moving them into operations.
Conclusions:
Although there is merit in NWS's goal of improving its operational
efficiency and allowing faster backup service during severe weather,
the agency lacked the rigorous planning it needed to justify, evaluate,
and obtain stakeholder involvement in its planned--and now suspended--
prototype of a new concept of operations. Specifically, the agency
developed a prototype plan that identified activities, gates, and
decisions, but it did not perform a cost-benefit assessment to justify
the prototype. Also, NWS identified goals and selected measures for
evaluating its prototype, but it did not establish an evaluation plan
that clearly identifies a comprehensive set of measures or how these
measures would be compared with baseline performance. Furthermore, the
agency involved internal stakeholders in planning for its prototype,
but it did not seek input from external stakeholders or establish a
plan for stakeholder involvement throughout the prototype. These
matters will be important to consider should NWS decide to proceed with
the prototype. Without undertaking a more rigorous approach, NWS would
run an increased risk that its prototype will not be a sound
investment, and that any resulting decisions could be based on flawed
data and analysis. Taking this more rigorous approach to the prototype
should not preclude NWS from considering options for improving the
ability to switch to a backup site when a weather forecast office
becomes disabled due to loss of power or communications.
Recommendations for Executive Action:
We are making four recommendations to the Secretary of Commerce. If NWS
decides to proceed with its prototype, we recommend the Secretary
direct the Assistant Administrator for the National Weather Service to:
* develop a cost-benefit analysis for the clustered peer approach
before implementing the prototype in any weather forecasting office;
* develop an evaluation plan that includes a comprehensive set of
measures that are linked to prototype goals and identifies the baseline
performance that the prototype will be compared with before
implementing the prototype in any weather forecasting office; and:
* develop a plan for internal and external stakeholder involvement,
which includes a list of relevant stakeholders, roles and
responsibilities for these stakeholders, and a schedule for when
stakeholders should be involved before implementing the prototype in
any weather forecasting office.
In addition, we recommend that the Secretary direct the Assistant
Administrator for the National Weather Service to evaluate moving
forward with technology upgrades to the AWIPS processing system to
allow weather forecast offices to switch to backup service more quickly
during high-impact weather situations; these upgrades could be handled
as an initiative that is separate and distinct from broader changes in
the agency's concept of operations.
Agency Comments and Our Evaluation:
We received written comments on a draft of this report from Commerce
(see app. III). In the department's response, the Deputy Secretary of
Commerce agreed with our recommendations to conduct a thorough cost-
benefit analysis, develop a rigorous evaluation plan with fully defined
performance measures, and involve internal and external stakeholders.
The response stated that our findings were included in the Under
Secretary's recent review of the concept of operations prototype. On
the basis of that review, the NOAA Administrator directed NWS to cease
all activities associated with the concept of operations prototype, and
to undertake a comprehensive and analytical review to determine new and
revised requirements for NWS products and services. This review is
expected to determine what, if any, changes are required. Should NWS
decide to proceed with a new concept of operations, we believe that our
recommendations to justify, evaluate, and involve stakeholders in any
such changes would still be relevant and useful.
In addition, Commerce agreed with our recommendation to evaluate moving
forward with technology upgrades to the AWIPS processing system to
allow weather forecast offices to switch to backup service more quickly
during high-impact weather situations. The department stated that NWS
has established a proposal for improving its service backup capability,
expects to approve requirements for the backup capability by September
2007, and plans to deploy this capability by September 2008.
As we agreed with your offices, unless you publicly announce the
contents of this report earlier, we plan no further distribution of it
until 30 days from the report date. At that time, we will send copies
of this report to interested congressional committees, the Secretary of
Commerce, the Director of the Office of Management and Budget, and
other interested parties. In addition, this report will be available at
no charge on our Web site at www.gao.gov.
If you have any questions about this report, please contact me at (202)
512-9286 or by e-mail at pownerd@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. GAO staff who made major contributions to
this report are listed in appendix IV.
Signed by:
David A. Powner:
Director, Information Technology Management Issues:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
Our objectives were to (1) determine the current status of and plans
for the prototype, (2) evaluate whether the justification for the
prototype is sufficient, (3) determine whether the National Weather
Service's (NWS) plans to evaluate the prototype are adequate, (4)
evaluate whether NWS is sufficiently involving stakeholders in its
prototype plans, and (5) determine how NWS plans to ensure that there
will be no degradation of service during and after the prototype.
To determine the current status of and plans for the prototype, we
reviewed agency documentation on the planned prototype's goals, major
components, and milestones, such as the Corporate Board's meeting
minutes, the prototype plan, and the preprototype laboratory report. We
also interviewed NWS officials to obtain clarifications on agency
plans.
To evaluate whether the justification for the prototype is sufficient,
we reviewed program documents such as the prototype plan and other NWS
planning documents, presentations, and communications about the goals
and costs of the prototype. We compared these planning documents with
best practices for justifying process-improvement proposals by the
Carnegie Mellon University Software Engineering Institute's Capability
Maturity ModelŽIntegration for Development[Footnote 9] and federal
guidance on conducting cost-benefit analyses.[Footnote 10] We also
interviewed agency officials and employees who were involved in the
planning of the prototype and laboratory exercise to determine the
potential costs, benefits, and risks of the prototype.
To determine whether NWS plans to evaluate the prototype are adequate,
we reviewed planning documents, such as the prototype plan and NWS
customer satisfaction scores, and compared them with best practices for
evaluating prototypes from the Capability Maturity ModelŽIntegration
for Development and GAO's guidance for developing and assessing
evaluation plans.[Footnote 11] In addition, we interviewed NWS
officials who were responsible for planning the prototype activities.
To evaluate whether NWS is sufficiently involving stakeholders in its
prototype plans, we reviewed NWS's plans, presentations, and other
communications about the prototype, including an external Web site that
presented these documents. We compared NWS's plans for involving
stakeholders with best practices from the Capability Maturity ModelŽ
Integration for Development. We spoke with members of stakeholder
groups to determine their knowledge of the prototype and their
interaction with NWS. In addition, we interviewed NWS officials and the
president of the National Weather Service Employees Organization to
determine the level of involvement of the employee union in planning
for the prototype.
To determine how NWS plans to ensure that there will be no degradation
of service during and after the prototype, we reviewed planning
documents--including the prototype plan, presentations, and
communications regarding the prototype--and interviewed NWS officials.
We performed our work at NWS headquarters in the Washington, D.C.,
metropolitan area; at NWS offices in Tampa, Florida, Detroit, Michigan,
and Kansas City, Missouri; and at a National Oceanic and Atmospheric
Administration (NOAA) office in Boulder, Colorado. We visited the Tampa
weather forecast office because the president of the employee union and
the vice president of the International Association of Emergency
Managers are located in the Tampa area. The Detroit weather forecast
office was included because of its prior experiences in working in a
paired backup with expanded domains. We visited the NWS Central Region
Headquarters in Kansas City because of its participation in the first
laboratory exercise and because it is where the prototype team chair
worked. The NOAA Global Systems Division office in Boulder was chosen
because of its participation in the first laboratory exercise. We
performed our work from October 2006 to April 2007 in accordance with
generally accepted government auditing standards.
[End of section]
Appendix II: NWS Prototype Plans, as of March 2007:
Prior to the suspension of the prototype by the Director of the
National Weather Service, the prototype team developed a plan for a
three-phased prototype of the clustered peer concept of operations. As
described in the plan, phase 1 would include planning activities, such
as training; phase 2 was meant to establish clustered peer operations
in 10 pairs of weather forecast offices across the country; and phase 3
would include offices grouped into 4-office clusters. NWS planned to
conclude prototype operations and to issue a report on the prototype by
July 2009. NWS was to decide at that time if it would proceed to
implement the clustered peer concept of operations on a national basis.
Details about each of the planned prototype phases are included in the
following text.
Phase 1, the planning phase, began in November 2006 with the Corporate
Board's approval of the prototype plan. NWS established a program
manager to oversee the prototype. In addition, cluster management
teams[Footnote 12] representing each cluster were to meet to decide on
the rules of engagement, common tools, and common forecast
methodologies. At the end of this phase, a second laboratory exercise
was planned to validate that software upgrades and hardware additions
to support the prototype were operationally ready. The second
laboratory exercise was scheduled for June 2007 and was to involve only
weather forecast offices that are currently service-backup pairs within
the original four geographic cluster areas. Upon completion of this
exercise, the Corporate Board was to decide on whether to initiate
Phase 2 of the prototype.
Phase 2 was planned to begin in August 2007. During this phase, NWS
planned to conduct training, develop guidance for how and when offices
would take over activities for other offices (called playbooks), define
the cluster methodology, and have 10 pairs of offices operate as
clustered peers. Also during this phase, NWS planned to develop the
software upgrades needed for expanded operations in the next phase.
Near the end of this phase, in February 2008, NWS planned to conduct a
third laboratory exercise to test the software necessary for moving to
clusters of 4 offices. After completion of this exercise, the Corporate
Board was to decide on whether to initiate Phase 3 of the prototype.
Phase 3 was planned to begin in April 2008 and was to determine if
clusters of 4 offices can work as a multioffice team sharing
forecasting responsibilities. NWS planned to complete a preliminary
prototype report in July 2009 and present it to the Corporate Board. At
that time, the board would have decided whether to implement the
clustered peer concept of operations across the country.
[End of section]
Appendix III: Comments from the Department of Commerce:
The Deputy Secretary Of Commerce:
Washington, D.C. 20230:
May 16, 2007:
Mr. David A. Powner:
Director, Information Technology Management Issues:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, D.C. 20548:
Dear Mr. Powner:
Thank you for the opportunity to review and comment on the Government
Accountability Office's draft report entitled Weather Forecasting:
National Weather Service's Operations Prototype Needs More Rigorous
Planning (GAO-07-650). On behalf of the Department of Commerce, I
enclose the National Oceanic and Atmospheric Administration's
programmatic comments on the draft report.
Sincerely,
Signed by:
David A. Sampson:
Enclosure:
Department of Commerce Comments on the Draft GAO Report Entitled
"Weather Forecasting: National Weather Service's Operations Prototype
Needs More Rigorous Planning" (GAO-07-650/June 2007):
General Comments:
The Department of Commerce's National Oceanic and Atmospheric
Administration (NOAA) appreciates the opportunity to review this draft
report. We are pleased GAO acknowledges the importance of continuing
upgrades to the Advanced Weather Interactive Processing System (AWIPS)
and associated services' backup capabilities. These upgrades are
critical to ensure the National Weather Service's (NWS) continuity of
services during high impact events, and to continue to help meet the
Nation's growing demand for improved weather, water, and climate
information.
NOAA agrees with the need to conduct a thorough cost benefit analysis,
involve internal and external stakeholders, and develop a rigorous
evaluation plan with fully defined performance measures to assess
impact of the concept of operation and prototype. These conclusions
were part of the findings and conclusions identified in the recent
review led by NOAA's Deputy Under Secretary. Based on the review, the
NWS has been directed to cease all activities associated with the
concept of operation and concept of operation prototype, and to
undertake a comprehensive and analytical review to determine emerging
new/revised requirements for products and services. This review will
consider internal NOAA and external user requirements, and involve the
external user community. The results of this review will determine what
if any, changes are required.
NOAA Response to GAO Recommendations:
The draft GAO report states, "We are making four recommendations to the
Secretary of Commerce. If NWS decides to proceed with its prototype, we
recommend the Secretary direct the Assistant Administrator for the
National Weather Service to:"
Recommendation 1: "Develop a cost-benefit analysis for the clustered
peer approach before implementing the prototype in any weather
forecasting office."
NOAA Response: NWS has been directed by the NOAA Administrator to cease
all activities associated with the concept of operation and concept of
operation prototype.
Recommendation 2: "Develop an evaluation plan that includes a
comprehensive set of measures that are linked to prototype goals and
identifies the baseline performance that the prototype will be compared
to before implementing the prototype in any weather forecasting
office."
NOAA Response: NWS has been directed by the NOAA Administrator to cease
all activities associated with the concept of operation and concept of
operation prototype.
Recommendation 3: "Develop a plan for internal and external stakeholder
involvement, which includes a list of relevant stakeholders, roles and
responsibilities for these stakeholders, and a schedule for when
stakeholders should be involved before implementing the prototype in
any weather forecasting office."
NOAA Response: NWS has been directed by the NOAA Administrator to cease
all activities associated with the concept of operation and concept of
operation prototype.
Recommendation 4: "In addition, we recommend that the Secretary direct
the Assistant Administrator for the National Weather Service to:
Evaluate moving forward with technology upgrades to the AWIPS
processing system to allow weather forecast offices to switch to backup
service more quickly during high-impact weather situations; these
upgrades could be handled as an initiative that is separate and
distinct from broader changes in the agency's concept of operations."
NOAA Response: NOAA concurs with this recommendation. NWS has
established a Statement of Need for a project within its internal
Operations and Services Improvement Process (OSIP) for an improved
service backup capability (OSIP 07-20, Service Backup Improvements).
Service backup requirements are scheduled to be approved by September
30, 2007. Targeted delivery of Phase 1, service backup improvements in
the AWIPS software in Operational Build 8.3 are scheduled to be
deployed by September 30, 2008.
[End of section]
Appendix IV: GAO Contact and Staff Acknowledgments:
GAO Contact:
David A. Powner, (202) 512-9286 or pownerd@gao.gov:
Staff Acknowledgments:
In addition to the contact person named above, Colleen Phillips,
Assistant Director; Kate Agatone; Neil Doherty; Amos Tevelow; and
Jessica Waselkow made key contributions to this report.
FOOTNOTES
[1] Geostationary environmental satellites orbit above Earth's surface
at the same speed as Earth rotates, so that each satellite remains over
the same location on Earth. Polar satellites constantly circle Earth in
an almost north-south orbit, providing global coverage of conditions
that affect the weather and climate.
[2] NWS's Corporate Board is chaired by the Director of the National
Weather Service and composed of senior officials responsible for
different aspects of the agency's mission, including the Chief
Information Officer and the Directors of the Office of Climate, Water,
and Weather Services; the National Centers for Environmental
Prediction; and the NWS regions. The board meets at least twice
annually to discuss NWS's budget and strategic issues. It also holds
meetings, as needed, to focus on special topics, such as an assessment
of weather services provided during Hurricane Charley in 2004.
[3] Carnegie Mellon University Software Engineering Institute,
Capability Maturity ModelŽ Integration for Development, Version 1.2
(Pittsburgh, PA: August 2006). The Carnegie Mellon University Software
Engineering Institute, recognized for its expertise in software and
system processes, has developed the Capability Maturity ModelŽ
Integration for Development to evaluate, improve, and manage system and
software development processes.
[4] Office of Management and Budget, Guidelines and Discount Rates for
Benefit-Cost Analysis of Federal Programs, OMB Circular A-94
(Washington, D.C.: Oct. 29, 1992).
[5] GAO, Weather Forecasting: National Weather Service Is Planning to
Improve Service and Gain Efficiency, but Impacts of Potential Changes
Are Not Yet Known, GAO-06-792 (Washington, D.C.: July 14, 2006).
[6] GAO, Information Technology Investment Management: A Framework for
Assessing and Improving Process Maturity, GAO-04-394G (Washington,
D.C.: March 2004); and Capability Maturity ModelŽIntegration for
Development, Version 1.2.
[7] NWS, National Collective Bargaining Agreement Between The National
Weather Service and The National Weather Service Employees Organization
(Oct. 25, 2001). This agreement calls for NWS to negotiate with the
union on the impact and implementation of changes affecting working
conditions before they can be implemented. The agreement also allows
the agency to avoid the need for negotiations if the union has
sufficient predecisional involvement.
[8] NWS, Focus on the Future (Silver Spring, MD: Apr. 5, 2007),
hyperlink, http://www.weather.gov/com/digitalera/ (viewed on Apr. 10,
2007).
[9] Carnegie Mellon University Software Engineering Institute,
Capability Maturity ModelŽ Integration for Development, Version 1.2
(Pittsburgh, PA: August 2006). Capability Maturity ModelŽ and
Capability Maturity Modeling are registered in the U.S. Patent and
Trademark Office. CMM is a service mark of Carnegie Mellon University.
[10] Office of Management and Budget, Guidelines and Discount Rates for
Benefit-Cost Analysis of Federal Programs, OMB Circular A-94
(Washington, D.C.: Oct. 29, 1992).
[11] GAO, Equal Employment Opportunity: DOD's EEO Pilot Program Under
Way, but Improvements Needed to DOD's Evaluation Plan, GAO-06-538
(Washington, D.C.: May 5, 2006); and Earned Income Tax Credit:
Implementation of Three New Tests Proceeded Smoothly, But Tests and
Evaluation Plans Were Not Fully Documented, GAO-05-92 (Washington,
D.C.: Dec. 30, 2004).
[12] The cluster management team provides operational oversight to the
cluster. The team is composed of the meteorologists-in-charge from the
cluster weather forecast offices, hydrologists-in-charge from the river
forecast centers, and a regional representative.
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site (www.gao.gov). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site.
To have GAO e-mail you a list of newly posted products every afternoon,
go to www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125
Washington, D.C. 20548:
Public Affairs:
Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548:
