Wildland Fire Management
Interagency Budget Tool Needs Further Development to Fully Meet Key Objectives Gao ID: GAO-09-68 November 24, 2008Wildland fires have become increasingly damaging and costly. To deal with fire's threats, the five federal wildland fire agencies--the Forest Service in the Department of Agriculture and four agencies in the Department of the Interior (Interior)--rely on thousands of firefighters, fire engines, and other assets. To ensure acquisition of the best mix of these assets, the agencies in 2002 began developing a new interagency budget tool known as fire program analysis (FPA). FPA underwent major changes in 2006, raising questions about its ability to meet its original objectives. GAO was asked to examine (1) FPA's development to date, including the 2006 changes, and (2) the extent to which FPA will meet its objectives. To do so, GAO reviewed agency policies and FPA documentation and interviewed agency officials.
FPA is both a computer model and a broader management system for developing the five agencies' wildland fire budget requests and allocating funds. FPA is intended to allow the agencies to analyze potential combinations of firefighting assets and potential strategies for reducing vegetation and fighting fires to determine the most cost-effective mix of assets and strategies. The agencies began developing FPA in 2002 and completed the first part of the model in October 2004. As the agencies began using FPA, however, agency officials raised concerns about its underlying science and the extent to which it met agency management and policy objectives. As a result, in 2006 the agencies conducted a review of FPA, which questioned FPA's basic modeling approach. The agencies made substantial changes to FPA after the review, some of which followed from the review's recommendations. For example, as recommended, the agencies established a new oversight body comprising senior agency leaders. The agencies also made fundamental changes to FPA's modeling approach for analyzing the firefighting assets needed to respond to fires, but these changes went beyond the review's recommendations and, despite FPA's importance and cost, the reasons for these changes were not fully documented. The agencies expected to complete the FPA model in November 2008--about a year later than initially estimated--and to begin using FPA's results in spring 2009 to develop their fiscal year 2011 budget requests, a delay of about 3 years from their initial goal of using FPA's preliminary results in 2006. FPA is expected to cost about $54 million to develop. Although it is not yet complete and GAO conducted only a limited review of its available components, FPA shows promise in achieving some of the key objectives originally established for it; nevertheless, the approach the agencies have taken hampers FPA from meeting other key objectives. Among the most important objectives, FPA will (1) provide a common framework for the five federal agencies to analyze firefighting assets and develop budget requests across agency jurisdictions, (2) analyze the most important fire management activities, and (3) recognize the presence of certain nonfederal firefighting assets that may be available to respond to fires on federal land. FPA falls short, however, with respect to other key objectives. First, FPA has limited ability to project the effects of different levels of vegetation reduction treatments and firefighting strategies over time, meaning that agency officials lack information that could help them analyze the long-term impact of changes in their approach to wildland fire management. Second, the modeling approach the agencies are taking cannot identify the most cost-effective mix and location of federal firefighting assets for a given budget but, rather, analyzes a limited number of combinations of assets and strategies to identify the most cost-effective among them. More broadly, the current FPA approach involves considerable discretion on the part of agency officials, increasing the importance of making decisions in a transparent manner so that Congress, the public, and officials throughout the agencies understand FPA's role in budget development and allocation.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-09-68, Wildland Fire Management: Interagency Budget Tool Needs Further Development to Fully Meet Key Objectives
This is the accessible text file for GAO report number GAO-09-68
entitled 'Wildland Fire Management: Interagency Budget Tool Needs
Further Development to Fully Meet Key Objectives' which was released on
November 25, 2008.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Chairman, Committee on Energy and Natural Resources,
U.S. Senate:
United States Government Accountability Office:
GAO:
November 2008:
Wildland Fire ManAgement:
Interagency Budget Tool Needs Further Development to Fully Meet Key
Objectives:
Fire Program Analysis:
GAO-09-68:
GAO Highlights:
Highlights of GAO-09-68, a report to the Chairman, Committee on Energy
and Natural Resources, U.S. Senate.
Why GAO Did This Study:
Wildland fires have become increasingly damaging and costly. To deal
with fire‘s threats, the five federal wildland fire agencies”the Forest
Service in the Department of Agriculture and four agencies in the
Department of the Interior (Interior)”rely on thousands of
firefighters, fire engines, and other assets. To ensure acquisition of
the best mix of these assets, the agencies in 2002 began developing a
new interagency budget tool known as fire program analysis (FPA). FPA
underwent major changes in 2006, raising questions about its ability to
meet its original objectives. GAO was asked to examine (1) FPA‘s
development to date, including the 2006 changes, and (2) the extent to
which FPA will meet its objectives. To do so, GAO reviewed agency
policies and FPA documentation and interviewed agency officials.
What GAO Found:
FPA is both a computer model and a broader management system for
developing the five agencies‘ wildland fire budget requests and
allocating funds. FPA is intended to allow the agencies to analyze
potential combinations of firefighting assets and potential strategies
for reducing vegetation and fighting fires to determine the most cost-
effective mix of assets and strategies. The agencies began developing
FPA in 2002 and completed the first part of the model in October 2004.
As the agencies began using FPA, however, agency officials raised
concerns about its underlying science and the extent to which it met
agency management and policy objectives. As a result, in 2006 the
agencies conducted a review of FPA, which questioned FPA‘s basic
modeling approach. The agencies made substantial changes to FPA after
the review, some of which followed from the review‘s recommendations.
For example, as recommended, the agencies established a new oversight
body comprising senior agency leaders. The agencies also made
fundamental changes to FPA‘s modeling approach for analyzing the
firefighting assets needed to respond to fires, but these changes went
beyond the review‘s recommendations and, despite FPA‘s importance and
cost, the reasons for these changes were not fully documented. The
agencies expected to complete the FPA model in November 2008”about a
year later than initially estimated”and to begin using FPA‘s results in
spring 2009 to develop their fiscal year 2011 budget requests, a delay
of about 3 years from their initial goal of using FPA‘s preliminary
results in 2006. FPA is expected to cost about $54 million to develop.
Although it is not yet complete and GAO conducted only a limited review
of its available components, FPA shows promise in achieving some of the
key objectives originally established for it; nevertheless, the
approach the agencies have taken hampers FPA from meeting other key
objectives. Among the most important objectives, FPA will (1) provide a
common framework for the five federal agencies to analyze firefighting
assets and develop budget requests across agency jurisdictions, (2)
analyze the most important fire management activities, and (3)
recognize the presence of certain nonfederal firefighting assets that
may be available to respond to fires on federal land. FPA falls short,
however, with respect to other key objectives. First, FPA has limited
ability to project the effects of different levels of vegetation
reduction treatments and firefighting strategies over time, meaning
that agency officials lack information that could help them analyze the
long-term impact of changes in their approach to wildland fire
management. Second, the modeling approach the agencies are taking
cannot identify the most cost-effective mix and location of federal
firefighting assets for a given budget but, rather, analyzes a limited
number of combinations of assets and strategies to identify the most
cost-effective among them. More broadly, the current FPA approach
involves considerable discretion on the part of agency officials,
increasing the importance of making decisions in a transparent manner
so that Congress, the public, and officials throughout the agencies
understand FPA‘s role in budget development and allocation.
What GAO Recommends:
GAO is recommending, among other things, that the agencies develop a
strategic plan for the continued development of FPA and provide
Congress with annual updates on (1) their progress in completing the
steps outlined in that plan and (2) how they used FPA in developing
their budgets. Interior disagreed with the need to develop a strategic
plan. In response to Forest Service and Interior comments on GAO
findings on FPA‘s cost-effectiveness approach, GAO‘s recommendation to
develop a strategic plan was revised to provide more flexibility. The
agencies generally concurred with the other recommendations.
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.gao.gov/products/GAO-09-68]. For more
information, contact Robin Nazzaro at (202) 512-3841 or
nazzaror@gao.gov. Highlights
[End of section]
Contents:
Letter:
Results in Brief:
Background:
Concerns about FPA's Early Performance Led to Significant Changes, Not
All of Which Were Transparent:
FPA Shows Promise in Achieving Some Objectives but Falls Short of
Others, Although the Agencies Are Considering Changes That May Improve
Its Performance:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Comments from the Department of Agriculture, Forest
Service:
GAO Comments:
Appendix III: Comments from the Department of the Interior:
GAO Comments:
Appendix IV: GAO Contact and Staff Acknowledgments:
Table:
Table 1: Agencies' Cost Estimates for Developing FPA:
Abbreviations:
Agriculture: Department of Agriculture:
FPA: fire program analysis:
Interior: Department of the Interior:
OMB: Office of Management and Budget:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
November 24, 2008:
The Honorable Jeff Bingaman:
Chairman:
Committee on Energy and Natural Resources:
United States Senate:
Dear Mr. Chairman:
Wildland fires increasingly threaten communities and natural resources,
and the cost of responding to those fires has risen dramatically. To
deal with fire's threats, the five federal agencies responsible for
managing wildland fires--the Forest Service in the Department of
Agriculture (Agriculture) and the Bureau of Indian Affairs, Bureau of
Land Management, Fish and Wildlife Service, and National Park Service
in the Department of the Interior (Interior)--call upon thousands of
firefighters and station fire engines, aircraft, and other equipment on
or near federal land across the country. The agencies also conduct
treatments to reduce vegetation, in an effort to lessen the potential
for severe wildland fires, decrease the damage caused by fires, and
restore and maintain healthy ecosystems. Despite these efforts, the
average number of acres burned annually in recent years has grown by
about 70 percent, and federal appropriations to prepare for and respond
to wildland fires have nearly tripled since the mid-1990s, to more than
$3 billion annually. Several factors have contributed to the increased
risk and cost, including (1) uncharacteristic accumulations of
vegetation, in part due to past land management activities and fire
suppression policies; (2) increasing human development in or near
wildlands, an area commonly known as the wildland-urban interface; and
(3) severe drought and other stresses, in part due to climate change.
Combined, these factors have contributed to wildland fires' burning
more intensely and spreading more quickly at the same time that
development has continued in fire-prone areas. Long-standing concerns
about the mounting risk from and cost of wildland fires, along with
growing recognition of the long-term fiscal challenges facing the
nation, have led Congress, the agencies, and others to focus on
ensuring that federal wildland fire activities are appropriate and
carried out in a cost-effective and efficient manner.
A key initial step in this effort was the development of the 1995
federal wildland fire management policy.[Footnote 1] The policy
recognized that new approaches to managing wildland fire were needed if
the agencies were to respond effectively to changing conditions. The
policy also found that differences in budgeting processes among the
five agencies hindered their response to wildland fires. Subsequently,
congressional committees directed the agencies to develop a common
budget process. In 2001, the agencies commissioned a report that
established a vision for an interagency budget process, a report the
agencies adopted as the basis for a new budget-planning system known as
fire program analysis, or FPA.[Footnote 2]
As envisioned in the 2001 agency report, as well as in congressional
committee and Office of Management and Budget (OMB) reports, FPA was
intended to help the agencies develop their wildland fire budget
requests and allocate funds. FPA's objectives include:
* providing a common budget framework to analyze firefighting assets
without regard for agency jurisdictions;
* examining the full scope of fire management activities, including
preparing for fires by acquiring and positioning firefighting assets
for the fire season, mobilizing assets to suppress fires, and reducing
potentially hazardous fuels;
* considering the availability of nonfederal firefighting assets, such
as state or county firefighters, that typically help respond to fires
on federal lands;
* considering the communities and resources to be protected and agency
land management objectives;
* modeling the effects over time of differing strategies for responding
to wildland fires and treating lands to reduce hazardous fuels; and:
* using this information to identify the most cost-effective mix and
location of federal wildland fire management assets.
* In addition, FPA was expected to be externally peer reviewed, which
could improve Congress's and the agencies' understanding of its
strengths and weaknesses.
To realize this vision, the agencies in 2002 began to develop FPA,
designing it as a computer model that analyzed numerous potential
combinations and locations of firefighting assets and, for any given
budget level, identified the optimal mix of these assets--that is, the
mix and locations of firefighting assets that would best protect
resources at risk. Data on potential combinations and locations of
assets were to be entered by fire officials at agency field units, and
the model's analysis of these combinations would then be evaluated by
agency budget officials at the national level. The agencies estimated
that FPA would cost more than $40 million to develop and would take
about 5 years to complete. In 2006, after 4 years of work, the agencies
conducted an internal review of FPA, in part because of concerns about
how well the computer model reflected the realities of the agencies'
fire management activities.[Footnote 3] Subsequently, the agencies made
substantial changes in how FPA analyzes needed firefighting assets and
determines where best to locate them. These changes raised questions
about the extent to which FPA would meet its original objectives. In
this context, you asked us to review FPA. This report examines (1) how
the agencies have developed FPA to date, including the process followed
as part of the internal review, and FPA's current status; and (2) the
extent to which FPA will meet its original objectives.
To address our objectives, we reviewed agency documents on FPA
development, including the interagency report and project charter that
provide FPA's foundation, the reports resulting from the internal
review, and numerous technical papers and other documentation
describing particular aspects of FPA. To further our understanding of
FPA's development, including changes made to FPA after the internal
review, we interviewed Forest Service and Interior officials in
Washington, D.C; FPA project staff in Boise, Idaho; and agency
officials in the field who were familiar with FPA. We also interviewed
agency and other scientists who have helped develop FPA. At the time of
our review, however, substantial portions of the model remained
incomplete, and the agencies had not documented the model sufficiently
to allow a comprehensive evaluation. We therefore limited our review to
a broad examination of FPA's various components and how they interact,
as well as a comparison of FPA's current approach and capabilities with
its original objectives. We did not compare the capabilities of the
current approach to those of the approach taken before the internal
review. Appendix I describes our scope and methodology in more detail.
We conducted this performance audit from September 2007 through
November 2008 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objectives.
We believe that the evidence obtained provides a reasonable basis for
our findings and conclusions based on our audit objectives.
Results in Brief:
The Forest Service's and Interior agencies' initial development and
implementation of FPA gave rise to concerns about its performance,
leading to an internal review and subsequent changes to the model.
These changes, however, went beyond the review's recommendations and
were not always clearly explained or fully documented. The agencies
began developing FPA in 2002 and completed the first part of the model
in October 2004. But as field units began to use the first part, senior
agency officials and some field staff raised fundamental concerns--
including concerns about the underlying science and the extent to which
FPA met agency management and policy objectives. As a result, in 2006
the agencies conducted an internal review of FPA, which questioned its
modeling approach and concluded, among other things, that agency
leadership needed to become more involved if FPA were to succeed. The
agencies made substantial changes to FPA after the review, some of
which followed from the review's recommendations. For example, the
agencies established a new oversight body comprising senior agency
leaders and an interagency science team. The agencies, with the
approval of the oversight body, also made fundamental changes to FPA's
modeling approach for analyzing the firefighting assets needed to
respond to fires, but these changes went beyond the review's
recommendations. The review, for example, did not conclude that a
different modeling approach was needed, instead recommending that the
agencies continue testing the initial model and refine it as necessary.
Rather than follow this recommendation, however, the agencies adopted
an entirely new modeling approach. Yet despite FPA's importance and
cost, the reasons for these changes were not fully documented, and a
formal, documented comparison of the original and revised approaches
was never conducted. The agencies expected to complete the FPA model in
November 2008--about a year later than initially estimated--and to
begin using FPA's results in spring 2009 to develop their fiscal year
2011 budget requests, a delay of about 3 years from their initial goal
of using FPA's preliminary results in 2006. Ultimately, FPA is expected
to cost about $54 million to develop.
Although it is not yet complete and we conducted only a limited review
of its available components, FPA shows promise in achieving some of the
key objectives that congressional committees, OMB, and the agencies
themselves established for it. Nevertheless, the approach the agencies
have taken hampers FPA from meeting other key objectives. Once FPA is
more fully developed and documented, a detailed, external peer review
may reveal more about the extent to which it will help the agencies
develop their wildland fire budget requests and allocate funds. Among
the most important objectives it is likely to achieve, FPA is to
provide a common framework for the five federal agencies to develop
their wildland fire budget requests and analyze needed firefighting
assets across agency jurisdictions--a significant step forward--and is
to analyze the three most important fire management activities
(preparedness, fire suppression, and fuel reduction). The agencies also
have developed FPA to be capable of recognizing the presence of
nonfederal firefighting assets that may be available to respond to
fires on federal land--another key objective--although the extent to
which these assets is to be included in the analysis is not yet clear.
And finally, FPA is also to consider specific land management
objectives and resources at risk, as suggested by the 1995 federal
wildland fire management policy, rather than simply assume that all
fires should be suppressed as quickly as possible (although if
implemented as currently developed, FPA will likely not allow the
agencies to consistently identify the locations that are most important
to protect from a national perspective). FPA falls short, however, with
respect to other key objectives in two critical areas. First, FPA's
ability to project the effects of different levels of fuel reduction
treatments and firefighting strategies over time appears limited.
Agency officials are therefore likely to lack information that would
help them analyze the extent to which increasing or decreasing funding
for fuel reduction treatments and responding more or less aggressively
to fires in the short term could affect the expected cost of responding
to wildland fires over the long term. Second, regardless of the extent
to which other key objectives are met, the modeling approach the
agencies have taken is unlikely to identify the most cost-effective mix
and location of federal firefighting assets for a given budget but only
whether a particular mix of assets is more or less cost-effective than
another. Since the different mixes of assets analyzed are limited to
the number of alternatives developed by agency units in the field,
these alternatives, even taken together, are unlikely to include the
single most cost-effective mix of assets nationwide. In addition, other
aspects of FPA may complicate its further development and
implementation, including the lack of an external peer review of the
model to date. Agency officials recognize many of these shortcomings
and have said that they are considering taking actions--such as further
adjusting the model (to better identify the most highly valued
resources to protect, for example) and submitting the model for peer
review--that have the potential to move FPA closer to meeting its key
objectives. Regardless of the specific objectives FPA achieves, the
modeling approach the agencies selected for FPA involves considerable
discretion on the part of agency decision makers, increasing the
importance of making decisions in a manner transparent enough that
Congress, the public, and officials throughout the agencies understand
how the decisions were made and FPA's role in them.
To improve the agencies' ability to use FPA in developing their
wildland fire management budget requests and allocating funds in a cost-
effective manner and to promote transparency in decision making-- and
recognizing that FPA is still under development and that completing it
will be an iterative process requiring the agencies' continued effort
to improve--we are recommending that the Secretaries of Agriculture and
the Interior (1) direct the agencies to develop a strategic plan for
the continued development of FPA, (2) report annually to Congress on
their progress in completing the steps outlined in this plan and on
FPA's ability to meet each of its key original objectives, (3) report
to Congress each year on how the agencies used FPA to develop their
budget requests and allocate funds, and (4) submit the model for
external peer review.
In written comments on a draft of this report, the Forest Service and
Interior disagreed with our finding that FPA is unlikely to allow the
agencies to identify the most cost-effective mix of firefighting
assets, stating they believed that FPA will allow them to meet the goal
of cost-effectiveness. They also commented that the revised approach
they are taking in developing FPA is more realistic and appropriate
than their original approach. We continue to believe, however, that,
regardless of the comparative strengths and weaknesses of the original
and revised approaches, FPA as it is being developed is unlikely to
allow the agencies to identify the most cost-effective location and mix
of assets and strategies--one of the agencies' original objectives for
FPA. To account for the agencies' views that the revised approach is
more realistic, we are modifying our recommendation that the agencies
develop a strategic plan for the continued development of FPA, adding
that the agencies should clearly state whether they believe any of
FPA's key original objectives are no longer appropriate. The Forest
Service commented that it fundamentally agreed with our recommendations
but believes there are better alternative approaches to carrying some
of them out. The agency described the steps it intended to take in
addressing two of them, but we do not believe that the steps outlined
in the letter are specific and transparent enough to meet the intent of
our recommendations. Interior concurred with three of our
recommendations but disagreed with our recommendation that the agencies
develop a strategic plan for the continued development of FPA, stating
that developing such a plan would delay deployment and increase the
cost of FPA. We do not agree that creating a strategic plan would
necessarily delay the agencies' implementation of FPA; further, because
our review raised questions about FPA's ability to meet certain key
objectives, we continue to believe it is important for the agencies to
create a strategic plan that directly and transparently evaluates FPA's
ability to meet its original objectives and identifies ways to improve
FPA to better meet those objectives. Comments from the Forest Service
and Interior, along with our responses to those comments, are reprinted
in appendixes II and III, respectively.
Background:
The agencies' wildland fire management program has three major
components: preparedness, suppression, and fuel reduction.[Footnote 4]
To prepare for a wildland fire season, the agencies acquire
firefighting assets--including firefighters, engines, aircraft, and
other equipment--and station them either at individual land management
units (such as national forests or national parks) or at centralized
dispatch locations. The primary purpose of these assets is to respond
to fires before they become large--a response referred to as initial
attack--thus forestalling threats to communities and natural and
cultural resources. The speed with which the agencies are able to
respond to a fire can be critical to their ability to suppress it while
it is small; increasing the number of firefighting assets available to
respond, and the number of locations they can respond from, can
therefore improve the agencies' initial attack success, although the
marginal utility of adding more firefighting assets decreases as the
number of assets goes up. The assets the agencies use for initial
attack are funded primarily from the agencies' preparedness budget
accounts.
In the relatively rare instances in which fires escape initial attack
and grow large, the agencies respond using an interagency system, in
which additional firefighting assets from federal, state, and local
agencies, as well as private contractors, are mobilized, regardless of
which agency or agencies have jurisdiction over the burning
lands.[Footnote 5] Federal agencies typically fund the costs of these
activities from their suppression budget accounts. To reduce the
potential for severe wildland fires, lessen the damage caused by fires,
limit the spread of flammable invasive species, and restore and
maintain healthy ecosystems, the agencies also reduce potentially
hazardous vegetation that can fuel fires. They remove or modify fuels
using prescribed fire, mechanical thinning, herbicides, certain grazing
methods, or combinations of these or other approaches.
The federal government's cost of preparing for and responding to
wildland fires has increased substantially over the past decade--an
increase that has led federal agencies to fundamentally reexamine their
approach to wildland fire management. For decades, federal agencies
aggressively suppressed wildland fires and generally succeeded in
reducing the number of acres burned. In some parts of the country,
however, rather than eliminating severe wildland fires, decades of
suppression contributed to the disruption of ecological cycles and
began to change the structure and composition of forests and
rangelands, thereby making lands more susceptible to fire.
Increasingly, the agencies have recognized the role that fire plays in
many ecosystems and the role that it could play in the agencies'
management of forests and watersheds. As a result, the agencies have
increased their efforts to reduce fuels and their emphasis on using
less aggressive firefighting strategies, which typically cost less and
can reduce fuels across a broader area than if fires are aggressively
suppressed. Such strategies are to be used only in appropriate
situations, such as in responding to fires that are not expected to
threaten communities or damage important natural or cultural resources.
This approach to managing wildland fires requires close integration of
planning and budgeting systems so that the agencies are able to
holistically analyze the full wildland fire management program. The
agencies historically have used different planning and budgeting
systems to help develop their budget requests and allocate the funds
Congress appropriates. The agencies have identified shortcomings with
this approach and have recognized that the existing systems were not
capable of analyzing the trade-offs among initial attack, the full
range of suppression strategies, and fuel reduction. Aggressively
suppressing a fire, for example, may cost less in the short term but
contribute to continued accumulation of vegetation, which can increase
both the risk from and cost of responding to fires in the future;
conversely, increasing investment in reducing fuels may cost more in
the short term but can provide future benefits. The agencies, following
congressional committee direction, committed to developing a new
system, which came to be known as FPA. FPA is a strategic tool that
agency budget officials expect to use to develop their wildland fire
budget requests and allocate their fire management funds to the field,
and that agency fire officials expect to use to model the effect that
differing mixes and locations of firefighting assets, and differing
levels of investment in reducing fuels, will have on their ability to
protect communities and resources. Because it is a strategic tool
rather than a tactical one, agency fire managers would not use FPA to
help the agencies respond to actual fires.
In developing and using FPA, the agencies must consider the process and
time frames of the annual federal budget cycle, which begins about 2
years before the fiscal year for which funds are being requested.
Agencies develop their budget requests in late spring and summer and
submit them to OMB in September. OMB prepares budget materials to
submit to the President in January. The President approves a budget
proposal and sends it to Congress by the first Monday in February. To
develop their fiscal year 2011 budgets, for example, the agencies, in
conjunction with their respective departments, expect to begin
developing their budget requests in spring 2009 and to submit them to
OMB in September 2009; subsequently, the President would submit his
budget request to Congress in February 2010 for Congress's
consideration.
Concerns about FPA's Early Performance Led to Significant Changes, Not
All of Which Were Transparent:
Concerns about FPA's early performance and about the policy and
scientific approaches the agencies used in FPA's early development led
the agencies to conduct an internal review of FPA in 2006.
Subsequently, the agencies made several significant changes to FPA, but
these changes went beyond those recommended by the review, and the
reasons for several of the changes were not fully documented. The
agencies do not expect to use preliminary FPA results to develop their
budget requests until 2009--3 years later than they had initially
planned--but the cost of completing FPA appears in line with previous
estimates.
Concerns During FPA's Initial Implementation Led to an Internal Review:
The staff who began to develop FPA in 2002, following congressional
committee direction, initially focused on developing the portion of the
model that analyzed the agencies' ability to successfully contain
wildland fires during initial attack. The staff selected an approach
that relied primarily on a modeling technique known as optimization.
Using this approach, FPA was to analyze, for any given budget level,
all possible combinations and locations of the firefighting assets
typically available to agency field units and identify the combination
of these assets that resulted in optimal protection of communities and
resources. To provide data on different potential firefighting assets
and locations, the agencies divided the country into 139 interagency
"fire planning units," each of which encompassed land managed by one or
more of the federal agencies responsible for wildland fire.[Footnote 6]
Fire management officials in each of these planning units then
identified the relative importance of protecting each acre within that
planning unit by assigning a weighting factor indicating each acre's
importance relative to other acres. The most important acres to
protect, such as those in the wildland-urban interface, were assigned a
weight of 1.0, while less important acres were assigned proportionately
lower weights. After analyzing historic fire occurrence and weather
patterns associated with each planning unit to determine where and when
fires were likely to start, and considering the relative importance of
acres to be protected, FPA was to analyze, for any given budget level,
all possible mixes and locations of firefighting assets typically
available to those units in order to determine which mix and locations
would afford the best level of protection.
Development of this "preparedness module" was completed in October
2004, and over the next 16 months, officials in the field began using
it to analyze their preparedness assets and budgets. By February 2006,
nearly all the fire planning units had submitted FPA results for their
units to the agencies' Washington offices, which in turn analyzed the
FPA results in an effort to identify the optimal mix and location of
firefighting assets across the country. During this time, however,
senior agency officials, as well as some field officials, began to
raise fundamental concerns about FPA's modeling approach. Weighting the
importance of individual acres within a fire planning unit, for
example, was a central component of FPA's early approach, and some
officials believed that even where resources to be protected were
similar across planning units, officials in those units assigned
substantially different weights to the resources, thereby undermining
the reliability of the results. Other officials were concerned that the
early FPA approach placed insufficient emphasis on containing fires
during initial attack, although the officials who developed this
approach noted that it reflected the interagency policy of responding
to fires on the basis of specific land and fire management objectives,
rather than simply assuming that all fires should be suppressed as
quickly as possible. Still other officials were concerned that the
initial approach could result in unrealistic shifts in the mix and
location of assets; a small change in budget, for example, could have
led FPA to suggest moving a large quantity of assets from one planning
unit to another or to dramatically change the relative proportion of
firefighters, engines, and aircraft within a planning unit.
Despite these concerns, senior agency officials told us, the early
development of FPA represented an important first effort, given the
difficulty of the project; in hindsight, they also recognized that
greater involvement by policy and budget officials and agency
scientists might well have averted some of the concerns and helped FPA
develop more quickly. In any case, despite having told congressional
and OMB staff that they intended to use the results of this initial
analysis to help develop their 2008 budget requests, agency officials
decided that the concerns about FPA were too great to justify doing so,
and they instead initiated a two-part review of FPA to evaluate the
issues that had surfaced.
The agencies conducted this two-part review of FPA in late 2005 and
early 2006. The reviews--performed by agency land managers, fire and
budget officials, and scientists, as well as a representative from a
state forestry agency--consisted of (1) an evaluation of the extent to
which FPA helped the agencies achieve their management and policy
objectives and (2) an evaluation of particular aspects of the
underlying science and modeling approach. The reviews reaffirmed FPA's
original objectives as articulated in the 2001 report, but they
identified several challenges to meeting these objectives and made
several recommendations intended to strengthen FPA's ability to do so.
The management review, for example, recommended that the agencies more
fully involve senior officials and scientists, submit FPA for external
peer review,[Footnote 7] and complete their analysis of the initial FPA
results. The science review likewise recommended that FPA be peer-
reviewed and, in addition, that the agencies further test and improve
the model and the data it uses. The agencies conducted the reviews
quickly, however, and did not intend them to be a comprehensive
evaluation of FPA; the science review, in particular, examined only
certain aspects of FPA.
The Agencies Made Significant Changes to FPA, Not All of Which Were
Fully Documented:
After the reviews, the agencies made several changes to the process
used for developing FPA, changes that generally followed from the
reviews' recommendations. In April 2006, the agencies established a new
oversight body comprising senior officials from the Forest Service and
Interior. This group was formed to make strategic decisions about FPA's
scope, determine how FPA would be used to help the agencies make
funding allocation decisions, and address any policy issues that FPA's
development raised. The group also was to keep the Wildland Fire
Leadership Council[Footnote 8] informed of FPA's status, including
issues that the council needed to resolve. The agencies also
established an interagency science team, made up of scientists from
both the Forest Service and Interior, as well as university scientists
outside the agencies. This science team was to assist FPA's developers
by reviewing and evaluating FPA's modeling approach and identifying
data sources and analytical techniques that could further FPA's
development.
The agencies also changed FPA's modeling approach considerably. Rather
than continue to use the initial optimization-based approach
(evaluating, for a given budget level, all possible combinations and
locations of firefighting assets typically available to local units and
identifying the asset combination that provided optimal protection of
communities and resources), the agencies switched to a simulation
modeling approach that evaluates a much smaller number of potential
asset combinations along with different options for fuel reduction
treatments and ranks them according to certain performance criteria--
which also differ from those used previously. The new approach is no
longer to simply assess the extent to which each asset combination
protects the areas field officials have identified as most important.
Instead, it is to evaluate each combination's predicted performance
against five separate performance measures the agencies have
established:
* total projected cost of suppressing fires;
* total number of acres burned in the wildland-urban interface;
* total number of acres meeting fire and fuels management objectives,
such as reducing the likelihood of intense fires;
* total number of acres burned containing resources the agencies define
as being highly valued, such as endangered species habitat or municipal
watersheds; and:
* percentage of fires contained while small (i.e., the initial attack
success rate).[Footnote 9]
The revised FPA approach encompasses both a computer model and a
management system to help the agencies develop their budget requests
and allocate funds. Agency officials in each of the 139 planning units
are to develop, for each of a given number of budget levels, an option
specifying the mix and location of firefighting assets they would
choose to acquire and an option specifying the number of acres they
would treat to reduce fuels. For example, officials might develop one
mix and location of firefighting assets and the acreage that would be
treated if their fire planning unit's budget remained the same as the
previous year, another option corresponding to a budget decrease from
the previous year, and a third option corresponding to a budget
increase from the previous year. The number of options the planning
units are to develop and the budget levels to which these options
correspond will depend on annual field guidance prepared by the
agencies' headquarters offices. Senior agency officials told us that
during FPA's initial implementation they were considering directing the
units to develop three preparedness options and three fuel treatment
options. These options were to correspond to each unit's 2007 budget
level and plus and minus 10 percent of these 2007 levels. As of
November 2008, however, the agencies had not finalized this step.
Once the planning units have developed their options and entered
information about firefighting assets and fuel treatments into the FPA
system, the computer model is to then analyze historical data on local
fire occurrence; local vegetation, geography, and weather; and the
predicted effect on fire behavior of reducing fuels. From this
analysis, FPA is to model the likelihood that wildland fire will damage
communities and resources within the fire planning unit, considering
the different mixes of assets and fuel treatments reflected in the
proposed options. To provide comparable information across planning
units, FPA is to evaluate each unit's options against the five
performance measures.
The FPA model is to then calculate a performance score for each of the
"alternatives" developed by each planning unit. (An alternative
consists of one preparedness option paired with one fuel treatment
option. If planning units were directed to prepare three preparedness
and three fuel treatment options, for example, nine alternatives would
be possible.) FPA is to then "roll up" the performance scores for each
alternative in all 139 planning units, so that senior agency officials
can evaluate the effects on the agencies' performance measures
nationwide of different combinations of alternatives. The senior agency
officials would then use FPA results in conjunction with other budget
information and processes to develop their budget requests.
The extent to which any particular alternative, or set of alternatives,
is considered cost-effective relies on the weights assigned to each of
the five measures. The agencies could weight these measures in several
ways to reflect their relative importance. If one measure were
overwhelmingly more important than the others--if the agencies wanted
to minimize suppression costs regardless of any other outcome, for
example--the agencies could select a mix of firefighting assets and
fuel reduction options predicted to maximize their ability to achieve
that measure and consider the other measures only to help them choose
between different mixes with similar outcomes for the most important
measure. The agencies could also group two or more measures as more
important than the others, or they could identify desired target levels
for each measure and select the mix of firefighting assets projected to
come closest to these targets. Senior officials will be able to use FPA
to explore the modeled effects of weighting the measures differently--
in effect, to evaluate the trade-offs associated with weighting any
particular measure more heavily than the others--as well as to identify
alternatives with high performance scores regardless of the weights
ultimately selected.
The agencies will also need to determine whether the relative
importance of the five measures is the same across different geographic
regions of the country. Some officials and scientists involved with
developing FPA have questioned whether applying a single weighting
system across the country would accurately reflect national priorities
or whether it is appropriate to emphasize different measures in
different locations. For example, protecting the wildland-urban
interface might be the most important consideration in some parts of
the country, but reducing the likelihood of intense fires or protecting
endangered species habitat might be more important elsewhere. FPA
officials said that the model could perform this type of analysis, and
agency budget officials said they would consider different approaches
to weighting the measures once FPA was completed and the field units
had submitted their different combinations of firefighting assets for
analysis.
The leaders of FPA's science team told us that this new approach
addressed specific concerns they had with the old approach. First, they
said that using multiple measures to evaluate different mixes and
locations of firefighting assets--rather than a single measure as in
the old approach--better reflected the complexity of wildland fire
management. Second, they said that the new approach is to analyze many
more potential fire scenarios, thus evaluating the asset alternatives'
predicted performance across a broader range of conditions than in the
old approach.[Footnote 10] Third, they said that because the old
approach relied on weighting the relative importance of acres, they
were concerned that different units would weight similar resources
differently, thus preventing meaningful comparisons among units, or
that some units might intentionally inflate the weights in an effort to
gain advantage. Fourth, they said that because the new approach is to
rely on alternatives developed by officials in the field, it can
identify possible mixes and locations of assets that are likely to be
more easily implemented than those identified through the old approach,
which considered all possible combinations of assets typically
available to local units and could suggest changes that might be
unrealistic. Finally, they said that because the new approach allows
field officials to identify the firefighting assets they would
typically dispatch to fires burning in specific areas under certain
conditions, it more closely follows how the agencies actually respond
to fires.
The changes to FPA's modeling approach, however, were not among the
recommendations stemming from the science review, which recommended
that the agencies further test the initial model and improve it. But
such testing and improvement of the initial model did not take place.
The leaders of the science team told us that refining the initial model
would not be useful, because the team had determined that the model was
fundamentally flawed and a new approach was needed. Instead, the
science team, in summer and fall 2006, developed five options for
continuing to develop FPA and presented these options to the Wildland
Fire Leadership Council, which selected one option in December 2006.
This process was generally consistent with the management review, which
recommended that an interagency science team examine the modeling
approach FPA initially used.
Still, the agencies' rationale for making the changes to FPA's modeling
approach was not fully documented, even though FPA is a major project
whose outcome is expected to influence the allocation of billions of
dollars. Although the science team's leaders told us they believed that
the changes improved FPA, they provided no documents describing either
the reasons for the changes or the process used to identify FPA's new
approach. For example, a formal, documented comparison of the old and
new approaches was never done; the science team's leaders told us they
considered the relative strengths and weaknesses of the old approach
and other possible approaches but did not document this consideration.
In any event, each of the five development options the science team
presented to the Wildland Fire Leadership Council included the same two
fundamental changes in modeling approach. Without a formal, documented
comparison of the old and new approaches, and without the opportunity
to consider options that used other modeling approaches, the council
lacked information that might have informed its choice.
FPA's Completion Has Been Delayed, but Costs Appear in Line with
Previous Estimates:
In addition, the changes apparently prevented the agencies from meeting
their commitment to use preliminary FPA results beginning in 2006.
Although FPA was not expected to be complete until late 2007, agency
officials believed they would be able to make some use of its
preliminary results in 2006. Accordingly, officials told congressional
committee staff and OMB in early 2006 that the agencies would begin
using FPA results that year to allocate their fiscal year 2007 funds
and to develop their fiscal year 2008 budget requests.[Footnote 11]
Agency officials told us, however, that they subsequently decided not
to use FPA's preliminary results because they did not believe it was
prudent in light of the concerns that arose during the internal review.
While it seems appropriate to delay using the model for budget
decisions until concerns about its utility have been resolved, the
agencies' position has been less than transparent; in August 2006--well
after they realized that FPA would be undergoing substantial changes--
they repeated their commitment to begin using FPA results in September
of that year.
The agencies now expect that the FPA model will be completed in
November 2008--about a year later than initially estimated--and that
they will begin using FPA's results in 2009 to develop their 2011
budget requests, a delay of about 3 years from their initial goal of
using preliminary results in 2006. When they began developing FPA in
2002, the agencies reported that FPA would be completed by the end of
2007. After the internal review, the agencies reported that a fully
functional FPA system would be developed by June 30, 2008, and used in
spring 2009 to inform the agencies' fiscal year 2011 budget requests.
In spring 2008, the agencies repeated their commitment to this time
frame. Agency officials attribute the delay in completing FPA to the
project's complexity. When our review ended, agency officials said they
expected fire planning units to begin using FPA in late 2008; about
half the field units are expected to complete their alternatives by
February 2009, with the remaining units completing their alternatives
by June 2009. Meeting this time frame, however, will require the
agencies to complete both the model and the guidance directing the
field on how to develop the options the FPA model will analyze--both of
which have experienced recent delays. Nonetheless, the agencies'
Washington offices remained committed to using FPA results beginning in
2009. Agency field officials, however, have worried that the delay in
completing FPA places an undue burden on the field by shortening the
time available for planning units to develop their alternatives. Field
officials also observed that senior agency officials have not clearly
articulated how the results from FPA's first year would be used,
although senior agency officials have stated that 2009 is to be a
"learning year" and that they do not expect FPA to influence
substantial changes to funding allocations in the first year.
The expected cost for completing FPA has been little affected by the
substantial changes it has undergone since 2006. FPA's project
development costs are expected to total about $43.9 million, according
to an April 2008 estimate by the senior project manager responsible for
FPA's budget.[Footnote 12] This cost is generally in line with the
agencies' previous estimates, particularly those developed after the
agencies began to determine FPA's full scope (see table 1). Agency
salaries and benefits, which were not included in yearly estimates of
project development costs, represent an estimated $9.7 million in
additional costs--for a total of about $53.6 million. According to the
senior project manager, the agencies did not begin to develop FPA's
second phase until 2005 and were still determining the scope of that
phase when they submitted projected cost estimates in 2003 and 2004.
The increase from 2003 to 2005 in the estimated cost for the second
phase therefore reflects the agencies' better understanding FPA's scope
and not a cost overrun, the project manager said.[Footnote 13]
Table 1: Agencies' Cost Estimates for Developing FPA:
Dollars in millions.
Year of cost estimate: 2003;
Phase 1 (fiscal years 2002-2006): $11.9;
Phase 2 (fiscal years 2005-2010): $22.0;
Total (fiscal years 2002-2010): $33.9.
Year of cost estimate: 2004;
Phase 1 (fiscal years 2002-2006): 12.2;
Phase 2 (fiscal years 2005-2010): 30.0;
Total (fiscal years 2002-2010): 42.2.
Year of cost estimate: 2005;
Phase 1 (fiscal years 2002-2006): 12.1;
Phase 2 (fiscal years 2005-2010): 36.2;
Total (fiscal years 2002-2010): 48.3.
Year of cost estimate: 2006;
Phase 1 (fiscal years 2002-2006): 11.6[A];
Phase 2 (fiscal years 2005-2010): 31.2;
Total (fiscal years 2002-2010): 42.8.
Year of cost estimate: 2007;
Phase 1 (fiscal years 2002-2006): 11.6[A];
Phase 2 (fiscal years 2005-2010): 32.3;
Total (fiscal years 2002-2010): 43.9.
Year of cost estimate: 2008;
Phase 1 (fiscal years 2002-2006): 11.6[A];
Phase 2 (fiscal years 2005-2010): 32.3;
Total (fiscal years 2002-2010): 43.9.
Source: GAO analysis of Forest Service data.
Note: Costs do not include salaries and benefits for all agency
employees who worked on the FPA project. The senior project manager
responsible for FPA's budget estimated these costs at $9.7 million.
[A] Actual, not estimated; the agencies completed phase 1 of FPA in
2005 at a cost of $11.6 million.
[End of table]
FPA Shows Promise in Achieving Some Objectives but Falls Short of
Others, Although the Agencies Are Considering Changes That May Improve
Its Performance:
Although FPA is not yet complete and our review was limited, FPA shows
promise in achieving some key objectives, including establishing a
common, interagency budget framework that includes important wildland
fire program activities. Nevertheless, FPA is unlikely to achieve all
its key objectives, including the critical objectives of analyzing the
effect over time of different funding allocation strategies and
identifying the most cost-effective mix of firefighting assets. The
agencies recognize that FPA will not fully meet all its key objectives
in 2008 and are considering several changes that may improve its
ability to meet certain objectives in the future. But because the
modeling approach the agencies selected for FPA involves considerable
discretion on the part of agency decision makers, transparency is
particularly vital.
FPA Is to Provide the Foundation for an Interagency Framework for
Analyzing Needed Firefighting Assets and Is to Examine Key Fire
Management Program Activities and Objectives:
If implemented as currently developed, FPA will provide the foundation
for a single framework for the five federal agencies to develop their
budget requests and allocate funds, a key objective. It is also likely
to help the agencies achieve another key objective by analyzing the
most important wildland fire management activities. The agencies have
developed FPA so that it can recognize the presence of nonfederal
firefighting assets that may be available to respond to fires on
federal land--a third key objective--although the extent to which these
assets will be included in the analysis is not yet clear. And finally,
FPA should help the agencies move toward a fourth key objective--
responding to wildland fires in ways that meet specific land and fire
management objectives, rather than simply assuming that all fires
should be suppressed as quickly as possible--although its ability to
fully achieve this objective is likewise uncertain.
FPA Is to Provide the Foundation for an Interagency Budgeting
Framework:
As the agencies are developing it, FPA is to provide the foundation for
a single framework for the five federal agencies to help develop their
wildland fire budget requests and allocate their fire management funds,
as envisioned in congressional guidance and the 2001 agency report--a
significant step forward. In implementing FPA, officials are to work
across agencies, both in the field and at headquarters. In the field,
officials from each agency will need to work together to identify
different mixes and locations of firefighting assets--information that
will enable the FPA model to analyze the effect of different mixes of
firefighting assets without regard to agency jurisdictional boundaries.
At headquarters, agency officials are to work together to determine how
to weight the five performance measures FPA incorporates to identify
the best mix of firefighting assets.
FPA Is to Analyze the Three Most Important Fire Management Program
Activities:
FPA also substantially moves the agencies toward achieving another key
objective by analyzing the three most important fire management
activities: preparedness, fuel reduction, and suppression. FPA is to
directly analyze preparedness and fuel reduction and then model the
effects that varying investments in these activities might have on
suppression costs.[Footnote 14] Together, these activities constitute
most of the agencies' overall fire management budgets.
To analyze the agencies' preparedness for wildland fires, FPA is to
model the potential effect of wildland fire on communities and
resources, depending on the mix and location of firefighting assets
that would be stationed in an area. FPA is to consider historical fire
occurrence and weather patterns to model the likelihood that a fire
might occur in specific areas. Using an interagency database known as
LANDFIRE to identify the fuel types and topography in the location
where a fire is predicted to ignite,[Footnote 15] FPA is to then model
a fire's likely intensity and rate of spread. Finally, FPA is to
identify the location of specific firefighting assets available for
initial attack and, considering the fire's intensity and rate of
spread, determine whether firefighters are likely to contain the fire
before it grows too large and whether the fire is likely to damage
communities or resources.
To analyze the effect of fuel reduction treatments within FPA,
officials in the field are to begin by identifying the attributes of
the fuel reduction treatments they most often undertake in their area,
including vegetation type (such as trees, shrubs, or grasses) and the
treatments' effect on vegetation density, height, and other
characteristics. The FPA model is to then predict the effect of those
treatments on fire behavior and compare the effectiveness of fuel
treatments at reducing fire damage in different areas.
Finally, to analyze suppression costs, FPA is to consider different
levels of investment in preparedness and fuel reduction and, for each
investment level (including the mix and location of firefighting
assets), estimate the number of fires likely to escape initial
suppression efforts. For each such "large" fire, FPA is to use another
model the agencies have developed to predict the cost of suppressing
the fire on the basis of the costs from previous fires with similar
characteristics, including fire size, fuel types, fire intensity,
physical terrain, proximity to the nearest community, and total value
of structures close to the fire. The costs of past fires with similar
characteristics vary widely, however, which limits the model's ability
to accurately predict suppression costs. Moreover, the model is based
on historical costs, and since the agencies have recently begun
emphasizing less aggressive strategies, it may not accurately predict
suppression costs for fires.[Footnote 16] The agencies are continuing
to improve this model, however, which could improve the accuracy of the
cost estimates.
FPA Is to Be Able to Analyze Nonfederal Firefighting Assets, but the
Extent to Which the Agencies Will Include These Assets in Their
Analysis Is Unclear:
Although the agencies are developing FPA to recognize the presence of
nonfederal firefighting assets that may be available to respond to
fires on federal land[Footnote 17]--a key objective of FPA--the extent
to which these assets will be included in the analysis is not yet
clear. When officials in the field enter into FPA the different
combinations of federal firefighting assets they would acquire for a
given budget level, they can also include nonfederal firefighting
assets that are stationed nearby, such as firefighters or fire engines
belonging to state agencies or area communities. FPA is then to
consider the availability of these nonfederal assets when it analyzes
the effect of different combinations of federal assets on the five
performance measures.
FPA officials recognize, however, that some nonfederal entities may
object to federal agencies' including nonfederal assets in their
analysis, for fear that doing so would lead to fewer federal
firefighting assets stationed in certain locations, which in turn could
lead to an additional workload for nonfederal entities in those
locations. The inclusion of nonfederal assets raised significant
concerns among nonfederal entities when the first FPA analysis was
conducted in 2006. And while FPA guidance to planning units in the
field generally directs them to include nonfederal assets, FPA
officials acknowledged the likelihood that field units would receive
"strong objections" to this direction from some nonfederal entities.
Such objections might cause field units to omit nonfederal assets from
the FPA analysis to satisfy the concerns of their nonfederal partners,
with whom they must maintain relationships. FPA officials said they
expect concerns from nonfederal officials to lessen over time, as those
officials become more knowledgeable about how FPA operates. Ultimately,
however, if agency planning units do not include nonfederal assets that
may be available to respond to fires, FPA will model fewer firefighting
assets than are actually present--and may therefore underestimate the
effectiveness of a given set of federal assets. In addition, if some
planning units include nonfederal assets and others do not, FPA's
ability to identify the best combination of federal firefighting assets
nationwide is likely to be compromised.
FPA Is to Consider Land and Fire Management Objectives, but Some
Shortcomings Remain to Be Addressed:
FPA should also help the agencies move toward achieving a fourth key
objective--responding to wildland fires so as to meet specific land and
fire management objectives, as suggested by the 1995 federal wildland
fire management policy, rather than simply assuming that all fires
should be suppressed as quickly as possible--although some agency
officials have concerns about how well FPA will consider land
management objectives. FPA should help the agencies move closer to this
objective in two ways. First, officials in the field are to be
responsible for identifying the number and type of firefighting assets
they would typically dispatch to a fire that ignited in a particular
location under particular conditions. The intent is to recognize that
agency responses vary from fire to fire, and fire managers are more
likely to dispatch more assets to a fire that threatens communities or
highly valued resources or ignites under conditions conducive to rapid
spread than to a fire ignited where it threatened few important
resources or was unlikely to spread. The FPA model is to use this
information to identify locations where stationing proportionately more
firefighting assets might be helpful. Second, officials in the field
are also to estimate the fire intensity beyond which resources in a
particular area are likely to be damaged. In some areas, for example,
officials might establish a relatively high intensity threshold to
recognize that moderate, or even severe, fires might be acceptable,
while in other areas--such as the wildland-urban interface--officials
would likely determine that any fire is undesirable. In evaluating
different mixes and locations of firefighting assets, FPA is to take
into account this variation in acceptable fire intensity. In
determining both the firefighting assets they would dispatch and the
intensity threshold, field officials are expected to use information
contained in local land and fire management plans, which the agencies
are required to develop.[Footnote 18]
Several issues, however, must be addressed for FPA to move the agencies
more fully toward achieving their objective of responding to fires
according to specific land and fire management objectives. First, one
of the measures FPA is to use in evaluating alternative mixes and
locations of firefighting assets is the predicted success of containing
fires before they become large. Although containing fires when they are
small is desirable in many circumstances, the agencies themselves have
also recognized that their legacy of successful suppression has
contributed substantially to the current increase in burned acres and
fire intensity. As noted, it is not clear how the agencies will weight
the relative importance of containing fires early (or indeed how they
will weight any of the five measures) in FPA, but early guidance to the
field indicates that early containment may be weighted heavily, which
would keep FPA from fully recognizing the potential benefits of fire in
some areas. Second, officials from the Fish and Wildlife Service and
National Park Service have expressed concern that FPA is to evaluate
the effects of reducing fuels solely by how the reduction affects the
likelihood of a severe fire, without considering whether the fuel
reduction treatment helps the agencies achieve broader land management
objectives, such as improving the ecological condition of the land over
time, as the 2001 report envisioned. A senior Fish and Wildlife Service
official also noted that many wildlife refuges consist of small parcels
of federal land interspersed among larger parcels of nonfederal land
and that FPA is not designed to consider the effects of fragmented
ownership.
Third, although FPA is to consider specific local land and fire
management objectives that recognize that some areas are more important
to protect than others, it will likely not allow the agencies to
consistently identify the locations that are most important to protect
from a national perspective. Within the five performance measures
evaluating the effects of different mixes and locations of firefighting
assets and fuel treatment options, FPA is to consider all acres as
equally important, despite significant variation in the resources on
those acres. For example, the agencies have established protection of
the wildland-urban interface as one of their most important policy
objectives, and FPA is to treat all interface acres identically,
regardless of whether an acre contains one or several houses.
Similarly, the agencies intend to increase the number of acres that are
meeting fire and fuel management objectives, such as reducing the
likelihood of uncharacteristically intense fires, and FPA is to
consider all acres within this measure identically. For example, FPA is
to consider an acre of a relatively common forest type, such as
ponderosa pine, the same as a relatively rare type, such as giant
sequoia--even though agency managers may place a much greater priority
on the condition of a sequoia forest. As a result, FPA will not likely
allow the agencies to give high priority to meeting objectives in
particularly important or rare areas. FPA is also to predict the
percentage of fires likely to be contained in initial attack. In
evaluating the effect that different mixes and locations of
firefighting assets have on this measure, however, FPA is to weight all
fires equally, regardless of the fires' potential to damage communities
or valuable natural or cultural resources. Agency officials analyzing
FPA results may therefore consider it more important to try to contain
multiple fires that do not pose a great threat than to try to contain a
single fire that does. The presence of the other measures helps to
mitigate this shortcoming, because if an uncontained fire damages
communities or valuable resources, the agencies' ability to meet the
other objectives will be compromised. The relative weights of the five
measures, however, have not yet been determined, and it is not clear
how the measures' interactions will play out.
One of the five measures the agencies will ultimately use to evaluate
different mixes and locations of firefighting assets specifically
considers resources the agencies regard as highly valued, which could
improve the agencies' ability to identify some of the most important
resources to protect. Nevertheless, FPA would still consider all acres
within a particular performance measure identically and therefore not
recognize that it is more important to protect some acres than others.
In August 2008, the agencies decided to include only two types of
resources in this measure in their 2009 analysis: municipal watersheds
and habitat for some endangered species. Senior officials from the four
Interior agencies, however, have criticized the approach the agencies
are developing for FPA to consider highly valued resources because it
does not sufficiently consider their agencies' land management
objectives.
As Designed, FPA Will Not Achieve All Its Key Objectives, Including
Examining the Effects over Time of Differing Funding Allocation
Strategies and Identifying the Most Cost-Effective Mix of Firefighting
Assets:
Even though FPA is likely to achieve several of its key objectives, it
is unlikely to help the agencies achieve others. In particular, the
modeling approach the agencies are taking has limited ability to
examine the effects over time of different funding allocation
strategies and is unlikely to allow them to identify the most cost-
effective mix of firefighting assets. Other aspects of FPA, including
the lack of an external peer review of the model, may complicate its
further development and implementation.
FPA's Ability to Examine the Temporal Effects of Differing Funding
Allocation Strategies Appears Limited:
FPA was envisioned as a way to help the agencies determine the extent
to which, in the short term, increasing or decreasing funding for fuel
reduction treatments and responding more or less aggressively to fires
would affect the expected cost of responding to wildland fires over the
long term. Although FPA is to analyze funding for both preparedness and
fuel reduction, its ability to evaluate the trade-offs associated with
increasing or decreasing one of these activities appears to be limited
to short-term effects. Spending funds to reduce fuels, however, is
generally considered a long-term investment, one whose value increases
over time as more of the landscape is treated. If FPA considers only
short-term effects, it may underestimate the benefit of reducing fuels
and may lead the agencies to place greater emphasis on suppressing
fires than warranted--with potentially far-reaching consequences.
FPA officials told us in September 2008 that they were working with the
interagency science team to develop an approach that would allow FPA to
better analyze the long-term effect of reducing fuels; the officials
expected to incorporate this approach into FPA by November
2008.[Footnote 19] Because the agencies had begun to develop this
approach only toward the end of our review, we were unable to evaluate
it. On the basis of our discussions with FPA officials and members of
the interagency science team, and from our review of the limited
documentation describing the approach, it appears that FPA's ability to
help the agencies achieve this objective will be limited.
Moreover, FPA is unlikely to examine the effects over time of different
firefighting strategies. Since adopting the 1995 fire management
policy, the agencies have increasingly emphasized appropriate
management response. FPA is to recognize that fire managers choose to
respond less aggressively in some cases; for example, it is to allow
field officials to model dispatching fewer firefighting assets to fires
that are unlikely to threaten important resources. Fires responded to
less aggressively are likely to burn many more acres than fires
suppressed quickly. Less aggressive strategies may therefore reduce
fuels on more acres--which in some cases could lower the risk from
future large fires. FPA does not recognize this benefit, however, and
will therefore be unable to help agency officials understand how
responding less aggressively now may reduce the size and intensity of
fires later, which could in turn help the agencies protect communities
and resources and lower the cost of suppressing fires.
More broadly, FPA's limited ability to examine the effect over time of
reducing fuels and implementing appropriate management response could
also limit its ability to help the agencies develop a long-term,
cohesive strategy for responding to wildland fires. We have long
recommended that the agencies develop a cohesive strategy identifying
available long-term options and associated funding for reducing
hazardous fuels and responding to wildland fires.[Footnote 20] Such a
strategy is fundamental if the agencies and Congress are to fully
understand the potential choices, and associated costs, for addressing
wildland fire problems. The agencies have consistently concurred with
our recommendation,[Footnote 21] and agency officials cited FPA as a
key step in enabling them to develop a cohesive strategy. In its
current state of development, however, FPA lacks important capabilities
to help inform strategic decisions about how to invest the agencies'
limited funds.
FPA Is Unlikely to Allow the Agencies to Identify the Most Cost-
Effective Mix of Firefighting Assets:
A primary objective of FPA, established by the 2001 agency report, is
to identify the most cost-effective fire management program for a given
budget. Accomplishing this objective requires the agencies to define
the fire management objectives they are trying to achieve and then to
identify the combination of fuel reduction treatments and suppression
strategies, including the best mix and location of firefighting assets,
that would result in the most effective use of program funds. The
modeling approach the agencies are using in FPA, however, does not
allow the agencies to meet this objective. Rather than analyzing all
possible combinations of assets typically available to local units, as
well as fuel reduction and fire suppression strategies, to identify the
most cost-effective combination, the approach the agencies are taking
allows them to compare only a limited number of asset mixes and
firefighting strategies (including fuel reduction options) to determine
whether one mix of assets and strategies is more or less cost-effective
than another. Because FPA is to compare only a limited number of
alternatives, the evaluated alternatives are unlikely to include the
most cost-effective mix of assets nationwide. Further, because the
evaluated alternatives are likely to reflect minor variations in budget
levels (e.g., plus and minus 5 percent or 10 percent of the prior
year's budget for each planning unit), the present FPA approach is
likely to generate results that differ only incrementally from the
asset mixes and strategies already in place, rather than evaluate
whether significantly different alternatives could yield significantly
better results.
Agency officials, including key scientists involved in FPA development,
told us they believed that although the modeling approach has not been
designed to identify the single most cost-effective mix of firefighting
assets, FPA would nevertheless provide useful information to help the
agencies develop their budget requests. In fact, several officials told
us they preferred the flexibility currently built into FPA, which
allows them to consider multiple potential budget scenarios--what one
official termed a "family of solutions"--over the rigidity built into
the old approach, which resulted in a single solution. Officials told
us that it would be unrealistic to expect that the complexities of
wildland fire management could be modeled accurately enough to yield a
single solution that is truly optimal and that by examining multiple
possible budget scenarios developed by officials in the field, FPA's
new approach would yield results that would be "among the most cost-
effective solutions," according to one official. Nevertheless, it is
not clear that examining only a small number of alternatives for each
planning unit will generate results that are among the most cost-
effective, particularly given current guidance to the field to consider
only slight variations from current funding levels when developing
alternatives.
Moreover, in analyzing trade-offs among different mixes and locations
of firefighting assets, FPA is to consider only those assets that are
stationed at individual management units, not those that are centrally
located and under regional or national control. These central assets,
which include large air tankers and helicopters and many of the most
qualified firefighters, are some of the agencies' most costly,
representing about $200 million of their budgets, according to agency
estimates. The agencies use these assets in two ways: to assist local
units with initial attack on small fires and to help suppress large
fires. FPA is to consider the presence of these central assets when
analyzing the likelihood that firefighters will be able to contain a
fire during initial attack--important because otherwise the model would
suggest that more firefighting assets would be needed at local planning
units. FPA is to consider the number of centrally located assets as a
given, however--that is, as a fixed input to the model, not a variable-
-rather than analyze the effects of changing the number of centrally
located assets or the proportion of assets under local or national
control. As a result, the model is not likely to determine the effect
of changing the number or type of these assets on the agencies'
firefighting abilities and costs, thus further limiting its ability to
identify the most cost-effective mix of assets nationwide.
Other Aspects of FPA Complicate Further Development and Implementation:
Although FPA is a new system, it will rely on many data sources,
models, and systems the agencies developed earlier, some of which have
known shortcomings. For example, FPA is to use data from the LANDFIRE
system to identify the fuel types across the country; yet the accuracy
of LANDFIRE data has been questioned, as has the frequency with which
the system will be updated to recognize changes in fuel conditions over
time due to insect outbreaks, large wildland fires, or other
disturbances. Over the past several years, FPA and LANDFIRE project
officials have worked together to develop a process to update LANDFIRE
data, which should benefit FPA. It is too early to tell how effective
the planned LANDFIRE improvements will be.
To predict how quickly a fire may spread in different fuel, weather,
and geographical conditions, FPA is to use the results of FSPro, a fire
growth model developed by Forest Service scientists. Fire officials
have recognized that the spread rate predicted by the model is not
always consistent with the rate of spread they observe during real
fires. To help compensate for this difference, the FPA model is to
allow field officials to calibrate the data used by FSPro to model
spread rates so that they more closely reflect conditions typically
observed in a particular area. It is not clear, however, how the
agencies will ensure that the calibrations are made consistently or
what the effects may be on the mix of firefighting assets FPA
identifies as most appropriate.
Even with their known shortcomings, some of FPA's component elements
are well-established applications that have been used by wildland fire
managers for many years and, in some cases, are based on peer-reviewed
science. The FPA model as a whole, however, including its component
parts, has not been externally peer-reviewed. This lack comes in part
because documentation on FPA's development and capabilities has not
been sufficiently developed to allow for peer review; instead,
according to agency officials, project staff have been devoting time to
model development. Until the model is peer-reviewed--including
validation that the overall logic is sound, the methods used are state
of the art, the results are consistent with empirical evidence, and the
system is adequate for its intended purpose--neither the agencies nor
outside parties will have a full understanding of FPA's strengths and
limitations or know how much confidence they should place in the
model's analysis. A peer review, moreover, may identify limitations not
revealed by our review.
Finally, the utility of FPA in identifying the best mix of firefighting
assets will depend heavily on the alternatives and data developed by
officials in the field, but some field officials expressed concerns
about this component of FPA. For example, some field officials are
concerned that they will receive little training on how to use FPA,
which may prevent them from developing the most realistic options, and
that the time needed to enter data into FPA and develop alternatives
for national consideration will substantially increase their workload.
Other officials told us that they are concerned that field staff may
try to "game" FPA in an attempt to get the model to identify their area
as needing more assets. Staff could, for example, develop a less-
effective alternative for their low-budget scenario to make their mid-
or high-budget scenarios appear more effective; similarly, staff may
find it expedient to develop alternatives that ensure that each agency
in their planning unit gains or loses comparable quantities of
firefighting assets in order to promote equity among the agencies,
rather than develop alternatives likely to best protect important
resources but which might affect one agency more than another. Senior
agency officials told us that gaming is a concern with any budgeting
system and that they are planning to establish a two-stage process to
review field submissions. In the first stage, officials from other
field units would review the alternatives to ensure that interagency
guidance was followed and information entered correctly; in the second,
regional officials would review the results to ensure they met regional
priorities. The exact steps this review process would follow, however,
have not been determined, so it is not yet clear whether this process
will ensure that only appropriately developed alternatives are
submitted.
The Agencies Are Considering Changes That May Increase FPA's Ability to
Meet Certain Key Objectives, Although These Changes Do Not Address All
Shortcomings:
Senior agency officials told us they recognize that in 2008 FPA will
not fully meet all its key objectives but said they are considering
making several changes that may improve its ability to meet certain key
objectives, including the following:
* submitting FPA for an external peer review;
* continuing to develop FPA's process for identifying the resources the
agencies consider to be highly valued and assessing the agencies'
ability to protect these resources; and:
* working with the interagency science team to improve how FPA is to
consider land management objectives, such as improving the ecological
condition of the land over time, when evaluating the benefits of
reducing fuels.
* Although these steps have the potential to move FPA closer to meeting
some of its key objectives, it is too early to determine how successful
they will be. Moreover, these steps do not address all shortcomings we
or others have identified, and taking these or other steps to improve
FPA will carry an additional cost, which is not included in current
agency estimates. The approximately $54 million estimated cost for FPA
includes basic operation and maintenance through fiscal year 2010 but,
according to agency officials, does not include funds to make the above
improvements.
The Approach Selected for FPA Increases the Importance of Transparency
in Decision Making:
The approach the agencies have taken in developing FPA allows for
considerable discretion on the part of agency decision makers in three
key areas: determining the relative importance (that is, the weights)
of the five performance measures used to evaluate locally developed
alternatives; using FPA results in combination with other information
to develop agency budget requests; and using FPA results, likewise in
combination with other information, to allocate funds to the field.
Although it is important that decision makers have the flexibility to
consider various options, that same flexibility makes it essential for
the agencies to ensure that these processes are fully transparent.
Otherwise, Congress, the public, and agency officials cannot be assured
of fully understanding the rationale behind decisions or FPA's role in
them. Although any changes to the existing allocation of funds among
agencies or across different geographical areas are likely to be
incremental at first, the agencies could consider larger funding
reallocations as their understanding of FPA increases--which would make
transparent decision making even more important.
First, as noted, the extent to which any particular alternative or set
of alternatives is considered cost-effective will depend on the
relative importance assigned to the five performance measures,
including any variation in their relative importance in different
regions of the country. FPA officials and the leaders of the
interagency science team said that FPA is being designed to allow for
the agencies to evaluate different weighting schemes, which senior
agency officials referred to as "exploring the decision space." Others,
however, have raised concerns that the flexibility inherent in setting
weights for the different performance measures will allow the agencies
to manipulate these weights until they reach a predetermined outcome.
Without understanding the weights assigned to each measure and the
rationale for assigning those weights--that is, without transparency in
this process--Congress and others will find it impossible to understand
and evaluate the reasonableness of FPA's results, and skepticism about
FPA's usefulness will be difficult to quell.
Second, senior agency officials emphasized that, despite its
importance, FPA will not be the sole determining factor in developing
their budget requests and allocating appropriated funds; rather, senior
agency officials would consider FPA results along with other
information and exercise managerial discretion in making these
decisions. Agency officials said, for example, that they would continue
to involve national and regional officials from the various agencies to
help ensure that their budget requests reflected differences in
priorities among the agencies or regions, although they recognized that
this process might lead FPA results to be used differently by different
agencies or in different regions. Although considering these factors is
important, as with the setting of the weights, it will also be
important for the agencies to clarify the additional factors beyond FPA
that they consider in developing their budget requests, so that
Congress and others can understand FPA's role in the process.
And third, once Congress has appropriated funds to the agencies, it is
not clear how the agencies will use FPA to help allocate these funds to
the field. If one agency allocated funds differently than suggested by
FPA--or if one agency's field unit acquired a different mix of assets
than it modeled--it could affect the other agencies' ability to protect
important resources, as well as the overall effectiveness of the
agencies' fire management program. Agency officials said they intended
for each agency to consider FPA results in allocating its funds and for
field units to consider FPA results in acquiring firefighting assets.
They also said they would not decide how much to deviate from the
allocation suggested by FPA until they had begun to analyze the first
year's results. Officials also said that it is important to recognize
that more than 2 years could elapse between field units' developing
their alternatives and Congress's appropriating funds on the basis of
that information--and that priorities could change substantially in the
interim, leading the agencies to allocate funds differently than
suggested by FPA. Agency officials also said that, in addition to FPA
results, they would consider specific congressional earmarks and
appropriations guidance when allocating funds. Moreover, the agencies
have existing systems outside of FPA for allocating fuel reduction
funds, which they have been working in recent years to
improve.[Footnote 22] As of November 2008, agency officials did not
know how they would consider the information from FPA in relation to
the agencies' other systems in allocating fuel reduction funds.
Conclusions:
As fires become more severe and development in fire-prone lands
continues, the Forest Service and Interior agencies face difficult
decisions about how to best protect the nation's communities and
natural and cultural resources. In particular, the agencies must
determine the best mix and location of firefighting assets to respond
to wildland fires, and they must balance the need to spend money
preparing for and fighting fires against the need to invest in reducing
potentially hazardous fuels so as to lower both the cost of suppressing
future fires and the risk to communities and important resources.
Complicating these decisions, our nation's long-term fiscal challenges
have constrained agency budgets, simultaneously limiting available
choices and making it even more important to spend funds efficiently
and effectively. The agencies believe that FPA can be a useful tool in
making these difficult choices, which will drive billions of dollars in
federal expenditures each year and directly affect millions of citizens
living in fire-prone areas. By establishing an interagency budget
framework that analyzes trade-offs among the most important fire
management program activities, FPA represents an important first step
in improving the agencies' cost-effectiveness.
Achieving the full potential of FPA, however, will depend on the extent
to which the agencies improve FPA's ability to live up to the promises
that were made on its behalf--namely, that it would allow the agencies
to develop rational budgets and allocate funds in a way that maximizes
the agencies' ability to manage wildland fire. Living up to these
promises presents a daunting challenge, given the inherent difficulty
of modeling the complexities and uncertainties of wildland fire and
given that FPA remains a work in progress. Nevertheless, an early
assessment of the model's capabilities raises several issues. The
overall modeling approach the agencies have chosen does not allow them
to identify the most cost-effective mix and location of firefighting
assets, one of FPA's key objectives. Moreover, without improvements,
FPA will be unable to identify, from a national perspective, the most
important resources to protect or the relative priority of different
values at risk; to evaluate the effect of different investments in fuel
reduction treatments and firefighting strategies over time; or to
analyze the effect of changes in the number of aircraft and experienced
firefighters that are under regional or national control. Without such
improvements, the agencies will continue to lack important information
on which to base decisions about how best to allocate scarce funds.
Further, the agencies have not yet determined how they will weigh the
relative importance of FPA's five performance measures or exactly how
they will use FPA to develop their budget requests and allocate funds.
Given the importance of and the uncertainty surrounding these
decisions, Congress--as well as the agencies and other interested
parties--would benefit if these fundamental budget decisions were made
in a transparent manner. And finally, an external peer review by an
independent entity, such as the National Academy of Sciences, would
achieve one of FPA's objectives and help the agencies identify the
strengths and limitations of the model, which could increase confidence
in their decisions and help them make needed changes more quickly.
Recommendations for Executive Action:
We recommend that the Secretaries of Agriculture and the Interior take
four actions to improve their agencies' abilities to develop their
budget requests and allocate funds using FPA.
First, to improve the FPA model's ability to identify needed
firefighting assets and the best locations for these assets--and
recognizing that developing FPA will be an iterative process that will
require the agencies' continued effort to improve--we recommend that
the Secretaries of Agriculture and the Interior direct the agencies to
develop a strategic plan for the continued development of FPA, which
would (1) include an evaluation of FPA's ability to meet its key
original objectives; (2) identify ways to improve the model to better
meet these objectives; (3) clearly state whether the agencies believe
any of the original objectives are no longer appropriate, and why; and
(4) identify the steps the agencies plan to take to improve FPA and the
expected time frames and associated budget needs for completing these
steps. To allow the agencies sufficient time to identify issues that
may arise as they implement FPA, the Secretaries of Agriculture and the
Interior should submit this plan to Congress no later than September
30, 2010. In particular, we believe that the strategic plan should, at
a minimum, address ways to improve FPA's ability to:
* evaluate different mixes and locations of firefighting assets, so
that FPA recognizes the relative priority of different values at risk
when assessing how best to protect the wildland-urban interface and
increase the number of acres meeting fire management objectives;
* identify the most highly valued resources, such as endangered species
habitat or important cultural sites, that the agencies seek to protect;
* model the effects over time of different investments in fuel
reduction treatments and firefighting strategies on the cost of
suppressing future wildland fires; and:
* analyze trade-offs between increases and decreases in firefighting
assets that are under national or regional control.
Second, we recommend that the Secretaries of Agriculture and the
Interior report annually to Congress on (1) their progress in
completing the steps outlined in the strategic plan for the continued
development of FPA and (2) FPA's ability to meet each of its key
objectives.
Third, to increase agency transparency in using FPA to develop their
budget requests and allocate funds, we recommend that the Secretaries
of Agriculture and the Interior report annually to Congress on FPA's
role in the budget development and allocation process. This report
should include, at a minimum, information on (1) how the agencies
weighted the measures FPA uses to evaluate different mixes and
locations of firefighting assets and the rationale for those weights,
(2) how FPA results were used in conjunction with other information in
developing the agencies' budget requests, and (3) the extent to which
the agencies' funding allocations to their field units reflected the
FPA results for a given year.
Fourth, to increase Congress's and the agencies' understanding of the
strengths and limitations of FPA--including the extent to which it
achieves the key objectives envisioned by the 2001 report--and to
fulfill one of the original objectives established for FPA, we
recommend that the Secretaries of Agriculture and the Interior direct
the agencies to submit the FPA model to external peer review. This
review should be initiated as soon as FPA is complete enough to allow
for a thorough review, but no later than November 2009, so that its
results can inform decisions about how FPA may be improved and the
extent to which additional funding should be provided to the project.
Agency Comments and Our Evaluation:
In written comments on a draft of this report, the Forest Service and
Interior disagreed with our finding that FPA is unlikely to allow the
agencies to identify the most cost-effective mix of firefighting
assets. The Forest Service commented that it fundamentally agreed with
our recommendations and described the steps the agency intended to take
in addressing them, but we do not believe that the steps outlined in
the letter are specific and transparent enough to meet the intent of
our recommendations. Interior disagreed with our recommendation that
the agencies develop a strategic plan for the continued development of
FPA but concurred with our other recommendations.
The Forest Service and Interior commented that they believe FPA will
allow them to meet the goal of cost-effectiveness. As their letters
state, we previously discussed our conclusions on this issue with the
agencies but did not resolve the differing points of view. As stated in
our report, FPA compares only a limited number of mixes of firefighting
assets and firefighting strategies, and the alternatives it evaluates
are likely to reflect only minor variations in budget levels. Given
this structure, we continue to believe that FPA is unlikely to allow
the agencies to identify the most cost-effective location and mix of
assets and strategies nationwide--an objective the agencies themselves
established in their 2001 report. In their responses, both agencies
raised questions about this objective. The Forest Service's comments
seek to invalidate the objective altogether, stating that identifying
the single most cost-effective mix of assets and strategies is not
realistic. Interior did not question the validity of the objective but
stated that the approach FPA is taking is more realistic than the
approach the agencies had taken when they first began developing FPA.
While we are not altering our conclusion that FPA's current approach
will likely keep the agencies from identifying the most cost-effective
solution, we are modifying our first recommendation to state that in
the strategic plan, the agencies not only identify ways to improve the
model to better meet FPA's original objectives, but also clearly state
whether they believe any of the original objectives are no longer
appropriate--and, if not, why not--in order to ensure that Congress and
other interested parties are fully informed about what they can
reasonably expect from FPA.
Regarding our recommendation that the agencies develop a strategic plan
for the continued development of FPA, the Forest Service concurred with
our recommendation and stated that it has a strategy for completing
FPA, although it is not clear from the letter whether this strategy is
or will be articulated in a written document directly addressing the
elements of our recommendation. In contrast, Interior disagreed with
this recommendation, stating that developing a strategic plan would
delay the deployment and increase the cost of FPA. Regarding Interior's
position, we are not suggesting that the agencies delay implementing
FPA until they have developed the strategic plan we recommend; rather,
we believe that such a plan can be developed concurrently with
implementation and in fact may benefit from incorporating lessons
learned during early use of FPA. More broadly, because of FPA's
importance and the concerns about its development--including the
questions raised in our review about its ability to meet its key
objectives--we believe it is important for the agencies to create a
strategic plan for FPA's continued development that directly and
transparently evaluates FPA's ability to meet its original objectives,
identifies ways to improve FPA to better meet those objectives, and
identifies the steps the agencies plan to take to improve FPA. Given
the agencies' comments about FPA's cost-effectiveness objective,
however, we modified the language of our recommendation on developing a
strategic plan, as discussed above.
The Forest Service and Interior generally agreed with our
recommendations to report annually to Congress on the continued
development of FPA and on FPA's role in the budget development process,
and to submit the FPA model to external peer review. The Forest
Service, however, also provided clarifications on two of our
recommendations that did not appear to be fully responsive in terms of
the amount of information and transparency we believe is warranted.
Specifically, in response to our recommendations that the agencies
report annually to Congress on (1) their progress in completing the
strategic plan for FPA's continued development, and on FPA's current
ability to meet each of its key objectives, and (2) FPA's role in the
agencies' budget development and allocation process, the Forest Service
stated that it has always been--and will continue to be--responsive to
congressional requests for information and that it would include
information on FPA's role in budget development and allocation in its
annual budget requests. We are not convinced, however, that this
approach will furnish Congress with the consistent, transparent, and
complete information we believe it needs--particularly given FPA's
importance in helping the agencies manage their $3 billion wildland
fire program and the concerns about its development. We continue to
recommend, therefore, that the Secretaries of Agriculture and the
Interior prepare an annual report to Congress about the status of FPA's
development and how the agencies have used FPA to help develop their
budget requests and allocate funds. The Forest Service's and Interior's
letters are reprinted in appendixes II and III, respectively, along
with our evaluation of specific comments.
We are sending copies of this report to interested congressional
committees, the Secretaries of Agriculture and the Interior; the Chief
of the Forest Service; the Directors of the Bureau of Indian Affairs,
Bureau of Land Management, Fish and Wildlife Service, and National Park
Service; and other interested parties. The report is also available at
no charge on the GAO Web site at [hyperlink, http://www.gao.gov].
If you or your staff have questions about this report, please contact
me at (202) 512-3841 or nazzaror@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. Key contributors to this report are
listed in appendix IV.
Sincerely yours,
Signed by:
Robin M. Nazzaro:
Director, Natural Resources and Environment:
[End of section]
Appendix I: Scope and Methodology:
To determine how the agencies have developed the fire program analysis
(FPA) budget-planning system to date, we reviewed agency documents from
each stage of FPA's development. To identify the key objectives
originally established for FPA, we reviewed congressional committee and
Office of Management and Budget guidance to the agencies; a 2001
report, commissioned and later adopted by the agencies, that
established the vision, key objectives, time frames, and rationale for
what FPA was intended to accomplish; the interagency memorandum of
agreement and project charter that established FPA as an interagency
project; and other agency documents. To further our understanding of
the broader context for the shortcomings FPA was intended to address,
we reviewed key agency documents, including the 1995 and 2001 federal
wildland fire management policies, the national fire plan, and related
documents. To identify changes the agencies made to FPA in 2006, and
the reasons for those changes, we reviewed the report the agencies
issued after their review of FPA's policy and scientific approaches; a
response to that report prepared by those who had helped to develop
FPA; and internal agency briefing materials about the changes. To
identify the likely capabilities of FPA as the agencies have been
developing it since 2006, we reviewed the draft interagency science
team report that formed the basis for FPA's new modeling approach and
numerous technical papers and other documentation describing particular
aspects of FPA. To further our understanding of FPA's development at
each of these stages, we interviewed Forest Service and Department of
the Interior officials in Washington, D.C; FPA project staff in Boise,
Idaho; and agency officials in the field familiar with FPA. We also
interviewed agency and other scientists who have helped develop FPA.
To determine the extent to which FPA meets its original objectives, we
compared--to the extent possible--the capabilities of FPA as the
agencies developed it with those envisioned in congressional committee
guidance and the 2001 report. At the time of our review, however,
substantial portions of the model remained incomplete, and the agencies
had not sufficiently documented the model to allow a comprehensive
evaluation. We therefore limited our review to a broad examination of
FPA's various components and how they interact. We also interviewed
senior agency officials, FPA project staff, agency field officials, and
agency and other scientists to obtain their views on the extent to
which FPA appears capable of meeting its key objectives, as well as
possible changes that could improve the model's ability to meet those
objectives.
We conducted this performance audit between September 2007 and November
2008 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objectives.
We believe that the evidence obtained provides a reasonable basis for
our findings and conclusions based on our audit objectives.
[End of section]
Appendix I: Comments from the Department of Agriculture, Forest
Service:
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
United States Department of Agriculture:
Forest Service:
Washington Office:
1400 Independence Avenue, SW:
Washington, DC 20250:
File Code: 1420/5100:
Ms. Robin M. Nazzaro:
Director, Natural Resources and Environment:
United States Government Accountability Office:
4441 G Street, N.W.:
Washington, DC 20548:
Dear Ms. Nazzaro:
Thank you for the opportunity to comment on the draft Government
Accountability Office (GAO) report, GAO-09-68, "Wildland Fire
Management: Interagency Budget Tool Needs Further Development to Fully
Meet Key Objectives." The Fire Program Analysis (FPA) project is a very
significant and challenging undertaking by the Federal wildland fire
agencies. We were pleased the audit team discussed the complexities
inherent in developing an interagency planning and budget system. While
we generally view the audit as supportive, we respectfully disagree
with GAO's conclusion that our approach hampers FPA from meeting the
key objective of cost effectiveness. We discussed this with the audit
team; they were receptive to the discussion and recommended that we
document our concern.
As indicated, we worked closely with GAO, commenting and clarifying
statements; however, not all of these have been reflected in key
aspects of the draft report. We believe GAO has not accurately
portrayed the system's ability to meet the cost effectiveness objective
and actions we have taken to assure that FPA is a useful planning and
budgeting tool. GAO takes exception to FPA system design modifications
in 2006 that it says compromises the agencies' ability to fully achieve
key goals. We strongly disagree that these modifications compromise
cost effectiveness.
See comment 1.
GAO contends that the analytical approach developed by the agencies
cannot identify the most cost effective mix and location of federal
firefighting assets for a given budget, but rather can only compare the
relative cost effectiveness of a small set of alternatives. This
assertion fails to recognize or acknowledge several key points. First,
the notion of a single most cost-effective solution is conceptual; it
is not based in reality. Any reasonable alternative can be shown to be
optimal under a hypothetical set of assumptions about fire occurrence,
weather, and values placed on the consequences of individual fires.
That same alternative would be judged inferior to other alternatives
under differing sets of assumptions, even if the differences are minor.
Furthermore, building an optimization model requires gross
simplification of the system being modeled in order to calculate a
solution. This simplification can lead to distorted or inaccurate
representations of the relationships between actions and outcomes,
which reduces the confidence placed in a given solution. These and
other shortcomings of the classical optimization approach were manifest
in the first phase of FPA and led the agencies to change their
approach.
The analytical approach recommended by the Interagency Science Team and
adopted by the agencies remains faithful to the goal of improving
firefighting effectiveness. It also improves the ability to: A) address
the uncertainty inherent in wildland fire due to random variations in
weather, fuels, and topography; B) more realistically model fire
behavior and the strategic and tactical choices made by wildland fire
managers and; C) build upon the corporate intelligence gained from
decades of firefighting by the agencies and their partners.
See comment 2.
This approach will allow us to systematically evaluate alternative
investment strategies and identify options that best reduce fire
losses, improve ecological conditions and increase cost efficiency. The
system is designed to explicitly address uncertainty and risk in
predicting future wildland fires. A combination of simulation models
and goal programming will array alternatives using quantitative
performance measures that display inherent risks and trade- offs at
both local and national levels. This approach is a more robust basis
for modeling real- world complexities than the linear optimization
approach originally used in FPA, while maintaining the ability to
compare the performance and effectiveness of alternative funding
decisions.
We would also like to offer comments relative to the system's ability
to evaluate fuel reduction investments over time. GAO states that we
are working "to develop an approach that would allow FPA to better
analyze the long-term effect of reducing fuels" but that due to the
development time frame they were unable to evaluate it, reaching the
opinion "it appears that FPA's ability to help the agencies achieve
this objective will be limited." GAO concludes that "without
improvements FPA will be unable to evaluate the effect of different
investments in fuel reduction and firefighting strategies over time."
We want to reaffirm our commitment to ensure that the system is useful
and that it supports, both near-term and long-term, fire planning and
budgeting. To that end, our development and science teams are
aggressively analyzing temporal modeling approaches for fuel treatments
which could be released later this year. In addition, the performance
metric "proportion of land meeting or trending toward the attainment of
fire and fuels management objectives" will recognize and consider acres
managed under the appropriate management response. The system now being
deployed is within the scope approved by the Agency. However, the
Forest Service has recognized the potential for a more comprehensive
analysis of vegetation and fuel treatments that will address the
concerns expressed by GAO.
See comment 3.
GAO Recommendations - The report identifies four recommendations to
improve FPA and its use in the budget process. The Forest Service
fundamentally agrees with GAO's recommendations, but believes there are
better alternative approaches for implementing three of the
recommendations than those proposed by GAO.
Recommendation 1: The Secretaries of Agriculture and the Interior
direct the agencies to develop a strategic plan for the continued
development of FPA, which would include: 1) an evaluation of the
strengths and weaknesses of FPA; 2) identify ways to improve the model
to better meet its intended objectives, and; 3) identify the steps the
agencies plan to take to improve FPA and expected time frames and
associated budget.
Response - The Forest Service has a strategy for completing development
and implementation of the FPA system consistent with the project's
charter, and its associated plans. The Forest Service has recognized
the need to implement FPA in an adaptive manner in FY 2009 through a
staged approach that allows for system adjustments as experience
dictates. This approach is both helping to avoid workload issues for
the system and development of personnel and facilitating our ability to
address system issues as they arise. The enclosed graphic displays the
schedule and methodology for implementation across all Fire Planning
Units. As we use the system and review and analyze its outputs, its
strengths and weaknesses will be identified and documented through
existing business processes. In addition, the planned external peer
review will provide insight into FPA's strengths and weaknesses.
See comment 4.
Enclosure not reprinted.
Major development of FPA will be complete in FY 2009 when the system
will transition from development to operation and maintenance. The
system's Operation and Maintenance Plan will provide for some
enhancements to the system's models. In addition, the Forest Service
has recognized other modeling components, such as a focused analysis of
national resources or an expanded analysis of fuels and vegetative
treatments, which could be useful and potentially provide a more
comprehensive range of alternatives. These will be considered as the
system is deployed and insights into outputs, and their utility, become
known. The FPA Executive Oversight Group will consider these and other
enhancements and provide guidance relative to their future inclusion
and development.
Key aspects of these actions and activities will be conveyed in
accordance with the Forest Service's response to Recommendations 2 and
3.
Recommendation 2: The Secretaries of Agriculture and the Interior
report annually to Congress on: 1) their progress in completing the
steps outlined in the strategic plan for the continued development of
FPA; and, 2) FPA's ability to meet the key objectives initially
established for it.
Response - The Forest Service agrees with informing Congress about the
progress in implementing FPA and how the results of FPA are being used
in agency decision processes. The agency has always been responsive to
Congressional requests for information through informal briefings,
responses to written questions, and formal testimony. The Forest
Service will continue to respond to any requests for information by
members and committees. In addition, the results and use of FPA
information will be clearly highlighted in the Agency's formal annual
budget requests to Congress.
See comment 5.
Recommendation 3: To increase agency transparency in using FPA to
develop their budget requests and allocate funds, the Secretaries of
Agriculture and the Interior report annually to Congress on FPA's role
in budget development and allocation process.
Response – Please see response to recommendation 2.
Recommendation 4: The Secretaries of Agriculture and the Interior
direct the agencies to submit the FPA model to external peer review.
Response - The Forest Service agrees with the recommendation. An
external peer review is planned as part of the FY 2009 development and
implementation strategy.
Please contact Sandy T. Coleman, Forest Service Assistant Director for
GAO/OIG Audit Liaison staff, at 703-605-4699, with any questions.
We look forward to working with GAO in the future.
Sincerely,
Signed by:
Abigail R. Kimbell:
Chief:
cc: Sandy T Coleman, Clarice Wesley, Tom Harbour, Bill Breedlove, Rick
Prausa:
The following are GAO's comments on the Department of Agriculture,
Forest Service's letter dated November 5, 2008.
GAO Comments:
1. As the Forest Service's comment letter indicates, we have had
extensive discussions with the agency on FPA's ability to identify the
most cost-effective mix of firefighting assets, without resolving our
differing points of view. As we describe in our report, FPA compares
only a limited number of mixes of firefighting assets and firefighting
strategies, and further, the alternatives it evaluates are likely to
reflect only minor variations in budget levels. Given this structure,
we continue to believe that FPA is unlikely to allow the agencies to
identify the most cost-effective location and mix of assets and
strategies--an objective the agencies themselves established in their
2001 report. Rather than directly contradicting our conclusion,
however, the Forest Service's letter seeks instead to invalidate this
objective altogether. The Forest Service commented that identifying the
most cost-effective mix of assets and strategies is not a realistic
objective and that FPA's current combination of simulation models and
goal programming is a preferable approach. We did not compare the
agencies' current approach with their initial approach, nor have we
concluded whether one approach is more suitable or realistic than the
other. Rather, in accordance with the objectives of our review, we
simply evaluated the extent to which FPA as it is currently being
developed is likely to meet the objectives originally established for
it. While we are not altering our conclusion that FPA's current
approach will likely not result in identifying the most cost-effective
solution, we are modifying our first recommendation to suggest that the
agencies clarify which of FPA's original objectives they believe are no
longer appropriate and why. See comment 4 below.
2. The Forest Service stated that FPA remains faithful to the goal of
improving firefighting effectiveness and that FPA's approach will
provide a more robust basis for systematically evaluating alternative
investment strategies. As discussed above, however, and as we noted in
our draft report, the objective originally established for FPA was to
identify the most cost-effective mix and location of firefighting
assets and strategies, not simply to improve firefighting
effectiveness.
3. The Forest Service's letter reaffirmed the agency's commitment to
ensuring that FPA is able to support both near-term and long-term
planning considerations in evaluating fuel reduction investments. In
reaffirming this commitment, the Forest Service stated that FPA, as it
is being developed, fulfills the scope that has been approved by the
agency. This approved scope, however, has evolved during FPA's
development and is not fully consistent with the objectives initially
established for FPA. Our conclusions about FPA are based on our
comparison of its current capabilities with the objectives originally
established for it. We did not determine whether the agencies were
developing FPA in a manner that fulfilled the scope approved by the
agencies in subsequent documents.
4. The Forest Service stated that it has a strategy for completing FPA
"consistent with the project's charter and its associated plans." It is
not clear from the letter, however, whether this strategy is, or will
be, articulated in a written document directly addressing the elements
of our recommendation. Because of FPA's importance, and the concerns
that have arisen during its development, we believe it is important for
the agencies to develop a single document that addresses these issues
transparently. Given the agencies' comments, however, we modified our
recommendation to suggest that the agencies use this plan not only to
identify ways to improve the model to better meet FPA's original
objectives, but also to clearly state whether they believe any of the
original objectives are no longer appropriate, and why, in order to
ensure that Congress and other interested parties are fully informed
about what they can reasonably expect from FPA.
5. The Forest Service stated it would respond to any requests for
information by Congress and would highlight how FPA's results were used
in the agency's annual budget request. We are not convinced, however,
that this approach will provide Congress with the consistent,
transparent, and complete information we believe it needs--particularly
given FPA's importance in helping the agencies manage their $3 billion
wildland fire program and the concerns about its development. We
continue to recommend, therefore, that the Secretaries of Agriculture
and the Interior prepare an annual report to Congress about the status
of FPA's development and how the agencies have used FPA to help develop
their budget requests and allocate funds.
[End of section]
Appendix II: Comments from the Department of the Interior:
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
The Associate Deputy Secretary Of The Interior:
Washington:
November 4, 2008:
Ms. Robin M. Nazzaro:
Director. Natural Resources and Environment:
Government Accountability Office:
441 G Street, NW:
Washington, D.C. 20548-0001:
Dear Ms. Nazzaro:
We appreciate the opportunity to review and comment on the draft
Government Accountability Office report entitled, "Wildland Management
Interagency Budget Tool Needs Further Development to Fully Meet Key
Objectives." (GAO-09-68). The Fire Program Analysis project is a very
significant and challenging undertaking by the Federal wildland fire
agencies We were pleased the audit team engaged in numerous and
constructive discussions to understand the complexities inherent in
developing an interagency planning and budget system. While we view the
audit as supportive of our effort we respectfully disagree with GAO's
conclusion that our approach hampers FPA from meeting key objectives.
We discussed this concern with the audit team: they were receptive to
the discussion and recommended that we document our concern.
We arc concerned that our comments and clarifying information have not
been reflected in the report. In particular, we believe that FPA will
allow us to meet the cost effectiveness objective and that actions we
have taken will ensure that FPA will he a useful planning and budgeting
tool. The FPA will he useful in meeting the cost effective objective
and he a useful budgeting and planning took because FPA is an
interagency analysis that is based on collaboration at the local level.
The results of the collaborative analysis process will display
efficiencies and effectiveness identified at the local planning level.
We also believe that the 2006 system modifications support the goal of
cost effectiveness. The cost effectiveness is based on quantitative
performance measures that display trade-offs locally and nationally
between various budget levels. The FPA system allows the local planning
unit to determine the most effective mix of resources and fuel
treatments at various budget levels. The FPA does not provide one
finite answer but instead evaluates the changes in modeled
effectiveness for each investment level analyzed.
See comment 1.
We believe that one of FPA's greatest strengths is the capability to
evaluate alternative investment strategies and identify options that
best reduce lire losses, improve ecology conditions. and increase cost
efficiency. This ability to array alternatives Mal display risks and
trade-offs at the local and national levels provide a robust basis for
comparing potential performance and cost effectiveness will optimize
the use of resources. We believe this is a much more realistic approach
then the single-dimensional optimization approach for the most cost
effective mix and location of firefighting asset for a given budget.
Our Interagency Science Team has recommended the more analytical
approach that we are now developing.
With respect to the system's ability to evaluate fuel reduction
investments over time, it is important to note that our development and
science teams are analyzing temporal modeling approaches for fuel
treatments, which may be released later this year. We agree with GAO
that there is potential for a more comprehensive analysis of vegetation
and fuel treatments.
See comment 2.
Our comments on the recommendations are as follows:
Recommendation 1” The Secretaries of Agriculture and the Interior
develop a strategic plan for the continued development of FPA, which
would (1) include an evaluation of the strengths and weaknesses of FPA,
(2) identify ways to improve the model to better men its intended
objectives, and (3) identify the steps the agencies plan to take to
improve FPA and expected time frames and associated budget needs for
completing these steps.
Response” The results of the reviews of FPA that were conducted by the
science team and the management team support the current course of
action. The system is expected to provide results that can be used in
2009 and to be available in time to develop the fiscal year 2011 budget
request. The fire community is anxiously awaiting access to this system
that will give them new tools to use in decision-making and improve
their ability to allocate resources in a cost effective manner.
Development of a strategic plan will further delay system deployment
and result in increased costs. We would prefer to continue forward with
the planned, staged approach that allows for adaptive use and
modification. We believe that this approach will allow us to ensure
that the system optimizes capabilities for evaluation of different
mixes and locations of firefighting assets, protection of valued
resources, modeling investments, and analyzing trade-offs in the
allocation of firefighting assets. In addition, our planned external
peer review is expected to further assist in identifying improvements
to the system.
See comment 3.
Recommendation 2” The Secretaries of Agriculture and the Interior
report annually to Congress on (I) their progress in completing the
steps outlined in the strategic plan for the continued development of
FPA, and (2) FPA's current ability to meet each of the key objectives
initially established for it. Response” The Department concurs with
GAO's recommendation to report annually to Congress on progress and
achievement of objectives.
Recommendation 3” Increase agency transparency in using FPA to develop
their budget requests and allocate funds by reporting to Congress on
FPA's role in budget development and allocation process.
Response” The Department concurs with GAO's recommendation and will
ensure that the results and use of FPA information is clearly depicted
in the budget request to Congress.
Recommendation 4” Submit the FPA model to external peer review.
Response”The Department concurs with GAO's recommendation and is
planning an external peer review.
We have closely coordinated our response with the U.S. Forest Service
and hold concurrent views. If you have any questions or concerns,
please contact Barbara Loving at the Office of Wildland Fire
Coordination at 202-606-3108.
We look forward to working with GAO in the future.
Sincerely,
Signed by:
James E. Cason:
Enclosure:
Enclosure not printed.
The following are GAO's comments on the Department of the Interior's
letter dated November 5, 2008.
GAO Comments:
1. As Interior's comment letter indicates, we have had extensive
discussions with agency officials on FPA's ability to identify the most
cost-effective mix of firefighting assets without resolving our
differing points of view. Interior commented that FPA will allow the
department to meet FPA's cost-effectiveness objective by evaluating
alternative investment strategies and identifying options that best
reduce fire losses, improve ecological conditions, and increase cost
efficiencies and that the agencies' current approach is much more
realistic than the approach taken initially. As we describe in our
report, FPA compares only a limited number of mixes of firefighting
assets and firefighting strategies, and further, the alternatives it
evaluates are likely to reflect only minor variations in budget levels.
Given this structure, we continue to believe that FPA is unlikely to
allow the agencies to identify the most cost-effective location and mix
of assets and strategies--an objective the agencies themselves
established in their 2001 report. And as noted in our response to the
Forest Service's comments, we did not compare the agencies' current
approach with their initial approach, nor do we conclude whether one
approach is more suitable or realistic than the other. Rather, in
accordance with the objectives of our review, we simply evaluated the
extent to which FPA as it is currently being developed is likely to
meet the objectives originally established for it. While we are not
altering our conclusion that FPA's current approach will likely keep
the agencies from identifying the most cost-effective solution, we are
modifying our first recommendation to suggest that the agencies clarify
which of FPA's original objectives they believe are no longer
appropriate and why. See comment 3 below.
2. Interior commented that the agencies are continuing to analyze how
FPA evaluates fuel reduction investments over time and may begin using
a new modeling approach later in 2008. It is not clear from Interior's
letter whether it believes the new approach will allow the agencies to
meet FPA's original objective of modeling the effects over time of
differing strategies for responding to wildland fires and treating
lands to reduce hazardous fuels. Our review of the limited
documentation describing this approach suggests that this approach is
unlikely to allow FPA to fully meet this key objective.
3. Interior stated that developing a strategic plan for the continued
development of FPA, as we are recommending, would further delay
deployment and increase the cost of FPA. We recognize it is important
for the agencies to continue to develop FPA, and we are not suggesting
that the agencies delay implementing FPA until they have developed the
strategic plan we recommend. On the contrary, we believe that such a
plan can be developed concurrently with implementation and that the
agencies may benefit from incorporating lessons learned during FPA's
early use into the plan. In any event, our review raised questions
about FPA's ability to meet certain of its key objectives, even with
the changes the agencies are considering making to FPA--and because of
FPA's importance, and the concerns about its development, we believe it
is important for the agencies to develop a single document that
directly and transparently evaluates FPA's ability to meet its original
objectives and identifies ways to improve FPA to better meet those
objectives. Given the agencies' comments, however, we modified our
recommendation to suggest that the agencies use this plan not only to
identify ways to improve the model to better meet FPA's original
objectives, but also to clearly state whether they believe any of the
original objectives are no longer appropriate, and why, in order to
ensure that Congress and other interested parties are fully informed
about what they can reasonably expect from FPA.
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Robin M. Nazzaro, (202) 512-3841 or nazzaror@gao.gov:
Staff Acknowledgments:
In addition to the contact person named above, Steve Gaty, Assistant
Director; David P. Bixler; Ellen W. Chu; Jonathan Dent; Richard
Johnson; Chester Joy; Mehrzad Nadji; Jacqueline Nowicki; and Dae Park
made key contributions to this report.
[End of section]
Footnotes:
[1] Departments of Agriculture and the Interior, Federal Wildland Fire
Management Policy and Program Review (Washington, D.C., December 1995).
This policy was subsequently reaffirmed and updated in 2001:
Departments of the Interior, Agriculture, Energy, Defense, and
Commerce; Environmental Protection Agency; Federal Emergency Management
Agency; and National Association of State Foresters, Review and Update
of the 1995 Federal Wildland Fire Management Policy (Washington, D.C.,
January 2001).
[2] Forest Service and Department of the Interior, Developing an
Interagency, Landscape-scale Fire Planning Analysis and Budget Tool
(Washington, D.C., November 2001).
[3] Departments of Agriculture and the Interior, Fire Program Analysis:
Scientific Review Team Report (Washington, D.C., January 2006); and
Management Review Team Report of the Fire Program Analysis (FPA)
Preparedness Module (Washington, D.C., March 2006).
[4] Together, preparedness, suppression, and fuel reduction make up
approximately 80 percent of the agencies' wildland fire management
budgets. Other federal wildland fire program components include
financial assistance to state foresters for fire management activities,
research and development, and rehabilitating burned federal lands.
[5] Federal and nonfederal agencies have established a framework to
share the costs of responding to fires that threaten both federal and
nonfederal resources. See GAO, Wildland Fire Suppression: Lack of Clear
Guidance Raises Concerns about Cost Sharing between Federal and
Nonfederal Entities, GAO-06-570 (Washington, D.C.: May 30, 2006).
[6] The number of planning units established by the agencies has
fluctuated over the course of FPA development. In this report, we refer
to the 139 planning units in existence at the time our review ended but
recognize that the actual number at any particular time may differ.
[7] Peer review is a process by which scientific research or technical
projects are subject to an independent assessment by scientists not
involved with the project who have knowledge and expertise comparable
to that of the scientists whose work they review.
[8] The Wildland Fire Leadership Council consists of senior Agriculture
and Interior officials, including the Agriculture Undersecretary for
Natural Resources and Environment; the Interior Assistant Secretary for
Policy, Management, and Budget; and the heads of the five federal
firefighting agencies. Other members include representatives of the
Intertribal Timber Council, the National Association of State
Foresters, and the Western Governors' Association, and a local fire
department chief.
[9] The size threshold for fires to be considered contained while small
is to vary according to criteria established by officials in each
planning unit, considering the circumstances under which they typically
consider a fire in their area "escaped" and then request additional
firefighting assets to help suppress it. This measure also includes the
number of fires the model predicts would be averted because of the
agencies' efforts to prevent human-caused fires.
[10] The program staff who helped develop the old approach told us that
they had recognized the small number of potential fire scenarios in
that approach limited its capabilities and that they were considering
how to improve it, but the agencies determined that a new modeling
approach was needed before they could make improvements.
[11] The agencies have provided brief updates on the status of FPA to
Congress in their annual budget justifications, and have provided
periodic briefings to congressional committee and OMB staff.
[12] These figures include the cost of developing FPA and operating and
maintaining it through fiscal year 2010.
[13] Similarly, the project manager said that the decrease in estimated
cost for the second phase from 2005 to 2006 was also due to a better
understanding of the project's scope.
[14] FPA is also to help the agencies model their investment in
preventing fires. The agencies carry out activities, such as increased
law enforcement patrols and public education programs, intended to
reduce the number of human-caused wildland fires. FPA is to predict the
number of fires that would have started if not for the agencies'
prevention activities.
[15] LANDFIRE is a geospatial data and modeling system designed to
assist the agencies in identifying the extent, severity, and location
of wildland fire threats to the nation's communities and ecosystems. At
the time of our review, LANDFIRE data were not available for the
eastern United States or for Alaska and Hawaii. FPA officials said that
until LANDFIRE data are available nationwide, they are using other
available data to provide similar information. FPA officials expect
that LANDFIRE data will be available nationwide by 2009.
[16] We have previously reported limitations of the model the agencies
use to predict suppression costs. See GAO, Wildland Fire Management:
Lack of Clear Goals or a Strategy Hinders Federal Agencies' Efforts to
Contain the Costs of Fighting Fires, GAO-07-655 (Washington, D.C.: June
1, 2007).
[17] In some cases, federal firefighting assets are also available to
respond to fires on nonfederal land.
[18] We have previously reported on the status of the agencies'
development of these plans. See GAO, Wildland Fire Management: Update
on Federal Agency Efforts to Develop a Cohesive Strategy to Address
Wildland Fire Threats, GAO-06-671R (Washington, D.C.: May 1, 2006).
[19] The interagency science team in 2006 proposed an option for
developing FPA that might have helped the agencies to better achieve
this objective, but the Wildland Fire Leadership Council did not
approve this option out of concern that the agencies would be unable to
complete it within the time and budget available. Interagency science
team members told us they could, if directed, continue to develop that
option and incorporate it into FPA later.
[20] GAO, Western National Forests: A Cohesive Strategy Is Needed to
Address Catastrophic Wildfire Threats, GAO/RCED-99-65 (Washington,
D.C.: Apr. 2, 1999); Wildland Fire Management: Important Progress Has
Been Made, but Challenges Remain to Completing a Cohesive Strategy, GAO-
05-147 (Washington, D.C.: Jan. 14, 2005); and GAO-06-671R.
[21] In 2008, however, we reported that the agencies had begun
retreating from their commitment to develop a cohesive strategy. See
GAO, Wildland Fire Management: Federal Agencies Lack Key Long-and Short-
Term Management Strategies for Using Program Funds Effectively, GAO-08-
433T (Washington, D.C.: Feb. 12, 2008).
[22] For information on the agencies' approaches to allocating fuel
reduction funds, see GAO, Wildland Fire Management: Better Information
and a Systematic Process Could Improve Agencies' Approach to Allocating
Fuel Reduction Funds and Selecting Projects, GAO-07-1168 (Washington,
D.C.: Sept. 28, 2007).
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
