Particulate Matter
EPA Has Started to Address the National Academies' Recommendations on Estimating Health Benefits, but More Progress Is Needed Gao ID: GAO-06-780 July 17, 2006A large body of scientific evidence links exposure to particulate matter--a widespread form of air pollution--to serious health problems, including asthma and premature death. Under the Clean Air Act, the Environmental Protection Agency (EPA) periodically reviews the appropriate air quality level at which to set national standards to protect the public against the health effects of particulate matter. EPA proposed revisions to these standards in January 2006 and issued a draft regulatory impact analysis of the revisions' expected costs and benefits. The estimated benefits of air pollution regulations have been controversial in the past. A 2002 National Academies report generally supported EPA's approach but made 34 recommendations to improve how EPA implements its approach. GAO was asked to determine whether and how EPA applied the Academies' recommendations in its estimates of the health benefits expected from the January 2006 proposed revisions to the particulate matter standards. GAO examined the draft analysis, met with EPA officials, and interviewed members of the National Academies' committee. In providing technical comments on the report, EPA officials said it was fair and balanced and noted the agency's progress in addressing recommendations via research and development and other analyses.
EPA has begun to change the way it conducts and presents its analyses of health benefits in response to recommendations from the National Academies. Specifically, EPA applied, at least in part, 22--or about two-thirds--of the Academies' recommendations to its health benefit analysis of proposed revisions to particulate matter standards. For example, in response to some of the recommendations, EPA took steps toward conducting a more rigorous assessment of uncertainty by, for instance, evaluating how benefits could change under different assumptions and discussing sources of uncertainty not included in the benefit estimates. In one case, EPA applied an alternative technique, called expert elicitation, for evaluating uncertainty by systematically gathering expert opinion about the uncertainty underlying the causal link between exposure to particulate matter and premature death. Consistent with the National Academies' recommendation to assess uncertainty by developing ranges of estimates and specifying the likelihood of attaining them, EPA used expert elicitation to develop ranges of reductions in premature death expected from the proposed revisions. EPA officials said that ongoing research and development efforts will allow the agency to gradually achieve more progress in applying the recommendations. We note that robust uncertainty analysis is important because estimates of health benefits can be highly uncertain, as the draft regulatory impact analysis for particulate matter illustrates. EPA viewed the estimates in this analysis as so uncertain that it chose not to present them in the executive summary. For various reasons, EPA has not applied the remaining 12 recommendations to the analysis, such as the recommendation to evaluate the impact of using the simplifying assumption that each component of particulate matter is equally toxic. EPA officials viewed most of these recommendations as relevant to its health benefit analyses and, citing the need for additional research and development, emphasized the agency's commitment to continue to respond to the recommendations. For example, EPA did not believe that the state of scientific knowledge on the relative toxicity of particulate matter components was sufficiently developed to include in the January 2006 regulatory impact analysis, and the agency is currently sponsoring research on this issue. In addition, a senior EPA official said that insufficient resources impeded the agency's progress in applying the recommendations, citing, in particular, the limited availability of skilled staff, time, and other resources to conduct the required analyses and research and development. EPA officials also said that some of the recommendations the agency did not apply to the draft analysis, such as one calling for a summary table describing key analytical information to enhance transparency, will be applied to the analysis supporting the final rule. To the extent that EPA continues to make progress addressing the Academies' recommendations, decision makers and the public will be better able to evaluate the basis for EPA's air regulations.
GAO-06-780, Particulate Matter: EPA Has Started to Address the National Academies' Recommendations on Estimating Health Benefits, but More Progress Is Needed
This is the accessible text file for GAO report number GAO-06-780
entitled 'Particulate Matter: EPA Has Started to Address the National
Academies' Recommendations on Estimating Health Benefits, but More
Progress Is Needed' which was released on July 19, 2006.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
July 2006:
Particulate Matter:
EPA Has Started to Address the National Academies' Recommendations on
Estimating Health Benefits, but More Progress Is Needed:
GAO-06-780:
GAO Highlights:
Highlights of GAO-06-780, a report to congressional requesters
Why GAO Did This Study:
A large body of scientific evidence links exposure to particulate
matter”a widespread form of air pollution”to serious health problems,
including asthma and premature death. Under the Clean Air Act, the
Environmental Protection Agency (EPA) periodically reviews the
appropriate air quality level at which to set national standards to
protect the public against the health effects of particulate matter.
EPA proposed revisions to these standards in January 2006 and issued a
draft regulatory impact analysis of the revisions‘ expected costs and
benefits.
The estimated benefits of air pollution regulations have been
controversial in the past. A 2002 National Academies report generally
supported EPA‘s approach but made 34 recommendations to improve how EPA
implements its approach. GAO was asked to determine whether and how EPA
applied the Academies‘ recommendations in its estimates of the health
benefits expected from the January 2006 proposed revisions to the
particulate matter standards. GAO examined the draft analysis, met with
EPA officials, and interviewed members of the National Academies‘
committee. In providing technical comments on the report, EPA officials
said it was fair and balanced and noted the agency‘s progress in
addressing recommendations via research and development and other
analyses.
What GAO Found:
EPA has begun to change the way it conducts and presents its analyses
of health benefits in response to recommendations from the National
Academies. Specifically, EPA applied, at least in part, 22”or about two-
thirds”of the Academies‘ recommendations to its health benefit analysis
of proposed revisions to particulate matter standards. For example, in
response to some of the recommendations, EPA took steps toward
conducting a more rigorous assessment of uncertainty by, for instance,
evaluating how benefits could change under different assumptions and
discussing sources of uncertainty not included in the benefit
estimates. In one case, EPA applied an alternative technique, called
expert elicitation, for evaluating uncertainty by systematically
gathering expert opinion about the uncertainty underlying the causal
link between exposure to particulate matter and premature death.
Consistent with the National Academies‘ recommendation to assess
uncertainty by developing ranges of estimates and specifying the
likelihood of attaining them, EPA used expert elicitation to develop
ranges of reductions in premature death expected from the proposed
revisions. EPA officials said that ongoing research and development
efforts will allow the agency to gradually achieve more progress in
applying the recommendations. We note that robust uncertainty analysis
is important because estimates of health benefits can be highly
uncertain, as the draft regulatory impact analysis for particulate
matter illustrates. EPA viewed the estimates in this analysis as so
uncertain that it chose not to present them in the executive summary.
For various reasons, EPA has not applied the remaining 12
recommendations to the analysis, such as the recommendation to evaluate
the impact of using the simplifying assumption that each component of
particulate matter is equally toxic. EPA officials viewed most of these
recommendations as relevant to its health benefit analyses and, citing
the need for additional research and development, emphasized the
agency‘s commitment to continue to respond to the recommendations. For
example, EPA did not believe that the state of scientific knowledge on
the relative toxicity of particulate matter components was sufficiently
developed to include in the January 2006 regulatory impact analysis,
and the agency is currently sponsoring research on this issue. In
addition, a senior EPA official said that insufficient resources
impeded the agency‘s progress in applying the recommendations, citing,
in particular, the limited availability of skilled staff, time, and
other resources to conduct the required analyses and research and
development. EPA officials also said that some of the recommendations
the agency did not apply to the draft analysis, such as one calling for
a summary table describing key analytical information to enhance
transparency, will be applied to the analysis supporting the final
rule. To the extent that EPA continues to make progress addressing the
Academies‘ recommendations, decision makers and the public will be
better able to evaluate the basis for EPA‘s air regulations.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-780].
To view the full product, including the scope and methodology, click on
the link above.
For more information, contact John B. Stephenson at (202) 512-3841 or
stephensonj@gao.gov.
[End of Section]
Contents:
Letter:
Results in Brief:
Background:
EPA Is in the Process of Addressing Many of the Academies'
Recommendations:
Concluding Observations:
Agency Comments:
Appendixes:
Appendix I: Scope and Methodology:
Appendix II: Recommendations Applied or Partially Applied to the Draft
Particulate Matter Regulatory Impact Analysis:
Appendix III: Recommendations Not Applied to the Draft Particulate
Matter Regulatory Impact Analysis:
Appendix IV: GAO Contact and Staff Acknowledgments:
Tables Tables:
Table 1: Recommendations Applied or Partially Applied to the Draft
Particulate Matter Regulatory Impact Analysis:
Table 2: Recommendations Not Applied to the Draft Particulate Matter
Regulatory Impact Analysis:
Abbreviations:
Academies: National Academies:
ACS: American Cancer Society:
Council : Advisory Council on Clean Air Compliance:
EPA: Environmental Protection Agency:
NAAQS: National Ambient Air Quality Standards:
OMB: Office of Management and Budget:
PM: particulate matter:
RIA: regulatory impact analysis:
SAB: Science Advisory Board:
July 14, 2006:
The Honorable James M. Inhofe:
Chairman:
Committee on Environment and Public Works:
United States Senate:
The Honorable George Voinovich:
Chairman:
Subcommittee on Clean Air, Climate Change, and Nuclear Safety:
Committee on Environment and Public Works:
United States Senate:
A large body of scientific evidence links exposure to particulate
matter--a ubiquitous form of air pollution commonly referred to as
soot--to serious health problems, including asthma, chronic bronchitis,
heart attack, and premature death. The many sources releasing
particulate matter into the air include cars, trucks, power plants,
industrial processes, forest fires, and waste incinerators. In 1971,
the Environmental Protection Agency (EPA) first established national
air quality standards to protect the public against the health effects
of particulate matter, one of the six widespread criteria pollutants
considered harmful to public health. Under the Clean Air Act, EPA
determines the appropriate level at which to set national air quality
standards, and the states must develop programs to achieve and maintain
compliance with them. Further, EPA must review the standards every 5
years to determine whether they adequately protect human health and
welfare, given the latest scientific information available, and revise
them if they do not.
In January 2006, after its most recent review of the national air
quality particulate matter standards, EPA proposed revisions to the
standards and issued a draft regulatory impact analysis. Overall, the
draft regulatory impact analysis discussed the scope and magnitude of
the particulate matter problem, the likely benefits of the proposed
revisions for public health and the environment, and the expected costs
of implementing the standards. Regarding public health, the analysis
presented estimates of expected health benefits for the particulate
matter revisions in five major urban areas, including reductions in the
number of premature deaths and emergency room visits for asthma. Among
the changes EPA said it plans to make in its final regulatory impact
analysis is providing national estimates of expected health benefits.
EPA is required under a court order to issue the final rule on
particulate matter standards by September 2006.[Footnote 1]
In 2000, at the direction of the Senate Appropriations Committee, EPA
asked the National Academies (Academies) to evaluate EPA's methodology
for estimating the health benefits of proposed air pollution
regulations.[Footnote 2] According to the National Academies, these
estimates have often been controversial, and the methods EPA has used
to prepare them have been questioned. For example, some observers, such
as researchers and industry groups, have expressed concerns that EPA
does not adequately factor uncertainty into its estimates of health
benefits. Some level of uncertainty is unavoidable, in part because the
scientific information used to develop estimates, such as the inventory
of particulate matter emissions, will never be perfect or complete.
However, according to some observers, EPA's estimates of benefits
appear more definitive than they really are because the agency does not
adequately account for uncertainty in its analyses or in its reporting
of health benefit estimates.
The National Academies' 2002 report on this subject generally supported
EPA's approach to estimating health benefits but, nevertheless, made 34
detailed recommendations to improve how EPA implements its
approach.[Footnote 3] Overall, these recommendations focus on
conducting more rigorous assessments of uncertainty, increasing the
transparency of how EPA estimates benefits, conducting more detailed
analyses of exposure, and estimating the benefits of each regulatory
option under consideration. Many of the recommendations include
qualifying language indicating that it is reasonable to expect that
they can be applied in stages, over time; and a number of the
recommendations are interrelated and, in some cases, overlapping.
You asked us to determine whether and how EPA applied the National
Academies' recommendations in its estimates of the health benefits
expected from the January 2006 proposed revisions to the particulate
matter standards. To respond to this objective, we reviewed EPA's draft
regulatory impact analysis presenting the costs and benefits of the
proposed rule and met with senior officials from EPA's Office of Air
and Radiation, which was responsible for developing the proposed rule
and analyzing its costs and benefits, and with officials from EPA's
Office of Policy, Economics, and Innovation. We also reviewed EPA's and
the Office of Management and Budget's (OMB) guidance on conducting
economic analyses, prior GAO reports on EPA's regulatory impact
analyses, and other relevant reports. As requested, our work addressed
the application of the National Academies' recommendations to EPA's
draft regulatory impact analysis supporting the 2006 proposed
particulate matter rule; thus, we did not examine how EPA applied the
recommendations to other recent air rules. Our work focused on broadly
characterizing EPA's progress toward applying the recommendations; we
did not evaluate the effectiveness and quality of the scientific and
technical actions the agency has taken to apply the recommendations.
See appendix I for a more detailed description of the scope and
methodology of our review. We performed our work from January 2006 to
July 2006 in accordance with generally accepted government auditing
standards.
Results in Brief:
EPA has begun to change the way it conducts and presents its analyses
of health benefits in response to the National Academies'
recommendations, which focused on conducting more rigorous assessments
of uncertainty, increasing the transparency of how EPA estimates
benefits, conducting more detailed analyses of exposure, and estimating
the benefits of each regulatory option under consideration. EPA
applied, at least in part, about two-thirds of the recommendations to
its health benefit analysis. Specifically, of the 34 recommendations,
EPA applied 8 and partially applied 14. For example, EPA responded to
some of the recommendations by taking steps to conduct a more rigorous
assessment of uncertainty by, for instance, evaluating how benefits
might change given alternative assumptions and discussing sources of
uncertainty not included in the benefit estimates. More specifically,
EPA applied an alternative technique for evaluating one important
source of uncertainty in its analysis--the uncertainty underlying the
causal link between exposure to particulate matter and premature death.
Consistent with the National Academies' recommendation to assess
uncertainty by developing ranges of estimates of benefits and
specifying the likelihood of attaining that level of benefits, EPA
systematically gathered expert opinions about this link--through a
process called expert elicitation--and developed ranges reflecting the
experts' confidence in attaining reductions in premature death expected
from the proposed revisions. However, the health benefit analysis does
not similarly assess how the benefit estimates would vary in light of
other key uncertainties as the Academies had recommended. Consequently,
this represents a partial application of the recommendation. Agency
officials told us that ongoing research and development efforts will
allow EPA to gradually achieve more progress in applying this and other
recommendations to future analyses.
For various reasons, EPA did not apply the remaining 12 recommendations
to the analysis, such as the recommendation to evaluate the impact of
using the assumption that the components of particulate matter are
equally toxic. EPA officials viewed most of these recommendations as
relevant to its health benefit analyses, but noted that the agency was
not ready to apply specific recommendations because of, among other
things, the need to overcome technical challenges stemming from
limitations in the state of available science. These officials
emphasized the agency's commitment to continue to respond to the
recommendations. For example, EPA did not believe that the state of
scientific knowledge on the relative toxicity of particulate matter
components was sufficiently developed to include it in the January 2006
regulatory impact analysis, and the agency is sponsoring research on
this issue. In addition, according to a senior EPA official,
insufficient resources have impeded the agency's progress in applying
the recommendations, including the limited availability of skilled
staff, time, and other resources to conduct and oversee the required
analyses and research and development. Finally, EPA officials stated
that some of the recommendations the agency did not apply to the draft
analysis, such as one calling for a summary table describing key
analytical information to enhance transparency, will be applied to the
regulatory impact analysis supporting the final rule. To the extent
that EPA continues to make progress addressing the Academies'
recommendations, decision makers and the public will be able to better
evaluate the basis for EPA's air regulations.
We provided a draft of this report to EPA for review. EPA provided
technical comments that we incorporated, as appropriate. Officials from
EPA's Office of Air and Radiation noted in their technical comments
that the report provides a fair and balanced representation of the
agency's application of the recommendations to the particulate matter
regulatory impact analysis and cited EPA's progress in meeting the
National Academies' recommendations through other analyses of air
programs and through research and development efforts.
Background:
EPA is required by the Clean Air Act to conduct reviews of the National
Ambient Air Quality Standards (NAAQS) for the six criteria pollutants,
including particulate matter, every 5 years. The overarching purpose of
such reviews is to determine whether the current standards are
sufficient to protect public health and welfare at large, with an
adequate margin of safety, given the latest scientific information
available at the time of the review. Major steps in the NAAQS process
include the following:
* developing a criteria document that synthesizes new research on
health and environmental effects;
* preparing a staff paper that assesses the policy implications of the
scientific information in the criteria document, which also discusses
possible ranges for air quality standards; and:
* determining whether and how EPA should revise the NAAQS.
If EPA decides to revise the NAAQS, the agency proposes the changes in
the Federal Register. As part of the federal rule-making process, EPA
is to comply with Executive Order 12866, which directs federal agencies
to analyze the costs and benefits of proposed and final rules expected
to affect the economy by $100 million or more per year.[Footnote 4] In
September 2003, the Office of Management and Budget (OMB) issued its
Circular A-4, which presents guidance and best practices and states
that agencies should analyze the costs and benefits in accordance with
the principles of full disclosure and transparency. Further, in cases
such as the particulate matter rule, where expected economic impacts
exceed $1 billion annually, Circular A-4 also states that agencies
should conduct a comprehensive assessment of key uncertainties in their
analyses of costs and benefits, which EPA also refers to as regulatory
impact analyses.[Footnote 5] EPA's January 2006 regulatory impact
analysis presents estimates of the costs and benefits for the proposed
particulate matter rule.
The focus of the National Academies' 2002 report was on how EPA
estimates the health benefits of its proposed air regulations. To
develop such estimates, EPA conducts analyses to quantify the expected
changes in the number of deaths and illnesses that are likely to result
from proposed regulations. The regulatory impact analyses also estimate
the costs associated with implementing proposed air regulations,
although, under the Clean Air Act, EPA is not permitted to consider
costs in setting health-based standards for the criteria air
pollutants, such as particulate matter.
Soon after the National Academies issued its report in 2002, EPA staff
identified key recommendations and developed a strategy, in
consultation with OMB, to apply some of the recommendations to benefit
analyses for air pollution regulations under consideration at the time.
EPA roughly approximated the time and resource requirements to respond
to the recommendations, identifying those the agency could address
within 2 or 3 years and those that would take longer. According to EPA
officials, the agency focused primarily on the numerous recommendations
related to analyzing uncertainty.
Both the National Academies' report and the OMB guidance emphasize the
need for agencies to account for uncertainties and to maintain
transparency in the course of conducting benefit analyses. Identifying
and accounting for uncertainties in these analyses can help decision
makers evaluate the likelihood that certain regulatory decisions will
achieve the estimated benefits. Transparency is important because it
enables the public and relevant decision makers to see clearly how EPA
arrived at its estimates and conclusions. In prior work on regulatory
impact analyses, we have found shortcomings in EPA's analyses of
uncertainty and the information the agency provides with its estimates
of costs and benefits.[Footnote 6]
EPA Is in the Process of Addressing Many of the Academies'
Recommendations:
EPA applied--either wholly or in part--approximately two-thirds of the
Academies' recommendations to its January 2006 regulatory impact
analysis and continues to address the recommendations through ongoing
research and development. The January 2006 regulatory impact analysis
demonstrated progress toward an expanded analysis of uncertainty and
consideration of different assumptions. EPA officials cited time and
resource constraints, as well as the need to mitigate complex technical
challenges, as the basis for not applying other recommendations.
According to EPA officials, the agency did not apply some of the more
complex recommendations because it had not achieved sufficient progress
in the research and development projects under way.
EPA Applied, at Least in Part, about Two-thirds of the Recommendations
to Its Particulate Matter Health Benefit Analysis in the Proposed Rule:
The January 2006 regulatory impact analysis on particulate matter
represents a snapshot of an ongoing EPA effort to respond to the
National Academies' recommendations on developing estimates of health
benefits for air pollution regulations. Specifically, the agency
applied, at least in part, approximately two-thirds of the
recommendations--8 were applied and 14 were partially applied--by
taking steps toward conducting a more rigorous assessment of
uncertainty for proposed air pollution regulations by, for example,
evaluating the different assumptions about the link between human
exposure to particulate matter and health effects and discussing
sources of uncertainty not included in the benefit estimates. According
to EPA officials, the agency focused much of its time and resources on
the recommendations related to uncertainty. In particular, one
overarching recommendation suggests that EPA take steps toward
conducting a formal, comprehensive uncertainty analysis--the systematic
application of mathematical techniques, such as Monte Carlo simulation--
and include the uncertainty analysis in the regulatory impact analysis
to provide a "more realistic depiction of the overall uncertainty" in
EPA's estimates of the benefits.[Footnote 7] A number of the other
recommendations regarding uncertainty are aimed at EPA's developing the
information and methodologies needed to carry out a comprehensive
uncertainty analysis.
Overall, the uncertainty recommendations suggest that EPA should
determine (1) which sources of uncertainties have the greatest effect
on benefit estimates and (2) the degree to which the uncertainties
affect the estimates by specifying a range of estimates and the
likelihood of attaining them. In response, EPA devoted significant
resources to applying an alternative technique called expert
elicitation in a multiphased pilot project. The pilot project was
designed to systematically obtain expert advice to begin to better
incorporate in its health benefit analysis the uncertainty underlying
the causal link between exposure to particulate matter and premature
death. EPA used the expert elicitation process to help it more
definitively evaluate the uncertainty associated with estimated
reductions in premature death--estimates that composed 85 percent to 95
percent of EPA's total health benefit estimates for air pollution
regulations in the past 5 years, according to the agency.[Footnote 8]
EPA developed a range of expected reductions in death rates based on
expert opinion systematically gathered in its pilot expert elicitation
project and provided the results of this supplemental analysis in an
appendix to the regulatory impact analysis.[Footnote 9] However, the
National Academies had recommended that EPA merge such supplemental
analyses into the main benefit analysis.
Moreover, the Academies recommended that EPA's main benefit analysis
reflect how the benefit estimates would vary in light of uncertainties.
In addition to the uncertainty underlying the causal link between
exposure and premature death that EPA analyzed, other key uncertainties
can influence the estimates. For example, there is uncertainty about
the effects of the age and health status of people exposed to
particulate matter, the varying composition of particulate matter, and
the measurements of actual exposure to particulate matter. EPA's health
benefit analysis, however, does not account for these key uncertainties
by specifying a range of estimates and the likelihood of attaining
them, similar to estimates derived from the expert elicitation
addressing causal uncertainty. For these reasons, EPA's responses
reflect a partial application of the Academies' recommendation.
In addition, the Academies recommended that EPA both continue to
conduct sensitivity analyses on sources of uncertainty and expand these
analyses. In the particulate matter regulatory impact analysis, EPA
included a new sensitivity analysis regarding assumptions about
thresholds, or levels below which those exposed to particulate matter
are not at risk of experiencing harmful effects. EPA has assumed no
threshold level exists--that is, any exposure poses potential health
risks.[Footnote 10] Some experts have suggested that different
thresholds may exist and the National Academies recommended that EPA
determine how changing its assumption--that no threshold exists--would
influence the estimates. The sensitivity analysis EPA provided in the
regulatory impact analysis examined how its estimates of expected
health benefits would change assuming varying thresholds.
Another recommendation that EPA is researching and partially applied to
the draft regulatory impact analysis concerns alternative assumptions
about cessation lags--the time between reductions in exposure to
particulate matter and the health response. The National Academies made
several recommendations on this topic, including one that EPA
incorporate alternative assumptions about lags into a formal
uncertainty analysis to estimate benefits that account for the
likelihood of different lag durations. In response, EPA has sought
advice from its Advisory Council on Clean Air Compliance Analysis on
how to address this recommendation and has conducted a series of
sensitivity analyses related to cessation lags. EPA is also funding
research to explore ways to address lag effects in its uncertainty
analysis. According to an EPA official, specifying the probability of
different lag effects is computationally complex, and the agency is
working to resolve this challenge.
In response to another recommendation by the National Academies, EPA
identified some of the sources of uncertainty that are not reflected in
its benefit estimates. For example, EPA's regulatory impact analysis
disclosed that its benefit estimates do not reflect the uncertainty
associated with future year projections of particulate matter
emissions. EPA presented a qualitative description about emissions
uncertainty, elaborating on technical reasons--such as the limited
information about the effectiveness of particulate matter control
programs--why the analysis likely underestimates future emissions
levels. EPA also applied the Academies' recommendation on the
presentation of uncertainty, which encouraged the agency to present the
results of its health benefit analyses in ways that convey the
estimated benefits more realistically by, for example, placing less
emphasis on single estimates and rounding the numbers. EPA's regulatory
impact analysis presented ranges for some of the benefit estimates.
Also, EPA sought to convey the overall uncertainty of its benefit
estimates in a qualitative manner by clearly stating that decision
makers and the public should not place significant weight on the
quantified benefit estimates in the regulatory impact analysis because
of data limitations and uncertainties.
Another example of EPA's response to the National Academies'
recommendations involves exploring the various regulatory choices
available to decision makers. The Academies recommended that EPA
estimate the health benefits representing the full range of regulatory
choices available to decision makers. In the particulate matter
analysis, EPA presented health benefits expected under several
regulatory options targeting fine particulate matter. Citing a lack of
data and tools needed to conduct an accurate analysis, EPA did not
estimate the benefits expected under the proposed regulatory options
for coarse particulate matter but, consistent with the National
Academies' recommendation, presented its rationale for not doing so.
Overall, we considered this a partial application of the
recommendation. (See app. II for more detail on the recommendations
that EPA has applied or partially applied to the draft particulate
matter regulatory impact analysis.)
EPA Plans to Address Some of the Remaining Recommendations in the Final
Rule and Has Research and Development Under Way to Address Others:
EPA did not apply the remaining 12 recommendations to the analysis for
various reasons. While EPA applied some recommendations--either wholly
or in part--that require additional studies, methodologies, or data to
its particulate matter analysis, the agency had not made sufficient
progress in addressing others and therefore did not apply them to the
analysis. EPA officials viewed most of these recommendations as
relevant to its health benefit analyses and, citing the need for
additional research and development, emphasized the agency's commitment
to continue to respond to the recommendations. According to a senior
EPA official, insufficient resources impeded the agency's progress in
applying the recommendations. This official cited limited availability
of skilled staff, time, and other resources to conduct the required
analyses and research and development. According to EPA, some of the
more complex, long-term recommendations include the following: relying
less on simplifying assumptions, such as the assumption that the
various components of particulate matter have equal toxicity;[Footnote
11] conducting a formal assessment of the uncertainty of particulate
matter emissions; and assessing the expected reduction of any harmful
effects other than air pollution or human health problems.
For example, EPA is in the process of responding to a recommendation
involving the relative toxicity of components of particulate matter, an
emerging area of research that has the potential to influence EPA's
regulatory decisions in the future.[Footnote 12] Specifically, the
agency could, hypothetically, refine national air quality standards to
address the potentially varying health consequences associated with
different components of particulate matter. The National Academies
recommended that EPA strengthen its benefit analyses by evaluating a
range of alternative assumptions regarding relative toxicity and
incorporate these assumptions into sensitivity or uncertainty analyses
as more data become available.[Footnote 13] EPA did not believe the
state of scientific knowledge on relative toxicity was sufficiently
developed at the time it prepared the draft regulatory impact analysis
to include this kind of analysis.[Footnote 14] However, EPA is
sponsoring research on this issue. For example, EPA is supporting long-
term research on the relative toxicity of particulate matter components
being conducted by EPA's intramural research program, its five
Particulate Matter Research Centers, and the Health Effects Institute,
an organization funded in part by EPA. In addition, an EPA contractor
has begun to investigate methods for conducting a formal analysis that
would consider sources of uncertainty, including relative toxicity and
lag effects. To date, the contractor has created a model to assess
whether and how much these sources of uncertainty may affect benefit
estimates in one urban area.
The National Academies also recommended that EPA incorporate an
assessment of uncertainty into the early stages of its benefit analyses
by characterizing the uncertainty of its emissions estimates on which
the agency is going to base its benefit estimates.[Footnote 15] While
the agency is investigating ways to assess or characterize this
uncertainty, EPA did not conduct a formal uncertainty analysis for
particulate matter emissions for the draft regulatory impact analysis
because of data limitations. These limitations stem largely from the
source of emissions data, the National Emissions Inventory,[Footnote
16] an amalgamation of data from a variety of entities, including state
and local air agencies, tribes, and industry. According to EPA, these
entities use different methods to collect data, which have different
implications for how to characterize the uncertainty. Furthermore, the
uncertainty associated with emissions varies by the source of
emissions. For example, the analytical methods for evaluating the
uncertainty of estimates of emissions from utilities would differ from
those for car and truck emissions because the nature of these emissions
and the data collection methods differ. In sum, to apply this
recommendation, EPA must determine how to characterize the uncertainty
of the estimates for each source of emissions before aggregating the
uncertainty to a national level and then factoring that aggregation
into its benefit estimates. According to EPA officials, the agency
needs much more time to resolve the complex technical challenges of
such an analysis. EPA officials also noted that the final particulate
matter analysis will demonstrate steps toward this recommendation by
presenting emissions data according to the level emitted by the
different kinds of sources, such as utilities, cars, and trucks.
Another recommendation that EPA is researching but did not apply to the
draft regulatory impact analysis concerns whether the proposed
revisions to the particulate matter standards would have important
indirect impacts on human health and the environment. According to an
EPA official, the agency could not rule out the possibility that the
revisions could have indirect impacts on the environment, such as
whether reductions to particulate matter emissions would reduce the
amount of particulate matter deposited in water bodies, thereby
decreasing water pollution. EPA has considered indirect impacts of air
pollution regulations on sensitive water bodies in the past and plans
to include a similar analysis in the final particulate matter rule. An
agency official further noted that ongoing research about environmental
impacts could reveal additional indirect impacts for future analyses.
Other recommendations that EPA did not apply to its benefit estimates
in the regulatory impact analysis concern issues such as transparency
and external review of EPA's benefit estimation process. For example,
the National Academies recommended that EPA clearly summarize the key
elements of the benefit analysis in an executive summary that includes
a table that lists and briefly describes the regulatory options for
which EPA estimated the benefits, the assumptions that had a
substantial impact on the benefit estimates, and the health benefits
evaluated. EPA did not, however, present a summary table as called for
by the recommendation or summarize the benefits in the executive
summary. As EPA stated in the particulate matter analysis, the agency
decided not to present the benefit estimates in the executive summary
because they were too uncertain. Specifically, officials said the
agency was not able to resolve some significant data limitations before
issuing the draft regulatory impact analysis in January 2006--a
deadline driven by the need to meet the court-ordered issue date for
the final rule in September 2006. According to EPA officials, EPA has
resolved some of these data challenges by, for example, obtaining more
robust data on anticipated strategies for reducing emissions, which
will affect the estimates of benefits. The officials also said that EPA
intends to include in the executive summary of the regulatory impact
analysis supporting the final rule a summary table that describes key
analytical information. EPA officials also acknowledged other
presentation shortcomings, including references to key analytical
elements that were insufficiently specific, that officials attributed
to tight time frames and the demands of working on other regulatory
analyses concurrently. They said they plan to address these
shortcomings in the final regulatory impact analysis.
Regarding external review, the National Academies recommended that EPA
establish an independent review panel, supported by permanent technical
staff, to bolster EPA's quality control measures for its regulatory
impact analyses, such as the one for particulate matter. The National
Academies noted that peer review of EPA's regulatory impact analyses
would be advantageous when the agency designs and conducts its economic
analysis.[Footnote 17] EPA has not directly addressed this
recommendation. According to the Director of the Office of Policy
Analysis and Review in EPA's Office of Air and Radiation, establishing
and supporting independent committees is costly, making it important
for EPA to take advantage of existing panels rather than set up new
ones. Further, an official in the Office of Air and Radiation who
oversees the development of regulatory impact analyses said that the
cost of reviewing all regulatory impact analyses would be substantial.
In this regard, EPA officials identified peer reviews the agency
received from its existing independent committees, such as the Clean
Air Scientific Advisory Committee and the Advisory Council on Clean Air
Compliance.[Footnote 18] For example, to respond to the Academies'
recommendations about lag effects, EPA sought independent advice on the
assumptions it was developing regarding the time between reduced
exposure to particulate matter and reductions in incidences of health
effects.[Footnote 19] Finally, EPA officials noted that although the
agency does not have each regulatory impact analysis peer reviewed, EPA
typically does have the methodologies that will be applied to
regulatory impact analyses peer reviewed. (See app. III for more detail
on these recommendations and others that EPA did not apply to the draft
particulate matter regulatory impact analysis.)
Concluding Observations:
While EPA has taken a number of steps to respond to the Academies'
recommendations on estimating health benefits, continued commitment and
dedication of resources will be needed if EPA is to fully implement the
improvements endorsed by the National Academies. In particular, the
agency will need to ensure that it allocates resources to needed
research on emerging issues, such as the relative toxicity of
particulate matter components; assessing which sources of uncertainty
have the greatest influence on benefit estimates; and estimating other
benefits, such as environmental improvements. In addition, it is
important for EPA to continue to improve its uncertainty analysis in
accordance with the Academies' recommendations. The agency's draft
regulatory impact analysis illustrates that estimates of health
benefits can be highly uncertain. In fact, EPA officials viewed these
estimates as so uncertain that they chose to not present them in the
executive summary of the regulatory impact analysis. While EPA
officials said they expect to reduce the uncertainties associated with
the health benefit estimates in the final particulate matter analysis,
robust uncertainty analysis will nonetheless be important for decision
makers and the public to understand the likelihood of attaining the
estimated health benefits. According to EPA officials, the final
regulatory impact analysis on particulate matter will reflect further
responsiveness to the Academies' recommendations by, for example,
providing additional sensitivity analysis and improving the
transparency of the regulatory impact analysis by highlighting key data
and assumptions in the executive summary. Moreover, these officials
emphasized the agency's commitment to further enhancing the
transparency of the analysis by presenting clear and accurate
references to the supporting technical documents, which detail the
analytical assumptions and describe the data supporting the estimates.
To the extent EPA continues to make progress addressing the Academies'
recommendations, decision makers and the public will be able to better
evaluate the basis for EPA's air regulations.
Agency Comments:
We provided a draft of this report to EPA for review. EPA provided
technical comments that we incorporated, as appropriate. Officials from
the Office of Policy Analysis and Review within EPA's Office of Air and
Radiation noted in their technical comments that the report provides a
fair and balanced representation of EPA's efforts to apply the National
Academies' recommendations to the draft particulate matter regulatory
impact analysis. However, these officials also cited progress made in
applying the National Academies' recommendations through analyses of
other air programs and through research and development efforts. We
note that this report does identify, as appropriate, EPA's research and
development efforts for recommendations EPA did not apply to the draft
particulate matter analysis, its plans to apply some additional
recommendations to the final particulate matter regulatory impact
analysis, and the agency's responses to recommendations in prior rule-
making analyses of air programs.
As agreed with your offices, unless you publicly announce the contents
of this report earlier, we plan no further distribution until 30 days
from the report date. At that time, we will send copies to the EPA
Administrator and other interested parties. We will also make copies
available to others upon request. In addition, the report will be
available at no charge on the GAO Web site at [Hyperlink,
http://www.gao.gov].
If you or your staff have any questions about this report, please
contact me at (202) 512-3841 or s [Hyperlink, stephensonj@gao.gov]
tephensonj@gao.gov. Contact points for our Offices of Congressional
Relations and Public Affairs may be found on the last page of this
report. Key contributors to this report are listed in appendix IV.
Signed by:
John B. Stephenson:
Director, Natural Resources and Environment:
[End of section]
Appendix I: Scope and Methodology:
We were asked to determine whether and how the Environmental Protection
Agency (EPA) applied the National Academies (Academies) recommendations
in its estimates of the health benefits expected from the January 2006
proposed revisions to the particulate matter national ambient air
quality standards. In response to this objective, we assessed EPA's
response to the Academies' recommendations and present an overview of
the agency's completed, ongoing, and planned actions addressing the
recommendations. To develop this overview, we reviewed EPA's
particulate matter regulatory impact analysis, EPA's economic analysis
guidelines, and Office of Management and Budget (OMB) guidance on
regulatory impact analysis. We also analyzed documentation addressing
current and future agency efforts to address the recommendations, such
as project planning memorandums and technical support documents
discussing the application of economic techniques. In addition, we met
with senior officials from EPA's Office of Air and Radiation, which was
responsible for developing the proposed rule and analyzing its economic
effects, and with officials from EPA's Office of Policy, Economics, and
Innovation to discuss the agency's responses to the recommendations. We
interviewed several experts outside EPA, including (1) the Chair and
other members of the National Academies' Committee on Estimating the
Health-Risk-Reduction Benefits of Proposed Air Pollution Regulations,
to clarify the basis for their recommendations; and (2) economists at
Resources for the Future, to discuss the technical issues underlying
the recommendations on uncertainty analysis.
While the 2002 National Academies' report is generally applicable to
EPA air pollution regulations, our review focused on the application of
the recommendations to the proposed revisions to the particulate matter
standards, as requested. Our work focused on broadly characterizing
EPA's progress toward applying the recommendations; we did not evaluate
the effectiveness or quality of the scientific and technical actions
the agency has taken to apply them. To assess whether and how EPA has
made progress in responding to the recommendations, we developed the
following recommendation classification continuum: applied, partially
applied, and not applied. The applied and partially applied categories
refer to completed and initiated actions in EPA's health benefit
analysis of particulate matter that corresponds to components of the
National Academies' recommendations. The not applied category includes
recommendations that EPA did not apply when conducting the analysis for
the January 2006 particulate matter regulatory impact analysis and
identifies those for which ongoing research and development efforts
were not far enough along to apply to the particulate matter analysis.
We performed our work from January 2006 to July 2006 in accordance with
generally accepted government auditing standards.
[End of section]
Appendix II: Recommendations Applied or Partially Applied to the Draft
Particulate Matter Regulatory Impact Analysis:
Table 1 provides a summary of the National Academies' recommendations
that EPA has applied or partially applied to its draft regulatory
impact analysis (RIA) for particulate matter (PM). This table also
provides GAO's assessment of EPA's progress in applying each
recommendation, in terms of steps EPA has taken thus far to address
issues highlighted in the National Academies' report. The final column
characterizes EPA's comments regarding each recommendation, including,
as pertinent, contextual information, potential impediments to
application, and intended next steps.
Table 1: Recommendations Applied or Partially Applied to the Draft
Particulate Matter Regulatory Impact Analysis:
1;
National Academies' recommendation: To the extent possible, EPA should
estimate the benefits for several regulatory options that represent the
full range of choices available to the decision maker. The regulatory
options should include graded levels of stringency requirements and the
time schedule for achieving reductions in emissions or exposures. If
options are eliminated at an earlier stage, the rationale for doing so
should be provided;
Status: Partially applied;
GAO assessment: EPA estimated the health benefits for the base case and
some of the proposed standards for fine particulate matter, but did not
estimate benefits that represented the full range of choices available
to the decision maker. In addition, EPA did not estimate the benefits
for coarse particulate matter. While EPA provided an explanation as to
why it did not estimate benefits for the regulatory options for coarse
particulate matter, it did not present its rationale for not estimating
the benefits of the full range of options for fine particulate matter;
EPA's response: EPA did not estimate benefits for coarse particulate
matter primarily because of a lack of data and tools needed to conduct
an accurate analysis. In addition, EPA said that time and resources
limited the number of regulatory options to be modeled. EPA said that
given these constraints, the agency selected regulatory options that
represent reasonable bounds on alternatives.
2;
National Academies' recommendation: EPA should modify the air quality
models used in translating predicted emissions into predicted levels of
ambient air quality to reduce resources required for air quality
modeling. This change is necessary if EPA is to evaluate multiple
regulatory alternatives and to evaluate each alternative at reasonable
time intervals, such as every 5 years. Evaluation of the ambient air
quality associated with more emissions scenarios is also essential if
the uncertainty inherent in emissions estimates is to be carried
through to the estimation of avoided cases of mortality and morbidity;
Status: Partially applied;
GAO assessment: EPA has modified air quality models but has not yet
demonstrated that these changes reduce resources. EPA reports faster
modeling runs once it has designated a model but has also needed to
conduct more complex model runs, which require time;
EPA's response: EPA characterized its response to this recommendation
as a work in progress. EPA said that the agency has achieved greater
efficiency in the models, but that the overall time and resources
devoted to modeling have not decreased because of the increased
complexity and increased volume of air quality model runs. Citing
emissions data limitations as an ongoing challenge, EPA reported
notable improvements to the emissions inventory that contributed to
efficiency gains in air quality modeling runs. Overall, EPA is working
to balance efforts to streamline models with the demands for more
sophisticated analyses. In addition, EPA cited additional analytical
requirements, such as new peer review policies and OMB's expanded data
quality guidelines, that add to the complexity of, and therefore time
and resources needed for, its models.
3;
National Academies' recommendation: EPA should incorporate estimates of
future trends in background mortality and morbidity for the major
health outcomes, such as those that make up two-thirds of total deaths
or lost life years that are under consideration;
Status: Partially applied;
GAO assessment: EPA incorporated future trends in background mortality
rates but did not do so for future morbidity trends;
EPA's response: EPA did not incorporate future morbidity trends because
of time constraints. As time permits, EPA plans to incorporate such
projections in the final analysis.
4;
National Academies' recommendation: As in all other stages of the
benefits analysis, EPA should justify and clearly describe the
assumptions and methods used to assess exposure, choose health
outcomes, and select studies and concentration-response functions,
paying careful attention to assessing and communicating key sources of
uncertainty;
Status: Partially applied;
GAO assessment: The draft PM RIA presented EPA's justifications and
included clear descriptions of a number of assumptions and methods used
in the benefit analysis. As discussed elsewhere, EPA's assessment of
key sources of uncertainty generally relied on qualitative discussion
and sensitivity analysis (see recommendation 14). However, the extent
to which EPA provided justifications and clear descriptions varied. For
example, EPA included detailed information about why it chose studies
related to the concentration-response function, but the agency did not
present its justification for an assumption used to assess exposure;
EPA's response: EPA intends to include references in the final RIA,
detailing the assumptions and methods underlying its health benefits
analyses. EPA stated that rather than restating information from prior
RIAs, it provides references to these discussions in order to manage
the length of the RIA and save time. Furthermore, EPA stated that it is
not necessary to document assumptions for tools that have already been
peer reviewed, such as BenMAP, the model used to estimate health
benefits.
5;
National Academies' recommendation: Because pollution modeling rarely
addresses the smaller-scale issue of how local concentrations from
specific source categories interact with human time-activity patterns,
EPA should examine how different major source categories--for example,
mobile versus large stationary sources--affect total exposures per unit
emissions;
Status: Partially applied;
GAO assessment: EPA assessed local concentrations in terms of how
different source categories, such as stationary and mobile sources of
PM, affect total exposures. However, EPA has not yet assessed how human-
time activity patterns, such as lifestyles, affect exposure to PM;
EPA's response: EPA focused on assessing the local concentrations from
specific source activities but assumed the same time and activity
patterns for each scenario. EPA stated that it does not intend to
conduct a detailed analysis of micro-environmental issues or human-time
activity patterns.
6;
National Academies' recommendation: EPA should consider data from U.S.
and non-U.S. studies to extrapolate beyond the age groups evaluated and
incorporate other relevant outcomes not evaluated in its current
benefit analyses;
Status: Applied;
GAO assessment: EPA considered data from additional U.S. studies as
part of its effort to expand the age groups in its estimates of the
health outcomes (premature mortality and illness). EPA extrapolated
beyond the age groups in some cases, but not all;
EPA's response: EPA determined, on the basis of advice from the Science
Advisory Board, that it would only extrapolate data to other ages when
it found a reasonable physiological basis for doing so. For example,
EPA used data from a study on asthma in children ages 7 to 11 to
estimate the reductions in asthma for the entire child age group--ages
6 to 18.
7;
National Academies' recommendation: EPA provided little information in
the benefit analyses reviewed by the committee on causal association
between particular types of air pollution and adverse health outcomes.
EPA should summarize the evidence for causality to justify the
inclusion or exclusion of the health outcomes and to assess the
uncertainty associated with the assumption of causality;
Status: Applied;
GAO assessment: EPA referred readers to a prior RIA for information on
the causal association between particulate matter and adverse health
outcomes;
EPA's response: [Empty].
8;
National Academies' recommendation: EPA should investigate and, if
necessary, develop methods of evaluating causal uncertainty relating to
key outcomes so that this uncertainty can be represented in the final
benefit estimates;
Status: Applied;
GAO assessment: EPA investigated one method--expert elicitation--to
evaluate how causal uncertainty affects final benefit estimates. See
recommendation 14 for more details;
EPA's response: EPA intends to continue its effort to better
characterize the uncertainty in key health outcomes.
9;
National Academies' recommendation: Although the committee believes the
use of the American Cancer Society (ACS) study to derive premature
mortality estimates was reasonable, EPA should thoroughly review its
selection of the best estimate for long-term effects of air pollution
on mortality. Several new studies have been published since the ACS
study, including an extended analysis of the ACS study, a new U.S.
cohort study, and other non-U.S. studies. EPA should also consider
whether the derivation of a weighted mean estimate from the cohort
studies is appropriate following review of the database;
Status: Applied;
GAO assessment: EPA reviewed its selection of the best estimate for
long-term effects of air pollution on mortality and concluded that data
from the extended analysis of the ACS study provided the best estimates
of premature mortality. EPA consulted the Science Advisory Board to
reach its decision to emphasize the ACS data. EPA also determined that
it was not appropriate to derive a weighted mean estimate from cohort
studies;
EPA's response: EPA incorporated new studies regarding estimates of
premature mortality and justified the concentration-response functions
derived from these studies. These justifications were included in
technical appendices to the draft PM RIA. EPA also consulted with the
Science Advisory Board regarding the best estimates for premature
mortality. EPA decided not to derive a weighted mean estimate because
the use of this estimate would risk losing the variability across
cohort studies.
10;
National Academies' recommendation: To evaluate short-term effects of
air pollution, EPA should use concentration-response functions from
studies that integrate over several days or weeks the exposure period
and the time period to the event (cumulative or distributed lag
models), rather than those that restrict these time periods to 1 or 2
days;
Status: Applied;
GAO assessment: EPA used studies that integrate distributed lag models
that account for the onset of health effects occurring more than
several days after exposure;
EPA's response: [Empty].
11;
National Academies' recommendation: Although the assumption of no
thresholds in the most recent EPA benefit analyses was appropriate, EPA
should evaluate threshold assumptions in a consistent and transparent
framework using several alternative assumptions in the formal
uncertainty analysis;
Status: Partially applied;
GAO assessment: EPA evaluated several different threshold assumptions
in a sensitivity analysis but has not yet considered these assumptions
in a formal uncertainty analysis. The sensitivity analysis was
transparent--EPA clearly explained the basis for the different
threshold assumptions;
EPA's response: EPA reported some progress toward improving its
approach to characterize uncertainties and to conduct a formal
uncertainty analysis. (See recommendation 14 for more information.)
12;
National Academies' recommendation: The committee found little
justification for the 5-year time course of exposure and outcome
assumed in more recent EPA analyses and recommends that EPA more fully
account for the uncertainty regarding lags in health effects by
incorporating a range of assumptions and probabilities about the
temporal relationship;
Status: Partially applied;
GAO assessment: EPA sought advice from the Advisory Council on Clean
Air Compliance (Council) and incorporated the advice by adjusting the
agency's assumptions regarding lag times. As for the uncertainty
surrounding lags in health effects, EPA conducted a series of
sensitivity analyses on the temporal relationship but did not assess
underlying probabilities;
EPA's response: Consistent with the recommendation made by the Council,
EPA now uses a 20-year lag model. The revised lag model involves a
higher percentage of mortality reductions occurring in the first year
(30 percent) than the previous EPA lag model (20 percent). EPA told us
that it will not include probability-based distributions in the final
RIA because of computational complexities.
13;
National Academies' recommendation: EPA is encouraged to estimate and
report benefits by age, sex, and other demographic factors. The
committee recognizes, however, that evaluating the differences for
various subgroups adds complexity and uncertainty to the analysis and
that caution must be exercised in the interpretation of such results;
Status: Partially applied;
GAO assessment: EPA included some health benefit estimates for specific
age groups in an appendix of the draft PM RIA. However, the agency did
not report the primary benefit estimates according to age groups or
other demographic factors;
EPA's response: EPA intends to present in the final RIA a detailed
summary table with the age breakdown of its sample population. EPA will
not, however, estimate and report benefits by other demographic factors
because of time and resource constraints. EPA also cited data
limitations, noting that it lacks data accounting for local variations
in demographic factors. The agency intends to overcome this limitation
using tract-level data, which captures details about local level
conditions, from the U.S. Census data in future analyses.
14;
National Academies' recommendation: EPA should begin to move the
assessment of uncertainties from its ancillary analyses to its primary
analyses. This shift will require the specification of a probability
distribution for each uncertainty source that is added to the primary
analysis and, as necessary, the specification of joint distributions
for the uncertainty sources that are not independent of each other.
Expert judgment, as well as data, will be required to specify these
distributions. Although the effect on the mean of the resulting
probability distribution might increase, decrease, or remain the same,
the effect on the spread of the distribution will be a predictable
widening and, therefore, a more realistic depiction of the overall
uncertainty in the analysis;
Status: Partially applied;
GAO assessment: EPA has taken some steps toward the formal uncertainty
analysis called for by this recommendation, but the primary analysis in
the draft PM RIA generally addresses uncertainty in a qualitative
manner. Overall, the numerical benefit estimates do not capture the key
sources of uncertainty. The agency generally relied on sensitivity
analysis to assess some uncertain factors one at a time rather than
using more comprehensive techniques for assessing probability
distributions of multiple variables. In addition, EPA used expert
elicitation to help assess uncertainty relating to the concentrations
of PM linked to premature death and the dollar value of risk reductions
associated with reductions in PM. However, these results were presented
in an appendix and not in the primary analysis in the draft RIA. In
addition, the health benefit analysis did not present a quantitative
assessment of how the benefit estimates would vary in light of other
key uncertainties;
EPA's response: EPA reported some progress toward improving its
approach to characterize uncertainties with particular emphasis on one
source of uncertainty--premature death linked to PM exposure.
Specifically, EPA used expert elicitation to begin to specify a
distribution for the uncertainty in concentration levels of PM linked
to premature death. The agency cited technical challenges, such as lack
of data or reliable methods, and resource constraints, including
limitations to its progress to fully characterize uncertainty.
Moreover, EPA stated that its focus on the expert elicitation technique
limited the time and resources necessary to address other aspects of
uncertainty in premature death as well as illnesses linked to PM
exposure.
EPA applied another formal method for assessing uncertainty--Monte
Carlo analysis--to a previous regulatory impact analysis but, according
to EPA, did not have time to incorporate this work in the PM RIA. Time
permitting, the agency plans to present the results of a Monte Carlo
analysis in the final PM RIA.
15;
National Academies' recommendation: Because the incorporation of expert
judgment when data are unavailable will influence the estimates of
health benefits as well as the uncertainty analyses, the committee also
recommends that EPA clearly distinguish between data-derived estimates
of some components, such as the concentration-response function, and
expert opinions about other components that are lacking in scientific
data, such as the degree of compliance with a particular regulation 30
years into the future. In this way, policymakers will better understand
how existing data and expert judgment combine to produce estimates and
where new data would be most valuable;
Status: Applied;
GAO assessment: EPA distinguished between data-derived estimates and
those from expert judgment. For example, in the appendix, EPA clearly
distinguished between data derived from experts and the data based on
an empirical study. In addition, EPA discusses the basis for
assumptions, which require analytical judgment, either directly in the
draft PM RIA or by reference to supporting documents;
EPA's response: [Empty].
16;
National Academies' recommendation: As EPA begins the transition to
incorporate additional sources of uncertainty into its primary health
benefits analyses, it should continue the sensitivity analyses it has
traditionally conducted. These analyses should be expanded, however, to
consider sources of uncertainty jointly rather than singly;
Status: Partially applied;
GAO assessment: EPA expanded the sensitivity analysis by, for example,
considering how benefit estimates change according to different
threshold assumptions, such as cut points-- concentrations of
particulate matter below which there would be no benefit to further
reductions. This sensitivity analysis examined how different assumed
cut points would change the estimates of avoided cases of death. EPA
did not, however, conduct a sensitivity analysis to consider the
sources of uncertainty jointly rather than singly;
EPA's response: EPA has begun to explore how to expand sensitivity
analysis and consider sources of uncertainty jointly. EPA noted that in
recent air pollution analyses, the agency considered sources of
uncertainty jointly for the last stage of the benefit estimation
process--valuing the benefits in dollar terms--by doing a probabilistic
assessment of the value of a statistical life.
In addition, EPA's policy and economics division has begun a long-term
project to expand uncertainty analysis. EPA reports that one technique
that considered how much different sources of uncertainty affect the
estimates is not ready to be applied to a rule-making analysis.
17;
National Academies' recommendation: In presenting the probability
distribution for each health benefit produced by a primary analysis,
EPA should emphasize even more than it has in the past the sources of
uncertainty that remain unaccounted for in the primary analysis. These
uncertainties should continue to be described as completely and
realistically as possible;
Status: Partially applied;
GAO assessment: EPA did not present a probability distribution for the
primary benefit analysis but included some discussion of sources of
uncertainty not incorporated into the benefit estimates. EPA also
referred the reader to a previous analysis involving particulate matter
for detailed discussions on sources of uncertainty. The citations are
incomplete, however, leaving readers to search within the voluminous
document for the relevant information;
EPA's response: EPA attributed incomplete or lacking references in the
draft PM RIA to time constraints, noting that it would present clear
references in the final version. EPA also clarified that it relies on
references to other documents, rather than repeating information that
has not changed, in order to keep the presentation to a manageable
size.
(See also recommendation 14 for EPA's basis for not including
probability distributions.)
18;
National Academies' recommendation: EPA should consider providing a
preliminary analysis that estimates, in current populations, the health
benefits resulting from hypothetical changes to current levels of
emissions. These preliminary analyses would help EPA develop an idea of
the lower bound on the uncertainty of future consequences and would
have fewer uncertainties than analyses of the impacts of proposed
regulatory actions on future exposures and health outcomes;
Status: Applied;
GAO assessment: According to EPA, the agency considered providing the
preliminary analysis described in this recommendation;
EPA's response: EPA determined that this information would not provide
meaningful information to the draft PM benefit analysis. Aside from its
questions about the technical feasibility of responding to this
recommendation, EPA expressed doubts that doing this analysis for
current populations would establish a lower bound on the uncertainty of
future consequences. One EPA official concluded that the analysis could
introduce more uncertainty into its benefit estimates because benefit
projections will depend on controls implemented in the future. Finally,
EPA clarified that, as provided by Circular A-4, it estimates benefits
for a future year when the regulatory revisions become effective, not
the current year, which would not account for changes in variables
other than emissions.
19;
National Academies' recommendation: EPA should continue to strive to
present the results of its health benefit analyses in ways that avoid
conveying an unwarranted degree of certainty. Such ways include
rounding to fewer significant digits, increasing the use of graphs,
presenting projected baselines along with projected health benefits,
and placing less emphasis on single numbers (for example, the mean of
the probability distribution for a health benefit) and greater emphasis
on ranges (for example, the range between 5th and 95th percentiles of
the distribution);
Status: Applied;
GAO assessment: EPA followed some of the National Academies'
suggestions to present the data in a way that avoids conveying an
unwarranted level of certainty. EPA rounded the estimates to fewer
significant digits. EPA increased the emphasis on ranges by presenting
some data in ranges of benefit and cost estimates. EPA summarized the
primary benefit estimates in a table;
EPA's response: [Empty].
20; National Academies' recommendation: EPA should perform similar
detailed analyses of uncertainty in the valuation of health benefits
and in the regulatory cost analyses that the committee recommends for
the health benefit analyses;
Status: Partially applied;
GAO assessment: The draft PM RIA demonstrated some steps toward
analysis of cost uncertainty but did not present an uncertainty
analysis of the benefit valuation--the stage when it assigns a dollar
value to the benefit estimates. In terms of cost uncertainty, EPA
presented a qualitative discussion of the uncertainties about costs and
the expected impact on the cost estimates but did not perform a formal
uncertainty analysis for the costs;
EPA's response: Overall, EPA cited technical challenges, including data
limitations, as the primary reason for not applying a formal
uncertainty analysis to the cost estimates in the PM RIA. According to
EPA, the number and breadth of PM sources considered under the National
Ambient Air Quality Standards rule poses a challenge to development of
a characterization of all costs and their uncertainties. Moreover, EPA
noted that the collection process for cost information is not
systematic, and there is a limited amount of information about the cost
of implementing some types of controls. Even when costs are known, it
may be difficult to specify the underlying probability distributions.
EPA told us that it will incorporate more refined information, such as
emissions control analysis, affecting cost estimates in the final RIA.
EPA expects the final RIA to include more reasonable cutoffs such that
cost estimates will not be based on ineffective controls and excessive
costs (i.e., controls that are not likely to be used).
In terms of health valuation uncertainty, EPA has made progress toward
a formal uncertainty analysis by doing a probabilistic assessment of
the value of a statistical life. EPA conducted this analysis in prior
RIAs but did not have enough time to complete model runs in time for
the draft RIA. EPA plans to incorporate an uncertainty analysis related
to the mortality valuation in the final PM RIA.
EPA has focused its expansion of uncertainty analysis more on benefits
than on costs, in part because the benefit estimates for air rules tend
to be much larger. EPA identified efforts to analyze cost uncertainty
factors, including sensitivity analyses in previous RIAs and plans for
accountability analysis comparing estimated costs to actual costs.
Finally, EPA pointed out that the work it conducts to understand the
benefits also carries over to the cost estimates.
21;
National Academies' recommendation: EPA should provide health benefit
estimates in ways that will support multiple kinds of analysis,
including various approaches to mortality valuation and aggregation of
benefits using quality-adjusted life years;
Status: Partially applied;
GAO assessment: EPA's detailed breakdown of the benefit estimates and
the reference to its benefit model allows others to apply various
approaches to mortality valuation, such as alternative estimates of the
value of a statistical life. EPA did not, however, aggregate benefits
using quality-adjusted life years;
EPA's response: Although EPA used the quality-adjusted life years
approach in a prior RIA to aggregate benefits, EPA did not use this
approach in the draft PM RIA because of time constraints. EPA told us
that it plans to aggregate the benefits using quality-adjusted life
years in the final PM RIA.
22;
National Academies' recommendation: Each analysis should provide
results according to demographic or other subgroups when the expected
changes in pollution and, thus, the health benefits are not distributed
uniformly across the population. This information would aid decision
makers in situations in which equity issues might be involved;
Status: Partially applied;
GAO assessment: EPA included some health benefit estimates for specific
age groups in an appendix, but the agency did not present the primary
benefit estimates according to age groups or other demographic factors.
(See also recommendation 13.);
EPA's response: EPA intends to present in the final RIA a detailed
summary table with the age breakdown of its sample population. The
agency noted that it will not present the benefits according to other
demographic factors, such as race and income, because of political
sensitivities and data limitations. EPA clarified that it lacks the
data on local conditions but that it intends to overcome this
limitation using tract- level data--which captures details about local
conditions--from the U.S. Census data in future analyses.
Source: GAO analysis of National Academies and EPA information.
[End of table]
[End of section]
Appendix III: Recommendations Not Applied to the Draft Particulate
Matter Regulatory Impact Analysis:
Table 2 provides a summary of the National Academies' recommendations
that EPA has not applied to its draft regulatory impact analysis (RIA)
for particulate matter (PM). This table provides GAO's assessment of
EPA's progress to date regarding recommendations that required
additional research and development, were deemed as not relevant to the
PM National Ambient Air Quality Standards (NAAQS) by the agency, or
were not included in the draft PM RIA due to time and resource
constraints. The final column characterizes EPA's comments regarding
each recommendation, including contextual information, potential
impediments to application, justification for not addressing the
recommendation, and intended next steps, if applicable.
Table 2: Recommendations Not Applied to the Draft Particulate Matter
Regulatory Impact Analysis:
1;
National Academies' recommendation: The uncertainty in emissions
estimates should be quantified and carried through the health benefit
analysis to the calculation of avoided cases of mortality and
morbidity;
Status: Not applied--research and development under way;
GAO assessment: EPA has not quantified the uncertainty related to
emissions because of limited data and computational complexities and
has therefore not yet carried such uncertainty through in the health
benefit analysis;
EPA's response: EPA stated that the application of this recommendation--
conducting a formal analysis of emissions uncertainty--requires long-
term research and development. EPA reports that it discussed the
possibility of conducting a quantitative uncertainty analysis for
emissions in its particulate matter analysis, but the fixed timeline
prevented EPA from doing this work. The primary challenge stems from
the nature of the emissions inventory--data are collected from a
plethora of entities, complicating the agency's ability to evaluate
uncertainty.
EPA told us that, currently, the only way to assess emissions
uncertainty is through qualitative means. EPA also stated that its
final particulate matter analysis will demonstrate steps toward this
recommendation because it will present a sensitivity analysis of the
emissions data and will present emissions data according to the level
emitted by the different kinds of sources, such as utilities, cars, and
trucks.
2;
National Academies' recommendation: Because a regulation to improve air
quality may affect pathways other than air, EPA should determine
whether there are likely to be any important indirect impacts of a
regulation on human health and the environment. If any such impacts are
identified, EPA should include in the analysis a plan to assess them
more completely;
Status: Not applied--research and development under way;
GAO assessment: While EPA did not provide information in the draft PM
RIA to show that it considered indirect impacts involving pathways
other than air, EPA's ongoing research on the environmental impacts may
identify important indirect impacts;
EPA's response: EPA stated that in past rules, the agency has looked at
indirect impacts in terms of deposition of nitrogen and sulfates to
sensitive water bodies. EPA plans to incorporate this analysis in the
final PM RIA. EPA said it has not yet identified any other indirect
impacts. While an EPA official suggested that this recommendation did
not seem relevant to the NAAQS analysis in terms of human health
impacts--EPA could not determine how human health would be affected by
exposure to PM from pathways other than air--the agency is conducting
research to identify important indirect impacts on the environment. EPA
characterized environmental impacts as an area of research.
3;
National Academies' recommendation: EPA has typically made the
assumption of equivalent potency across particle types because of
insufficient scientific information. As more data become available, EPA
should strengthen its benefit analyses by evaluating a range of
alternative assumptions regarding relative particle toxicity and
incorporate these assumptions in sensitivity or uncertainty analyses;
Status: Not applied--research and development under way;
GAO assessment: Although EPA assumed equivalent toxicity in the PM RIA
and did not include related sensitivity or uncertainty analyses, EPA is
sponsoring research directed at incorporating its findings on relative
toxicity into future analyses;
EPA's response: EPA stated that it does not have sufficient information
to distinguish between particle components for the final rule. EPA is
funding long-term research on relative toxicity, including technical
studies to understand any differential toxicities as well as economic
analyses to explore ways to characterize the uncertainty in benefit
estimates. For example, an EPA contractor conducted a sensitivity
analysis of relative toxicity of particle components, including
carbons, nitrates, crustal material, and sulfates. To date, the
contractor has created a model to assess whether and how much these
sources of uncertainty may affect benefit estimates in one urban area.
EPA is also supporting research to explore relative toxicity through
its intramural research program, its five Particulate Matter Research
Centers, and the Health Effects Institute, an organization funded in
part by EPA. EPA's science grant program recently awarded $40 million
to the five Particulate Matter Research Centers.
4;
National Academies' recommendation: As it incorporates additional
sources of uncertainty into its primary health benefit analyses, EPA
should consider conducting analyses to determine which uncertainty
sources have the greatest influence on the mean and spread of the
probability distribution. The need for these sensitivity analyses will
be particularly great for distributions that are based on expert
judgment. The uncertainty sources that have the greatest consequences
for decision making, including those that have the greatest impact on
the spread of the distribution, should be given high priority for
additional research;
Status: Not applied--research and development under way;
GAO assessment: In the draft PM RIA, although EPA presented a
qualitative discussion about the importance of its assumptions that
impact uncertainty, it did not consider which sources of uncertainty
have the greatest influence on the mean and spread of the probability
distribution. EPA is sponsoring research to incorporate influence
analysis in future analyses. (See recommendation 14; EPA did not
specify probability distributions for uncertainty in its primary
analysis);
EPA's response: An EPA contractor is researching techniques for
influence analysis, an expanded form of uncertainty analysis that would
determine which uncertainty sources have the greatest influence on the
benefit estimates. The influence analysis work targeted three sources
of uncertainty, including the concentration-response function (which
involves threshold and slope); lag effects; and relative toxicity. The
contractor compiled a draft report discussing techniques to conduct
this analysis and incorporate the uncertainty analysis in benefit
estimates.
EPA and the contractor stated that because these techniques are in the
exploratory stage, it is premature to apply this work to a specific
rule-making analysis. For example, EPA cited the need to determine how
much uncertainty is explained by differential toxicity or by different
thresholds. Finally, EPA stated that because it focused its resources
on the expert elicitation work (see recommendation 14), the influence
analysis received fewer resources and has not advanced as quickly.
5;
National Academies' recommendation: EPA should estimate the benefits
over the regulatory time period, including both the implementation
period and the expression period of all important health effects.
Because calculating benefits for every future year is resource
intensive and unlikely to show true increases in precision,
calculations can be made, for example, every fifth year with simple
interpolation techniques applied to estimate benefits for intervening
years;
Status: Not applied;
GAO assessment: EPA did not estimate benefits covering the
implementation period and expression period of all important health
effects. EPA estimated the benefits expected in one year only--the base
case year, 2015. The year 2015 is the first attainment date when states
should be in compliance with the new standards. The Clean Air Act
allows for up to a 5-year extension for states that cannot meet the
standards by the attainment year. Therefore, the implementation period
for some states may extend to 2020;
EPA's response: EPA cited limited time and resources to estimate
benefits for years other than 2015, but plans to also include estimates
for 2020, in the final RIA. In addition, EPA did not think that this
recommendation is particularly meaningful to the NAAQS analysis because
the only variable change over the course of the PM implementation
period is population. EPA concluded that there would not be much
difference between the benefit estimates given in 5-year increments
(i.e., 2010 and 2015).
6;
National Academies' recommendation: The components of emissions
estimates (such as number of vehicles in a class, average miles
traveled per vehicle, and emissions per mile) should be presented with
and without implementation of the regulation at the national level.
This will help readers judge how reasonable these predictions are and
will suggest which components of emissions estimates drive the
emissions reductions associated with the regulation. Historical trends
in these components should also be presented;
Status: Not applied;
GAO assessment: EPA presented a qualitative discussion about emissions,
but did not address components of emissions estimates or provide
information to allow readers to understand which components of
emissions estimates drive the reductions associated with regulation,
such as activity level or emissions intensity. EPA did not present
historical trends;
EPA's response: EPA stated that the kind of information called for by
this recommendation--key input data, assumptions, and intermediate
modeling outcomes--would be useful in future rule-making analyses, but
that the agency needs to review a large amount of data to determine
which elements would be most helpful. EPA questioned the value of
incorporating all of the recommended information in regulatory impact
analyses, noting that it works to present enough information to readers
while maintaining a document of manageable length.
EPA stated that it needs time to design data reporting strategies that
would be; appropriate for the different scales and scopes of the
regulatory impact analyses. EPA plans to explore this recommendation
more thoroughly as part of its comprehensive economic analysis of the
Clean Air Act; Finally, EPA officials suggested that the final PM RIA
will respond to this recommendation in part by providing information
about how emissions reductions might vary across PM sources in order to
show the primary drivers of emissions reductions in the final RIA. EPA
also plans to compare current data to the historical trends predicted
in the past and show that EPA's current predictions are for the future.
7;
National Academies' recommendation: EPA should quantify uncertainties
with regard to future population distributions and background disease
rates. EPA should also summarize what is known about the potential
importance of disease interactions and competing risks affecting the
health outcomes of primary interest and discuss the possible biases
that might be introduced in the final analysis by changes in those
factors;
Status: Not applied;
GAO assessment: EPA did not quantify the uncertainties related to
future population distributions and background disease rates. EPA did
not summarize what is known about the potential importance of disease
interactions and competing risks affecting health outcomes of primary
interest;
EPA's response: EPA said its models are not configured to quantify
these sources of uncertainty. EPA would need to modify the model in
order to do this analysis. Agency officials noted that the agency tries
to be selective when determining which sources of uncertainty to assess
because the cost of doing this work might outweigh the value added from
the information.
In addition, EPA officials said the agency does not have all of the
data necessary to reconfigure the models to quantify key sources of
uncertainty. EPA disagrees with the National Academies' comment that
lack of information should not preclude the quantification of
uncertainty. EPA believes that using the techniques to quantify
uncertainty without empirical data would generate results that could be
more misleading than the results that do not account for uncertainty.
EPA plans to add relevant uncertainty characterizations as it obtains
data.
8;
National Academies' recommendation: The lack of clear categorization of
severity of certain health outcomes in benefits analyses has
implications for the quantification and the valuation of these
outcomes. Although EPA has made some attempt to recognize this issue,
it should continue to develop and improve methods used to reconcile
differences between the severity of disease described in air pollution
epidemiology and that commonly used to develop estimates of background
disease prevalence and incidence;
Status: Not applied;
GAO assessment: The model EPA used to estimate benefits, BenMAP, did
not account for the potential variations in severity of illnesses and
prevalence and expected incidence of health effects;
EPA's response: EPA has begun work on categorizing severity of some
health outcomes, including chronic bronchitis and asthma incidence. EPA
is continuing to refine its BenMAP model to better quantify and
monetize health outcomes.
9;
National Academies' recommendation: EPA should give more emphasis to
the assessment, presentation, and communication of changes in
morbidity. Although often difficult to quantify, these factors may
begin to play a more dominant role in benefit analysis if the value
assigned to mortality decreases;
Status: Not applied;
GAO assessment: EPA did not place more emphasis on the assessment,
presentation, and communication of changes in morbidity. Prior RIAs
have quantified expected changes in morbidity, such as reductions in
asthma and chronic bronchitis. The draft PM RIA did not include
additional morbidity information. Moreover, the main benefit estimates
in the draft PM RIA did not include morbidity estimates;
EPA's response: EPA acknowledged the importance of morbidity--i.e.,
illness--benefits and is working to expand that analysis. For example,
EPA told us that it plans to include sensitivity analysis of changes in
illnesses in the final PM RIA. EPA also stated that the final PM RIA
will reflect updates to its model-- EPA is working to include
projections that will allow the agency to evaluate expected changes in
illnesses such as asthma and chronic bronchitis.
10;
National Academies' recommendation: There is a common misperception
that a high degree of certainty is required for regulatory actions to
take place to protect public health. As a result, primary health
benefit analyses that more fully and accurately portray the
uncertainties might not be considered useful. It is unrealistic for EPA
to defer decisions until it can make them on the basis of perfect
science. A careful and deliberate balancing of the benefits and costs
is required, and this balancing must be informed by a fair assessment
of the current levels of uncertainty and a realistic evaluation of the
likely reductions in uncertainty attainable through further research;
Status: Not applied;
GAO assessment: EPA did not balance the costs and benefits--the Clean
Air Act prohibits EPA from basing revisions to the NAAQS on costs. We
note that in response to other recommendations, EPA has taken steps
toward assessing current levels of uncertainty--see appendix II,
recommendations 14 and 20;
EPA's response: EPA characterized this recommendation as not germane to
the particulate matter regulatory impact analysis because the Clean Air
Act prohibits the agency from considering the costs when revising
NAAQS.
11;
National Academies' recommendation: EPA should provide a summary of the
analysis containing information as outlined in the National Academies'
report (table 6-1). This information would allow the reader to evaluate
the study design and verify estimates obtained in the analysis;
Status: Not applied;
GAO assessment: EPA did not provide the summary table in the executive
summary as outlined in the National Academies' report, including a
description of regulatory options, boundaries of analysis, regulatory
baseline, and assumptions that have a significant impact on results of
analysis;
EPA's response: Due to data and time constraints, EPA did not summarize
its conclusions in the executive summary of the PM RIA.
12;
National Academies' recommendation: To enhance the quality of future
regulatory benefit analyses, a standing, independent technical review
panel should advise EPA in the initial stages of its benefit analysis.
This panel should have expertise in regulatory options analysis,
emissions and exposure assessment, toxicology, epidemiology, risk
analysis, biostatistics, and economics and should be appointed with
strict attention to avoiding conflict of interest, balancing biases and
ensuring broad representation. This panel should be supported by
permanent technical staff to ensure consistency of reviews over time.
EPA should follow the panel's guidance on the need for peer review;
Status: Not applied;
GAO assessment: EPA did not convene a standing group of experts to
guide the agency's initial work on the draft PM RIA. In the course of
developing economic methodologies, EPA sought and considered
information from existing advisory committees on the soundness of
certain assumptions, such as cessation lags;
EPA's response: EPA stated that in addition to the advice it sought
from the National Academies, the agency continues to seek input from
independent committees, including the Clean Air Scientific Advisory
Committee, the Advisory Council on Clean Air Compliance Analysis, and
subcommittees chartered by the Science Advisory Board's Environmental
Economics Advisory Committee. The committees advised EPA on data,
methods, and modeling choices applicable to various economic analyses,
including the draft PM RIA.
EPA stated that the costs involved in convening an entirely new panel
could be prohibitively expensive, due to organizing costs and travel
expenses and the scope of the Academies' recommendation. As a result,
EPA said that it has attempted to take advantage of existing groups,
such as the Advisory Council on Clean Air Compliance Analysis, rather
than arrange for a new panel; Furthermore, EPA noted that new
methodologies and assumptions used in the PM RIA were peer-reviewed and
that previously used methods used in the RIA had already been reviewed
and validated in a prior context.
Source: GAO analysis of National Academies and EPA information.
[End of table]
[End of section]
Appendix IV: GAO Contact and Staff Acknowledgments:
GAO Contact:
John B. Stephenson, (202) 512-3841 or s [Hyperlink,
stephensonj@gao.gov] tephensonj@gao.gov:
Staff Acknowledgments:
In addition to the contact named above, Christine Fishkin, Assistant
Director; Kate Cardamone; Nancy Crothers; Cindy Gilbert; Tim Guinane;
Jessica Lemke; and Meaghan K. Marshall made key contributions to this
report. Timothy Bober, Marcia Crosse, and Karen Keegan also made
important contributions.
(360660):
FOOTNOTES
[1] American Lung Ass'n v. Whitman, No. 1:03CV00778 (D. D.C. 2003).
[2] The National Academies comprises four organizations: the National
Academy of Sciences, National Academy of Engineering, Institute of
Medicine, and National Research Council.
[3] National Research Council, Estimating the Public Health Benefits of
Proposed Air Pollution Regulations (Washington, D.C., 2002).
[4] The Clean Air Act prohibits EPA from basing revisions to the
national air quality standards on costs. Because most national air
quality revisions qualify as significant actions under Executive Order
12866, EPA usually assesses the expected benefits and costs of the
standards under the Executive Order.
[5] Specifically, Circular A-4 states that agencies should conduct a
formal probabilistic assessment of key uncertainties underlying its
cost-and-benefit estimates. OMB Circular No. A-4, Regulatory Analysis
(Sept. 17, 2003).
[6] GAO, Air Pollution: Information Contained in EPA's Regulatory
Impact Analyses Can Be Made Clearer, GAO/RCED-97-38 (Washington, D.C.:
Apr. 14, 1997); and Cost-Benefit Analysis Can Be Useful in Assessing
Environmental Regulations, Despite Limitations, GAO/RCED-84-62
(Washington, D.C.: Apr. 6, 1984).
[7] Monte Carlo simulation refers to a computer-based analysis that
uses probability distributions for key variables, selects random values
from each of the distributions simultaneously, and repeats the random
selection over and over. Rather than presenting a single outcome--such
as the mostly likely or average scenario--Monte Carlo simulations
produce a distribution of outcomes that reflect the probability
distributions of modeled uncertain variables.
[8] It is important to note, according to EPA, that quantified benefit
estimates do not include other potential benefits, such as reduction in
certain illnesses or environmental impacts, because of limited data.
The fraction of total benefits attributable to reductions in mortality
may therefore vary as other benefits are incorporated in the numerical
estimates.
[9] The pilot expert elicitation, based on methods that were peer-
reviewed, involved structured, daylong interviews with five experts
about particulate matter exposure and death. EPA then analyzed the
experts' responses to develop ranges reflecting the experts' confidence
in estimated reductions in premature death associated with the proposed
revisions. Pending completion of a peer review, EPA plans to include
the analysis of a full-scale expert elicitation panel in the final
regulatory impact analysis.
[10] Recent EPA analyses used the natural background concentrations of
particulate matter, rather than zero, for its assumption of no
threshold level. The National Academies supported the assumption of no
threshold level, but it recommended that EPA conduct a consistent and
transparent sensitivity analysis to consider various threshold levels.
[11] Particulate matter is a highly complex mixture comprising
particles emitted directly from sources and particles formed through
atmospheric chemical reactions. Particles span many sizes and shapes
and consist of hundreds of different chemicals. EPA identifies the
major components of fine particulate matter as carbon, sulfate and
nitrate compounds, and crustal/metallic materials such as soil and ash.
[12] Relative toxicity refers to the premise that different components
of particulate matter have different levels of potency affecting
premature mortality and illness. In the draft particulate matter
regulatory impact analysis, EPA assumed equivalent toxicity, stating
that "while it is reasonable to expect that the potency of components
may vary across the numerous effect categories associated with
particulate matter, EPA's interpretation of scientific information
considered to date is that such information does not yet provide a
basis for quantification beyond using fine particle mass." EPA, Draft
Regulatory Impact Analysis for the PM-2.5 National Ambient Air Quality
Standards (Washington, D.C., 2006), 3-21.
[13] In the context of the National Academies recommendations, a
sensitivity analysis would assess how changes in one or more variables
affect the outcome, whereas a comprehensive or formal uncertainty
analysis would evaluate the probability distributions of multiple
variables.
[14] In a separate report issued in 2004, the National Academies
identified relative toxicity as a priority research topic, noting that
technical challenges have impeded research progress. The Clean Air
Scientific Advisory Committee also noted the need for more research and
concluded in 2005 that not enough data are available to base the
particulate matter standards on composition. OMB, however, encouraged
EPA in 2006 to conduct a sensitivity analysis on relative toxicity and
referred the agency to a sensitivity analysis on relative toxicity
funded by the European Commission.
[15] Because the precise levels of total emissions are not knowable but
rather approximations based on a sample of measurements, there is
uncertainty about the true quantity of emissions.
[16] EPA compiles the National Emissions Inventory, a national database
of air emissions data that includes estimates of annual emissions, by
source, of air pollutants in each area of the country on an annual
basis.
[17] In prior work on regulatory economic analyses, we recommended that
OMB direct agencies to obtain peer review of these analyses. See GAO,
Regulatory Reform: Agencies Could Improve Development, Documentation,
and Clarity of Regulatory Economic Analyses, GAO/RCED-98-142
(Washington, D.C.: May 26, 1998).
[18] The Advisory Council on Clean Air Compliance, composed of rotating
membership, serves to advise the agency on choices relating to data,
modeling, and methodology associated with air programs and does not
review regulatory impact analyses. Established under the Clean Air Act
and operated through the Science Advisory Board, the advisory council
advises EPA, under Section 812 of the act, on developing the
"statutorily mandated comprehensive analyses of the total costs and
total benefits of programs implemented pursuant to the Clean Air Act."
These analyses are commonly referred to as the 812 studies.
[19] In 2004, EPA asked for and received guidance from the Advisory
Council on Clean Air Compliance on the lag estimates the agency should
use in the particulate matter analyses.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
