Pipeline Safety
Systematic Process Needed to Evaluate Outcomes of Research and Development Program Gao ID: GAO-03-746 June 30, 2003From 1998 through 2002, a total of 1,770 pipeline accidents occurred, resulting in 100 fatalities and $621 million in property damage. The Office of Pipeline Safety (OPS) within the Department of Transportation operates a research and development (R&D) program aimed at advancing the most promising technologies for ensuring the safe operation of pipelines. In fiscal year 2003, OPS received $8.7 million for its R&D program, a sevenfold increase since fiscal year 1998. In response to a directive from the House Committee on Appropriations, GAO (1) assessed OPS's distribution of funding among various areas of R&D and the alignment of this funding with its mission and goals, (2) surveyed experts to obtain their views on R&D priorities, and (3) determined how OPS evaluates R&D outcomes.
OPS distributes its R&D budget among four main areas. For example, in fiscal year 2003, the office plans to allocate its $8.7 million budget as follows: 46 percent ($4.0 million) to developing new technologies to prevent damage to pipelines and prevent leaks; 21 percent ($1.9 million) to improving technologies for operating, controlling, and monitoring the condition of pipelines; 19 percent ($1.7 million) to improved pipeline materials, such as materials that are resistant to damage and defects; and 14 percent ($1.2 million) to efforts to improve data on the location and safety performance of pipelines. On the basis of our work, we believe that OPS's R&D funding is generally aligned with its mission and pipeline safety goals. OPS has taken a number of steps to ensure this alignment. For example, it obtained the views of a variety of experts and stakeholders in deciding on its R&D priorities and has described in various plans how its R&D efforts can lead to new and improved technologies that can help achieve its safety performance goals, such as reducing the impacts of pipeline accidents. The pipeline safety R&D priorities of the experts we surveyed are generally consistent with OPS's R&D priorities. For example, most assigned a high priority to the two areas of R&D that receive the highest amount of funding from OPS. OPS's efforts to evaluate the outcomes of its R&D have been limited. The agency has taken some preliminary steps toward developing an evaluation process for its R&D program, such as identifying possible measures of program results. Leading research organizations, the Office of Management and Budget, and GAO have identified a number of best practices for systematically evaluating the outcomes of federal R&D programs, such as setting clear R&D goals, measuring progress toward goals, and reporting periodically on evaluation results. These best practices can help OPS to determine the effectiveness of its R&D program in achieving desired outcomes, such as the development and use of new and improved technologies that can enhance pipeline safety.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-03-746, Pipeline Safety: Systematic Process Needed to Evaluate Outcomes of Research and Development Program
This is the accessible text file for GAO report number GAO-03-746
entitled 'Pipeline Safety: Systematic Process Needed to Evaluate
Outcomes of Research and Development Program' which was released on
June 30, 2003.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Subcommittees:
June 2003:
Pipeline Safety:
Systematic Process Needed to Evaluate Outcomes of Research and
Development Program:
GAO-03-746:
GAO Highlights:
Highlights of GAO-03-746, a report to congressional subcommittees
Why GAO Did This Study:
From 1998 through 2002, a total of 1,770 pipeline accidents occurred,
resulting in 100 fatalities and $621 million in property damage. The
Office of Pipeline Safety (OPS) within the Department of
Transportation operates a research and development (R&D) program aimed
at advancing the most promising technologies for ensuring the safe
operation of pipelines. In fiscal year 2003, OPS received $8.7 million
for its R&D program, a sevenfold increase since fiscal year 1998. In
response to a directive from the House Committee on Appropriations,
GAO (1) assessed OPS‘s distribution of funding among various areas of
R&D and the alignment of this funding with its mission and goals, (2)
surveyed experts to obtain their views on R&D priorities, and (3)
determined how OPS evaluates R&D outcomes.
What GAO Found:
OPS distributes its R&D budget among four main areas. For example, in
fiscal year 2003, the office plans to allocate its $8.7 million budget
as follows:
* 46 percent ($4.0 million) to developing new technologies to prevent
damage to pipelines and prevent leaks;
* 21 percent ($1.9 million) to improving technologies for operating,
controlling, and monitoring the condition of pipelines;
* 19 percent ($1.7 million) to improved pipeline materials, such as
materials that are resistant to damage and defects; and
* 14 percent ($1.2 million) to efforts to improve data on the location
and safety performance of pipelines.
On the basis of our work, we believe that OPS‘s R&D funding is
generally aligned with its mission and pipeline safety goals. OPS has
taken a number of steps to ensure this alignment. For example, it
obtained the views of a variety of experts and stakeholders in
deciding on its R&D priorities and has described in various plans how
its R&D efforts can lead to new and improved technologies that can
help achieve its safety performance goals, such as reducing the
impacts of pipeline accidents.
The pipeline safety R&D priorities of the experts we surveyed are
generally consistent with OPS‘s R&D priorities. For example, most
assigned a high priority to the two areas of R&D that receive the
highest amount of funding from OPS.
OPS‘s efforts to evaluate the outcomes of its R&D have been limited.
The agency has taken some preliminary steps toward developing an
evaluation process for its R&D program, such as identifying possible
measures of program results. Leading research organizations, the
Office of Management and Budget, and GAO have identified a number of
best practices for systematically evaluating the outcomes of federal
R&D programs, such as setting clear R&D goals, measuring progress
toward goals, and reporting periodically on evaluation results. These
best practices can help OPS to determine the effectiveness of its R&D
program in achieving desired outcomes, such as the development and use
of new and improved technologies that can enhance pipeline safety.
What GAO Recommends:
To better determine the effectiveness of its R&D program, GAO
recommends that OPS develop a systematic process for evaluating
program outcomes, using recognized best practices, and include the
results of R&D evaluations in its annual reports to Congress.
OPS officials told us that they generally agreed with the report‘s
findings and will follow our recommendations as they continue to
develop an evaluation process for their R&D program.
www.gao.gov/cgi-bin/getrpt?GAO-03-746.
To view the full product, including the scope and methodology, click
on the link above. For more information, contact Peter Guerrero at
(202) 512-2834 or guerrerop@gao.gov.
[End of section]
Letter:
Results in Brief:
Background:
OPS's R&D Funding Is Aligned with Its Mission and Pipeline Safety
Goals:
Experts Generally Support OPS's R&D Priorities:
OPS Lacks a Systematic Process for Evaluating R&D Outcomes:
Conclusions:
Recommendations for Executive Action:
Agency Comments:
Appendixes:
Appendix I: Experts‘ Views on R&D Priorities and OPS‘s R&D Funding, by
Type of R&D:
Appendix II: Scope and Methodology:
Appendix III: GAO Contacts and Staff Acknowledgments:
GAO Contacts:
Acknowledgments:
Tables:
Table 1: Major Categories of R&D Related to Pipeline Safety:
Table 2: Views of Experts from Three Subgroups on Pipeline Safety
R&D Priorities:
Figures:
Figure 1: OPS‘s R&D Budget, Fiscal Years 1998-2003:
Figure 2: OPS‘s Planned Allocation of R&D Funding for Fiscal Year 2003:
Figure 3: OPS‘s R&D Funding by Area of R&D, Fiscal Years 2001-03:
Figure 4: Expert Ratings of Categories of Pipeline Safety R&D:
Abbreviations:
DOE: Department of Energy:
DOT: Department of Transportation:
MMS: Minerals Management Service:
NIH: National Institutes of Health:
OPS: Office of Pipeline Safety:
R&D: research and development:
RSPA: Research and Special Programs Administration:
Letter June 30, 2003:
The Honorable Ernest Istook, Jr.
Chairman
The Honorable John Olver
Ranking Minority Member
Subcommittee on Transportation, Treasury, and Independent Agencies
Committee on Appropriations
House of Representatives:
The Honorable Richard C. Shelby
Chairman
The Honorable Patty Murray
Ranking Minority Member
Subcommittee on Transportation, Treasury, and General Government
Committee on Appropriations
United States Senate:
Pipelines transport nearly all of the natural gas and nearly two-thirds
of the crude oil and refined oil products in the United States.
Although pipelines have a better safety record than other modes of
freight transportation, their cargo is dangerous and leaks or ruptures
can have serious consequences, including fatalities, harm to the
environment, and property damage. For example, pipeline ruptures in
Bellingham, Washington, in 1999 and in Carlsbad, New Mexico, in 2000
together resulted in a total of 15 deaths and property and other
damages totaling about $46 million. Investigators have determined that
one of the probable causes of the Bellingham accident was excavation
damage and that the cause of the Carlsbad accident was severe internal
corrosion.
The Office of Pipeline Safety (OPS), within the Department of
Transportation's (DOT) Research and Special Programs Administration, is
responsible for pipeline safety regulation and research. The agency
operates a research and development (R&D) program aimed at enhancing
the safety and reducing the potential environmental impacts of
transporting natural gas and hazardous liquids through pipelines.
Specifically, the program seeks to advance the most promising
technological solutions to problems that impede pipeline safety, such
as damage to pipelines from excavation or corrosion. From fiscal years
2001 through 2003, the budget of OPS's R&D program more than tripled,
from $2.8 million to $8.7 million, partly as a result of congressional
interest in achieving technological advances that can improve pipeline
safety.
In House Report 107-722, which accompanied the Department of
Transportation and Related Agencies Appropriations Bill for fiscal year
2003, the House Appropriations Committee raised concerns regarding the
effective management and utilization of these significant increases in
funding for the department's pipeline safety R&D program. The committee
directed GAO to review the effectiveness of the program. In subsequent
discussions with staff of the Subcommittee on Transportation, Treasury,
and Independent Agencies of the House Appropriations Committee, we
agreed to determine (1) OPS's distribution of funding among various
areas of pipeline safety R&D since fiscal year 2001 and the extent to
which this funding is aligned with the agency's mission and pipeline
safety goals, (2) the views of experts on pipeline safety R&D
priorities, and (3) how OPS evaluates the outcomes of the pipeline
safety R&D it funds.
To carry out this work, we reviewed legislation and agency documents
pertaining to the R&D program and interviewed agency officials
responsible for this program. We also interviewed key experts and
stakeholders regarding their views on R&D priorities and gaps and on
OPS's management of its R&D program, including the alignment of the
agency's research agenda with its mission and goals. We identified best
practices for evaluating the outcomes of R&D through a review of
relevant literature. In addition, we sent a questionnaire to selected
experts to obtain their views on pipeline safety R&D priorities. We
selected experts who are informed about pipeline safety or the
development of new pipeline safety technologies, including
representatives of federal and state agencies, pipeline safety advocacy
groups, industry associations, pipeline companies, technical and
consulting organizations, and research institutes. We received
responses from 49 of 55 experts we contacted, for a response rate of 89
percent. Our results pertaining to experts' views on R&D priorities
represent the views of only the experts who responded to our
questionnaire and cannot be generalized to a broader population. (See
app. II for additional details on our scope and methodology.):
Results in Brief:
OPS distributes its R&D budget to three major areas involving the
research and development of pipeline safety technologies as well as to
a fourth area--efforts to improve the agency's pipeline mapping and
information systems. For example, in fiscal year 2003, OPS plans to
allocate its $8.7 million R&D budget as follows:
* $4.0 million (46 percent) to developing new technologies for
preventing damage to pipelines and detecting leaks,
* $1.9 million (21 percent) to improving technologies for operating,
controlling, and monitoring the condition of pipelines,
* $1.7 million (19 percent) to improving pipeline materials, and:
* $1.2 million (14 percent) on efforts to improve pipeline mapping data
and data on the safety performance of pipelines.[Footnote 1]
On the basis of our work, we believe that OPS's R&D funding is
generally aligned with its mission and pipeline safety goals. The
agency has obtained the views of external experts and stakeholders in
determining what types of R&D are aligned with its mission of ensuring
the safe, reliable, and environmentally sound operation of the nation's
pipeline transportation system. OPS has also recently improved
coordination with other federal agencies that fund pipeline R&D in
order to avoid overlap between their R&D programs. Both expert review
and coordination among agencies are recognized as best practices that
help ensure that federal agencies' R&D activities are relevant to their
missions and do not overlap. OPS has also described, in various plans,
how its R&D efforts can lead to new and improved technologies that can
help achieve its performance goals of reducing the impacts of pipeline
incidents, including fatalities and injuries, and reducing spills of
hazardous material. Key experts and stakeholders we contacted generally
told us that, in their view, the agency has chosen appropriate R&D
areas to fund.
The pipeline safety R&D priorities of the experts who completed our
questionnaire are generally consistent with OPS's R&D priorities. The
ranking of the major R&D areas based on the responses to our
questionnaire is similar to the relative levels of funding OPS has
assigned to these areas:
* 92 percent (45 of 49) of the experts assigned a high priority to the
development of new technologies for preventing damage to pipelines and
detecting leaks,
* 80 percent (39 of 49) assigned a high priority to improvements in
technologies for operating, controlling, and monitoring the condition
of pipelines, and:
* 31 percent (15 of 49) assigned a high priority to improvements in
pipeline materials.
However, the experts' level of support for improvements in pipeline
materials was much lower than that for the other two main R&D areas
that OPS is funding and this level of support differed across different
groups of experts. Although 70 percent (7 of 10) of experts from
research organizations indicated that this area should receive high
priority, only 21 percent (8 of 39) of the remaining experts--from
government, public interest, industry, and technical and consulting
organizations--indicated that it should receive high priority. OPS
officials told us that they are currently updating their research
agenda, using the input of experts and stakeholders, and that they will
consider our questionnaire results in this process.
Despite the significant growth in its R&D budget since fiscal year
2001, OPS has not developed a systematic process for evaluating the
outcomes of the R&D it funds. For example, the agency tracks and
disseminates information on the progress of individual R&D projects but
has not developed a process for assessing and reporting on the results
of its R&D program as a whole. Without such a process, OPS cannot
determine and demonstrate the progress of its R&D program in achieving
intended results, such as the development and use of new and improved
technologies that can enhance pipeline safety. The agency has taken
some preliminary steps toward developing an evaluation process for its
R&D program, such as identifying possible measures of program results,
and could benefit from adopting identified best practices for
systematically evaluating the outcomes of federal R&D programs. Leading
research organizations, the Office of Management and Budget, and GAO
have identified a number of such practices, including setting clear R&D
goals and measuring progress toward these goals, using expert review to
evaluate the quality of research outcomes, and reporting periodically
on evaluation results. The results of evaluations can be used to
refocus R&D priorities periodically, as necessary, to ensure that
program resources are most effectively utilized. The Pipeline Safety
Improvement Act of 2002 requires that, starting in December 2003, DOT,
the Department of Energy (DOE), and the National Institute of Standards
and Technology jointly provide annual reports to Congress on their
pipeline R&D efforts but does not fully specify what types of
information should be included in these reports. This requirement
provides an opportunity for OPS to keep Congress informed about the
results of evaluations of its R&D program.
To improve OPS's ability to demonstrate the effectiveness of its R&D
program and to make the most effective use of program resources, we are
recommending that the agency develop a systematic process for
evaluating program outcomes, using identified best practices, and
include the results of R&D evaluations in the required annual reports
to Congress on pipeline R&D. We provided DOT with a draft of this
report for its review and comment. DOT officials generally agreed with
the report's findings and conclusions. They emphasized that they have
started to develop a framework for evaluating the effectiveness of
their pipeline safety R&D program and that they intend to follow our
recommendations as they move forward in developing and implementing
this framework.
Background:
Three primary types of pipelines form a 2.2 million-mile network across
the nation.
* Natural gas transmission pipelines transport natural gas over long
distances from sources to communities.
* Natural gas distribution pipelines continue to transport natural gas
from transmission lines to consumers.
* Hazardous liquid pipelines transport crude oil to refineries and
refined oil products, such as gasoline, to product terminals.
OPS, within DOT's Research and Special Programs Administration (RSPA),
is responsible for enhancing the safety of and reducing the potential
environmental impacts of transporting natural gas and hazardous liquids
through pipelines. The agency primarily carries out this responsibility
through regulation, oversight, enforcement, and R&D. OPS sets and
enforces regulations that pipeline operators must follow in designing,
constructing, maintaining, and operating pipelines. State agencies
responsible for overseeing pipeline safety help OPS to enforce its
regulations.[Footnote 2] In December 2000, it began implementing a new
risk-based regulatory approach, called "integrity management." Under
this approach, operators are required, in addition to meeting minimum
safety standards, to better protect pipeline segments where a leak or
rupture could have significant consequences, such as near highly
populated areas, by conducting new tests of these segments, completing
repairs according to specified schedules, and developing comprehensive
plans for addressing the range of risks facing these segments.[Footnote
3] The agency's R&D program is aimed at advancing the most promising
technologies for ensuring the safe operations of pipelines. For
example, current R&D projects seek to develop new and improved
techniques for assessing the condition of pipelines and detecting
anomalies--such as leaks, corrosion, and damage from excavators--that
can lead to pipeline accidents. From 1998 through 2002, a total of
1,770 pipeline accidents occurred, resulting in 100 fatalities and $621
million in property damage.[Footnote 4]
OPS's R&D program has undergone major changes in the last several
years. In particular, the agency has developed a new agenda for its R&D
program, using the input of key experts and stakeholders, and has
received significant increases in funding for this program.
* Until 2001, most of the research funded by OPS was aimed at helping
the agency perform its regulatory function or was in response to an
accident investigation or congressional direction. In November 2001,
the agency held an R&D planning workshop to gain the perspectives of a
variety of experts and stakeholders on areas of R&D that have the most
potential for enhancing pipeline safety. Attendees included
representatives of federal and state agencies, research organizations,
industry groups, pipeline companies, and technical organizations that
set industry safety standards. OPS used the R&D priorities identified
in this workshop to develop a new agenda for its R&D program, focusing
on three main areas: (1) developing new technologies for preventing
damage and detecting leaks, (2) improving technologies for operating,
controlling, and monitoring the condition of pipelines, and (3)
improving pipeline materials. From March through December 2002, the
agency issued announcements requesting project proposals in these
areas, asking that prospective funding recipients provide at least 50
percent of the proposed project's cost. As of May 2003, it had funded
10 R&D proposals it received in response to these
announcements.[Footnote 5] In addition, after its November 2001 R&D
workshop, OPS established a Web site on its R&D program in order to
improve communications with experts, stakeholders, and the public about
its R&D agenda and activities.
* OPS's budget for its R&D program has risen more than sevenfold since
fiscal year 1998, with the most significant increases occurring since
fiscal year 2001. Figure 1 shows the agency's budgeted amounts for R&D
from fiscal years 1998 through 2003.[Footnote 6] OPS's budget for R&D
rose steadily from fiscal year 1998 to fiscal year 2001, from $1.3
million to $2.8 million. In fiscal year 2002, the agency received $4.8
million for its R&D program, which was $2 million more than RSPA had
requested for the program. Agency officials attribute this funding
increase to increased concerns for pipeline safety within Congress
following the tragic pipeline accidents in Bellingham, Washington
(1999), and Carlsbad, New Mexico (2000), which together caused 15
fatalities. For fiscal year 2003, RSPA requested and received about $4
million in additional funding for the program, for a total of $8.7
million. OPS officials told us that this requested increase was a
response to heightened congressional interest in achieving
technological solutions to pipeline safety, as evidenced by legislative
proposals that called for increased attention to this area.[Footnote 7]
RSPA is proposing funding for OPS's R&D program of $9.2 million in
fiscal year 2004, an increase of about $0.5 million above the fiscal
year 2003 amount. OPS officials explained that they intend to use most
of this increase for a study, required by the Pipeline Safety
Improvement Act of 2002, to assess the performance of controllers who
monitor pipeline operations. Overall, agency officials also attribute
recent increases in funding for OPS's pipeline safety R&D program to a
recognition of the challenges posed by the agency's new integrity
management regulatory approach and the criticality of the nation's
pipeline infrastructure, in the aftermath of the terrorist attacks of
September 11, 2001.
Figure 1: OPS's R&D Budget, Fiscal Years 1998-2003:
[See PDF for image]
Note: Figures are in constant fiscal year 2003 dollars.
[End of figure]
OPS's pipeline safety R&D program is continuing to evolve in response
to new directives in the Pipeline Safety Improvement Act of 2002 for
the planning and reporting of federal pipeline R&D efforts. The act,
which became law in December 2002, assigned the Secretary of
Transportation responsibility for developing a 5-year plan for pipeline
R&D and transmitting the plan to Congress by December 2003, in
coordination with DOE and the National Institute of Standards and
Technology. (OPS officials told us that the Secretary has delegated
this responsibility to OPS.):
DOE operates an R&D program that is focused on developing future
technologies to improve the integrity, reliability, and security of the
natural gas infrastructure, including pipelines and storage facilities.
In comparison with OPS's R&D program, which focuses on the development
of quick-to-market technologies that could become available in the
short term (1-3 years) or midterm (3-5 years), DOE's program focuses on
technologies that could become available in the midterm (3-5 years) or
longer term (5-8 years). The National Institute of Standards and
Technology does not operate an R&D program focused on pipelines, but,
reflecting its expertise in materials research, the act assigns it a
key role in planning future pipeline R&D.
The Department of the Interior's Minerals Management Service (MMS),
although not assigned an R&D planning role in the act, funds pipeline
R&D, including research on offshore pipeline safety. Consequently, OPS
plans to include that agency in efforts to develop a 5-year plan for
pipeline R&D. The act requires the heads of DOT, DOE, and the National
Institute of Standards and Technology to jointly report annually to
Congress, beginning in December 2003, on the status and results of
implementation of the plan.
OPS's R&D Funding Is Aligned with Its Mission and Pipeline Safety
Goals:
Since fiscal year 2001, OPS has allocated its rising R&D funding to
three main areas of pipeline safety R&D that were identified at its
2001 workshop: (1) developing new technologies for preventing damage to
pipelines and detecting leaks, (2) improving technologies for
operating, controlling, and monitoring the condition of pipelines, and
(3) improving the performance of pipeline materials. The agency has
also allocated some R&D funding to a fourth area, efforts to improve
the agency's mapping and information systems.
On the basis of our work, we believe that the agency's R&D funding is
generally aligned with its mission and pipeline safety goals. The
agency has obtained the views of external experts and stakeholders in
determining what types of R&D are aligned with its mission of ensuring
the safe, reliable, and environmentally sound operation of the nation's
pipeline transportation system. The agency has also recently improved
coordination with other federal agencies that fund pipeline R&D in
order to avoid overlap between their R&D programs. Both of these
practices have been recommended by leading organizations that conduct
scientific and engineering research. OPS has also linked its R&D
efforts with its performance goals of reducing the impacts of pipeline
incidents, including fatalities and injuries, and reducing spills of
hazardous material. In its plans, the agency has described how new and
improved technologies resulting from its R&D funding can help achieve
these performance goals. Finally, a number of key experts and
stakeholders told us that, in their view, the agency has chosen
appropriate R&D areas to fund.
OPS Allocates Pipeline Safety R&D Funding to Four Major Areas:
OPS allocates its R&D budget to three major areas involving the
research and development of pipeline safety technologies as well as to
a fourth area--efforts to improve the agency's pipeline mapping and
information systems--that does not involve such research and
development. Figure 2 shows how the agency plans to distribute its
fiscal year 2003 R&D budget of $8.7 million among these areas.[Footnote
8]
OPS plans to spend the largest share of its R&D budget, 46 percent, or
$4.0 million, on the area of Damage Prevention and Leak Detection,
which includes the development of new technologies to prevent damage to
pipelines, detect pipeline defects, and quickly and accurately locate
and control pipeline leaks. Damage to pipelines from "third parties,"
such as companies performing excavation work, is the leading cause of
pipeline failures and can lead to property damage and injuries or
fatalities.[Footnote 9]
OPS plans to allocate 21 percent of its R&D budget, $1.9 million, to
the area of Enhanced Operations, Controls, and Monitoring, which
includes improvements in technologies for operating, controlling, and
monitoring the integrity of pipelines to help identify and prioritize
pipeline safety problems and solutions.
The agency intends to spend a slightly lesser amount, 19 percent of its
R&D budget, or $1.7 million, on the area of Improved Materials
Performance, which includes improvements in pipeline materials in order
to extend the integrity and lifetime of installed pipelines and their
various components.
Finally, the agency plans to allocate the smallest portion of its R&D
budget, 14 percent, or $1.2 million, to the area of Mapping and
Information Systems, which includes efforts to improve the collection,
integration, and analysis of data on the location and safety
performance of pipelines. These efforts make pipeline mapping
information available to federal, state, and local officials and
support pipeline inspection activities of OPS and its state partners.
Figure 2: OPS's Planned Allocation of R&D Funding for Fiscal Year 2003:
[See PDF for image]
Note: Shaded areas represent the major pipeline safety R&D areas funded
by OPS. Dollar figures have been rounded.
[End of figure]
Since fiscal year 2001, OPS's allocation of funding to each of the
three main areas of pipeline safety R&D--Damage Prevention and Leak
Detection; Enhanced Operations, Controls, and Monitoring; and Improved
Materials Performance--has risen significantly, while its allocation to
Mapping and Information Systems efforts has remained level. The
tripling of the agency's R&D budget--from $2.8 million in fiscal year
2001 to $8.7 million in fiscal year 2003--has enabled it to increase
funding for these three R&D areas. Specifically, OPS has increased
funding for R&D efforts in Damage Prevention and Leak Detection from
$1.3 million in fiscal year 2001 to $4.0 million in fiscal year 2003,
an increase of over 200 percent. The agency has increased funding for
Enhanced Operations, Controls, and Monitoring from $309,000 in fiscal
year 2001 to $1.9 million in fiscal year 2003, an increase of more than
500 percent. OPS started funding Improved Materials Performance
research in fiscal year 2002, increasing funding in this area to a
level of $1.7 million in fiscal year 2003.
Agency officials explained to us that they allocated funding to these
three R&D areas in fiscal years 2002 and 2003 based on the results of
their 2001 R&D planning workshop.[Footnote 10] For example, they added
Improved Materials Performance to their R&D agenda because it was
identified as a priority area at the workshop. They have also
considered other factors in deciding how to allocate funding. For
example, the agency significantly increased funding for R&D in the
areas of Damage Prevention and Leak Detection and Enhanced Operations,
Controls, and Monitoring because of a great need for improved
performance in these areas. OPS officials explained that, because the
agency's new risk-based regulatory approach requires pipeline operators
to assess and mitigate risks to pipeline segments where a leak or
rupture could have significant consequences, these operators need
better tools and methods for monitoring pipelines and making necessary
repairs. They also noted that OPS's R&D results assist in the creation
of industry standards on the appropriate use of new technologies. In
addition, officials explained that they decided to allocate a
significant portion of their R&D budget to the area of Improved
Materials Performance because, on the basis of current information on
the development of pipeline technologies, they believed that advances
in this area held much promise for improving pipeline safety.
Finally, OPS has allocated about $1.2 million per year to the Mapping
and Information Systems area since fiscal year 2001 in order to
maintain efforts to improve these systems.[Footnote 11] (See fig. 3.):
Figure 3: OPS's R&D Funding by Area of R&D, Fiscal Years 2001-03:
[See PDF for image]
Note: Figures are in constant fiscal year 2003 dollars and represent
the agency's budgeted amounts for each area. In some cases, OPS spends
less than the budgeted amount in a fiscal year. The agency has been
allowed up to 3 years to spend amounts appropriated for R&D.
[End of figure]
OPS has provided $3.0 million in funding to 10 projects related to
Damage Prevention and Leak Detection since fiscal year 2001. Examples
of funded projects include the following:
* OPS provided $0.6 million in funding to five projects focused on
improving in-line inspection techniques, including "smart pigs" and
other technologies, for detecting damage and defects in pipe
walls.[Footnote 12] Such improved techniques can help to prevent
pipeline leaks or ruptures by making possible the early detection and
repair of damage and defects.
* In partnership with the U.S. Air Force, OPS provided $1.2 million in
funding to a project focused on developing an approach for detecting
pipeline leaks using an airborne laser system that measures levels of
chemicals in the atmosphere just above the earth's surface.
OPS has provided $0.9 million in funding to six projects related to
Enhanced Pipeline Operations, Controls, and Monitoring since fiscal
year 2001. Most of this funding--$0.6 million--has been allocated to
two projects to improve alternative inspection techniques, called
direct assessment, for identifying internal and external corrosion and
other defects in pipelines that cannot accommodate smart pigs.[Footnote
13] This is a significant issue for natural gas pipelines. One industry
association estimates that only about 35 percent of the total natural
gas pipeline mileage can accommodate smart pigs, which are typically
used to assess the condition of liquid pipelines. OPS officials told us
that they are planning to fund three additional R&D projects in this
area in June 2003.
As of May 2003, OPS has provided $0.1 million in funding to one project
in the area of Improved Materials Performance. This project seeks to
develop a "smart" composite pipe that will allow for real-time
monitoring of the condition of the pipe through a remote monitoring
system. The agency requested proposals in this area in December 2002
and expects to start funding some of these proposals in the summer of
2003. Among the types of proposals that OPS has requested are proposals
to develop:
* materials that better withstand third-party damage, corrosion, and
cracking;
* higher grade/strength steels; and:
* materials that facilitate the operation of pipelines at higher design
pressures.
Finally, of the roughly $1.2 million that OPS has allocated each year
since fiscal year 2001 to the Mapping and Information Systems area, it
spent or plans to spend:
* about $800,000 each year for efforts to improve the National Pipeline
Mapping System, which depicts the location of pipelines in relation to
areas that are populated or environmentally sensitive, and:
* about $400,000 each year for efforts to integrate information systems
the agency uses in overseeing pipeline safety in cooperation with the
states.
The agency expects to continue funding this area at this level for the
foreseeable future in order to improve and update these systems
continually. OPS officials explained that these mapping and information
systems assist OPS inspectors and state and local officials in their
efforts to oversee pipelines and protect the community and environment
from pipeline leaks or ruptures.
Expert Review and Coordination Help OPS Align Its R&D Funding with Its
Mission and Goals:
OPS's mission is to ensure the safe, reliable, and environmentally
sound operation of the nation's pipeline transportation system. It has
indicated in its budget and plans that its R&D program supports this
broad mission as well as the following more specific performance goals:
(1) to reduce deaths, injuries, property damage, and economic
disruptions resulting from pipeline incidents and (2) to reduce the
amount of oil and other hazardous liquids spilled from pipelines. The
agency has described how new and improved technologies resulting from
its R&D funding can help achieve these performance goals. For example,
the number of pipeline incidents and the amount of hazardous material
spilled could be reduced through the use of improved technologies for
detecting third-party damage, corrosion, and defects and the use of
improved pipeline materials that can better withstand damage and
corrosion.
The Committee on Science, Engineering, and Public Policy--a joint
committee of the National Academy of Sciences, the National Academy of
Engineering, and the Institute of Medicine--has recommended the use of
expert review to determine whether a research program is focused on the
subjects most relevant to an agency's mission.[Footnote 14] Under this
form of review, experts in related fields as well as potential users of
the research evaluate the relevance of research to an agency's mission
and goals and its potential value to intended users.
OPS has used expert review to help it develop a research agenda that is
aligned with its mission and goals. At its November 2001 R&D planning
workshop, it asked a variety of experts as well as potential users of
research to identify the types of R&D that would be most likely to
enhance pipeline safety. Participants included representatives from
federal and state agencies with pipeline responsibilities, pipeline
companies and their associations, research groups, and technical
organizations that set industry safety standards for pipelines. The
agency subsequently used the results of this workshop in developing its
research agenda, guided by an R&D planning panel composed of key
experts from such groups.
OPS has also used peer review, a form of expert review, in deciding
which R&D proposals to fund, a practice that is recommended by the
Committee on Science, Engineering, and Public Policy. OPS's review
panels have included representatives from other federal agencies that
conduct pipeline R&D, industry associations, and associations of state
agencies with pipeline safety responsibilities.
The Pipeline Safety Improvement Act, enacted in December 2002, requires
that the Secretary of Transportation consult with a variety of groups
in preparing a 5-year plan for pipeline safety R&D, which must be
provided to Congress by December 2003. In response, OPS is continuing
to involve various experts and stakeholders in its R&D planning. Agency
officials have told us that, in preparation for developing this 5-year
plan, they are in the process of obtaining updated external views in
order to reassess research priorities. This has involved participating
in the pipeline R&D planning efforts of industry associations and
research organizations, discussing R&D priorities with state agency
officials, and reconvening their R&D planning panel of outside experts.
In developing the plan, agency officials also plan to consult with
OPS's two technical advisory committees. Finally, OPS plans to hold
another R&D workshop during the winter of 2003-04.
Coordination among federal agencies that conduct related research helps
to avoid duplication and ensure that each agency performs research that
is aligned with its particular mission and goals. The Committee on
Science, Engineering, and Public Policy has recommended that agencies
establish a formal process for coordinating similar fields of research,
in order to improve collaboration, help keep important questions from
being overlooked, and avoid duplication of effort.[Footnote 15] Since
2001, OPS has increased efforts to coordinate pipeline R&D with DOE and
the Department of the Interior's MMS, both of which also conduct
research related to pipelines. This increased coordination has taken
the form of mutual participation in panels that review R&D proposals
and workshops to plan R&D activities. According to OPS officials,
officials of these agencies have used these opportunities to
communicate about their respective pipeline R&D efforts and avoid
duplication. However, these agencies have not had a formal mechanism in
place that defines each agency's responsibilities for pipeline R&D.
The Pipeline Safety Improvement Act requires that the heads of DOT,
DOE, and the National Institute of Standards and Technology develop a
memorandum of understanding to formally coordinate pipeline R&D
efforts. (Although the institute does not operate an R&D program
focused on pipelines, the act assigned it a key role in pipeline R&D
based on its expertise in materials research.) In response, OPS, DOE,
and the institute have developed such a memorandum and are in the
process of finalizing it.[Footnote 16] The Pipeline Safety Improvement
Act also requires that DOT coordinate with DOE and the National
Institute of Standards and Technology in developing a 5-year plan for
pipeline R&D. In response, OPS is involving DOE and the institute, as
well as MMS, in efforts to develop such a plan. These agencies are also
considering holding joint workshops on pipeline R&D in the future. In
addition, OPS and the National Institute of Standards and Technology
have started to participate in each others' proposal review panels and
are discussing entering into an agreement to have the institute conduct
some research on pipeline materials.
We asked a number of key experts and stakeholders for their views on
the extent to which OPS's R&D agenda is aligned with its mission and
goals. These individuals included officials in DOE and MMS,
representatives of four industry associations, a former head of a state
agency that regulates gas pipelines, the heads of two leading pipeline
R&D organizations, two foremost technical experts in pipeline safety,
and an environmentalist active in pipeline safety. Six of these
individuals have been or are members of OPS advisory committees or R&D
planning or review panels. They generally told us that, in their view,
the agency has chosen to fund appropriate areas.
Experts Generally Support OPS's R&D Priorities:
The pipeline safety R&D priorities of the experts who completed our
questionnaire are generally consistent with OPS's R&D priorities. Of
the three main R&D areas that OPS is currently funding, Damage
Prevention and Leak Detection received the most scores of high or very
high funding priority; Enhanced Operations, Controls, and Monitoring
received the second highest number of such scores; and Improved
Materials Performance received the third highest number. This ranking
corresponds to the relative levels of funding OPS has assigned to these
areas, as described in the previous section. However, the experts'
level of support for Improved Materials Performance was much lower than
that for the other two main R&D areas that OPS is funding. OPS
officials told us that they are currently updating their research
agenda, using the input of experts and stakeholders, and that they will
consider our questionnaire results in this process.
To obtain the views of experts on pipeline safety R&D priorities, we
asked 55 experts to complete a questionnaire indicating the funding
priority they would assign to various types of pipeline safety R&D,
using categories identified as part of OPS's 2001 R&D planning
workshop. Table 1 provides a description of the main categories of R&D
we asked experts to prioritize. The first three categories correspond
to the main areas of R&D that OPS is currently funding. Although the
fourth category--Arctic and Offshore Technologies--was identified as a
main area of pipeline R&D at its workshop, OPS decided not to include
it as a main area in its R&D agenda. Agency officials told us that they
made this decision because R&D related to Arctic and Offshore
Technologies was not considered to be a high priority by participants
at its workshop and because MMS funds some R&D in this area and is the
primary offshore regulator. We did not include Mapping and Information
Systems--an area that OPS is currently funding from its R&D budget--as
a category for the experts to rate because it was not identified as a
main category of R&D at the 2001 workshop.
Table 1: Major Categories of R&D Related to Pipeline Safety:
Category of R&D: Damage Prevention and Leak Detection; Description:
Develop new technologies to prevent third-party damage, detect pipeline
defects, and quickly and accurately locate and control pipeline leaks.
Category of R&D: Enhanced Operations, Controls, and Monitoring;
Description: Improve technology for operating, controlling, and
monitoring the integrity of pipelines to help identify and prioritize
pipeline safety problems and solutions.
Category of R&D: Improved Materials Performance; Description: Improve
pipeline materials to extend the integrity and lifetime of installed
pipelines and their various components.
Category of R&D: Arctic and Offshore Technologies; Description: Develop
safer, more cost-effective materials and procedures to support Arctic
and offshore pipeline applications.
Sources: Materials from OPS's 2001 R&D planning workshop and other OPS
documents related to pipeline safety R&D.
[End of table]
Figure 4 shows how the 49 experts who completed our questionnaire rated
the four categories of pipeline safety R&D. We also asked experts to
rate specific types of R&D within each category. (See app. I for how
the experts rated specific types of R&D within these main categories
and for information on the agency's funding of these specific types of
R&D. See app. II for information on our methodology for selecting
experts and obtaining their views.):
The experts who completed our questionnaire strongly supported the
Damage Prevention and Leak Detection and Enhanced Operations, Controls,
and Monitoring categories of R&D as important areas for OPS to fund.
Ninety-two percent of the experts (45 of 49) indicated that the Damage
Prevention and Leak Detection category should receive high or very high
funding priority.[Footnote 17] Within this category, experts assigned
the most scores of high or very high funding priority to the following
types of R&D: improvements in the ability of in-line inspection tools,
such as "smart pigs," to detect damage and defects (39 of 49), and the
development of new technologies, such as the innovative application of
ultrasonics, that can be used for inspecting pipelines (38 of 49).
Several experts we interviewed highlighted the need to improve methods
for detecting damage to pipelines, citing the fact that third-party
damage is the leading cause of pipeline accidents. According to both
liquid and gas pipeline associations, current inspection tools cannot
reliably detect such damage to pipelines.
Figure 4: Expert Ratings of Categories of Pipeline Safety R&D:
[See PDF for image]
Note: Percentages are based on 49 respondents.
[End of figure]
Eighty percent of the experts (39 of 49) indicated that the Enhanced
Operations, Controls, and Monitoring category should receive high or
very high funding priority. Within this category, the type of R&D that
received the most scores of high or very high funding priority (37 of
49) was the improvement of alternative inspection techniques, called
direct assessment, to identify corrosion and other defects in pipelines
that cannot accommodate in-line inspection devices known as smart pigs.
This is a significant issue for natural gas pipelines because the
majority of these pipelines cannot currently accommodate smart pigs,
which are typically used to assess the condition of liquid pipelines.
In contrast to the experts' views on the importance of these first two
categories, less than one-third of the experts considered the remaining
two categories of R&D, Improved Materials Performance and Arctic and
Offshore Technologies, to be a high priority for OPS to fund. Thirty-
one percent of the experts (15 of 49) assigned scores of high or very
high funding priority to the Improved Materials Performance category,
and 20 percent (10 of 49) assigned such scores to the Arctic and
Offshore Technologies category. However, within the category of
Improved Materials Performance, about half (25 of 49) of the experts
indicated that the type of R&D aimed at developing damage-and defect-
resistant materials should receive high or very high funding priority.
Such materials could be used in the replacement of existing pipe or in
the installation of new pipe. One researcher we interviewed noted that
such materials are particularly important for the gas pipeline
industry, which is expanding its infrastructure in response to
increased demands for natural gas. One industry association estimates
that the natural gas industry will need to install about 49,500 miles
of transmission pipeline from 2001 through 2015 to meet increased
demand in the United States.
Some differences exist in the views of experts from the following three
subgroups: (1) federal and state government and public interest
organizations, (2) pipeline industry and technical and consulting
organizations, and (3) research organizations.[Footnote 18] As shown in
table 2, experts from all three subgroups generally gave the category
of Damage Prevention and Leak Detection the highest ranking, followed
by the category of Enhanced Operations, Controls, and Monitoring.
However, experts from research organizations considered the categories
of Improved Materials Performance and Arctic and Offshore Technologies
to be more important for OPS to fund than did experts from the other
two subgroups. For example, 70 percent of experts from research
organizations (7 of 10) rated Improved Materials Performance as a high
or very high priority compared with 19 percent of experts from
government and public interest organizations (3 of 16) and 22 percent
of experts from pipeline industry and technical and consulting
organizations (5 of 23). In addition, 60 percent of the researchers (6
of 10) rated Arctic and Offshore Technologies as a high or very high
priority for OPS compared with 19 percent of experts from government
and public interest organizations (3 of 16) and only 4 percent of
experts from pipeline industry and technical and consulting
organizations (1 of 23).
Table 2: Views of Experts from Three Subgroups on Pipeline Safety R&D
Priorities:
Category of R&D: Damage Prevention and Leak Detection; Number of
experts who assigned a high or very high funding priority to category:
Experts from government and public interest organizations (16): 16;
Number of experts who assigned a high or very high funding priority to
category: Experts from pipeline industry and technical and consulting
organizations (23): 20; Number of experts who assigned a high or very
high funding priority to category: Experts from research organizations
(10): 9.
Category of R&D: Enhanced Operations, Controls, and Monitoring; Number
of experts who assigned a high or very high funding priority to
category: Experts from government and public interest organizations
(16): 13; Number of experts who assigned a high or very high funding
priority to category: Experts from pipeline industry and technical and
consulting organizations (23): 18; Number of experts who assigned a
high or very high funding priority to category: Experts from research
organizations (10): 8.
Category of R&D: Improved Materials Performance; Number of experts who
assigned a high or very high funding priority to category: Experts from
government and public interest organizations (16): 3; Number of experts
who assigned a high or very high funding priority to category: Experts
from pipeline industry and technical and consulting organizations (23):
5; Number of experts who assigned a high or very high funding priority
to category: Experts from research organizations (10): 7.
Category of R&D: Arctic and Offshore Technologies; Number of experts
who assigned a high or very high funding priority to category: Experts
from government and public interest organizations (16): 3; Number of
experts who assigned a high or very high funding priority to category:
Experts from pipeline industry and technical and consulting
organizations (23): 1; Number of experts who assigned a high or very
high funding priority to category: Experts from research organizations
(10): 6.
Source: GAO.
[End of table]
An OPS official told us that he believed that researchers rated the
Improved Materials Performance category more highly than other experts
did because researchers have the best and most current information
about the "state of the art" in technology development and are more
aware of opportunities in this area. A leading expert from a pipeline
research organization noted that the foundation of pipeline R&D has
been the development of defect-resistant steels and that, as a
consequence, researchers in this area are very interested in R&D that
will lead to further improvements in the performance of pipeline
materials. He also explained that researchers may have rated the Arctic
and Offshore Technologies category more highly than the other types of
experts who completed our questionnaire because researchers may be more
aware of the need for such R&D to support the construction of new
pipelines in these areas in order to reach new energy supplies.
OPS Lacks a Systematic Process for Evaluating R&D Outcomes:
Although OPS has received significant increases in funding for its R&D
program in recent years, the agency has not developed a systematic
process for evaluating the effectiveness of its R&D program. For
example, the agency tracks and disseminates information on the progress
of individual R&D projects but has not developed a process for
assessing and reporting on the results of its R&D program as a whole.
Such a process is needed to demonstrate the program's progress toward
achieving its objectives, such as the development and use of new
technologies that can improve pipeline safety. OPS has taken some
preliminary steps toward developing an evaluation process for its R&D
program and could benefit from adopting identified best practices for
systematically evaluating the outcomes of federal R&D programs. Leading
research organizations, the Office of Management and Budget, and GAO
have identified a number of such practices, including setting clear R&D
goals and measuring progress toward these goals, using expert review to
evaluate the quality of research outcomes, and reporting periodically
on evaluation results. The results of evaluations can be used to
refocus the direction of R&D programs periodically, as necessary, to
ensure that resources are most effectively utilized.
OPS's Efforts to Evaluate Research Outcomes Have Been Limited:
Although OPS has funded R&D to develop pipeline safety technologies
since the mid-1990s, the agency's efforts to evaluate the outcomes of
this R&D have been limited and have focused on individual
projects.[Footnote 19] OPS's R&D contracts define project goals and
require research performers to meet specific milestones for the
development of a technology. Contracts also require research performers
to report quarterly and at the end of the project on results, including
milestones achieved and patents applied for and received. OPS has made
some efforts to disseminate the results to date of individual R&D
projects. For example, it has started to put "success stories" on its
Web site that describe achievements in ongoing projects, such as the
development of product prototypes. These success stories help to
communicate the results of individual projects to industry and other
interested parties.
At the program level, OPS has not yet established specific quantifiable
goals for its R&D program or a method for measuring progress toward
these goals. OPS has indicated, in various planning documents, that its
R&D program will help achieve its performance goals of reducing the
impacts of pipeline incidents, including fatalities and injuries, and
reducing spills of hazardous material. However, agency officials have
acknowledged that it is difficult to show the effect of the R&D program
on these performance goals. A more immediate objective of the program,
according to agency plans, is to promote the transfer of new and
improved pipeline safety technologies to the market in the near term.
In deciding which R&D proposals to fund, OPS gives preference to those
that plan to bring a new product to market within 5 years. In addition,
agency officials told us that OPS plans to promote the use of new
technologies by providing information to potential users and its state
partners about them and, when appropriate, by encouraging their use
through regulation.[Footnote 20] Agency officials told us that the R&D
program aims to have 80 percent of its projects result in products on
the market within 5 years. Such an objective is specific and
measurable, but OPS has not formally established it as a goal in any
plan or developed a method for measuring progress toward achieving it.
Furthermore, since the agency has not yet established specific goals or
outcome measures for its R&D program, it does not have a process for
documenting and reporting on the extent to which this program is
achieving its goals.
OPS officials explained that they have not yet developed a process for
evaluating the outcomes of the agency's R&D program because, prior to
2001, the program's budget was relatively low and, since restructuring
the program in 2001, they have focused program efforts on building a
process for setting research priorities. However, officials do
recognize the need for evaluating R&D outcomes and have taken some
preliminary steps toward developing an evaluation process for their R&D
program.
OPS is considering some possible measures of the outcomes of its R&D
program as a whole, such as the number of new patents resulting from
R&D efforts. In addition, agency officials told us that, although
tracking the transfer to the market of new pipeline safety technologies
can be challenging, OPS intends to track the use of new technologies in
the future through its process for inspecting operators' "integrity
management" programs.[Footnote 21] For example, OPS inspectors could
document the use of new or improved technologies by companies to
evaluate the condition of their pipelines. Agency officials noted that
the agency will develop inspection protocols that require inspectors to
collect data on the use of new technologies after their proposed
integrity management rule for natural gas transmission pipelines is
finalized.
OPS is also considering the number of documented R&D "success stories"-
-summaries of the accomplishments of individual R&D projects--as a
possible measure of program results. However, in previous reviews of
R&D programs operated by other federal agencies, we have found that the
success story approach is selective and does not adequately assess
programwide performance.[Footnote 22]
In early June 2003, OPS presented a potential set of performance
measures for its R&D program to its R&D planning panel of outside
experts in order to obtain their views on these measures. This panel
includes representatives of DOE, MMS, the National Institute of
Standards and Technology, pipeline industry associations, state
agencies with pipeline responsibilities, and a key pipeline research
organization. OPS intends to refine its set of measures based on
comments received from this panel and to continue obtaining the views
of this panel as it moves forward in developing an evaluation process
for its R&D program.
Finally, OPS officials also told us that the agency intends to obtain
the views of experts on its R&D outcomes as well as on its future R&D
priorities at its next R&D workshop, scheduled for the winter of 2003-
04. However, OPS is in the beginning stages of planning this workshop
and has not defined a process for using experts' views to evaluate the
outcomes of its R&D program.
OPS officials told us that they are considering including information
on the effectiveness of the agency's R&D program in the annual reports
to Congress on pipeline R&D that the agency is required to submit,
starting in December 2003. The Pipeline Safety Improvement Act requires
that DOT, DOE, and the National Institute of Standards and Technology
jointly provide these annual reports to Congress, but does not fully
specify what types of information should be included in these reports.
Best Practices Help Agencies Systematically Evaluate Research Outcomes:
Since OPS is in the beginning stages of developing an evaluation
process for its R&D program, it could benefit from adopting best
practices for systematically evaluating federal R&D programs. Leading
organizations that conduct scientific and engineering research, the
Office of Management and Budget, and GAO have identified a number of
these best practices. Although the uncertain nature of research
outcomes over time can make it challenging to demonstrate the results
of such R&D programs, these practices are designed to enable agencies
to systematically assess and report on these results regularly in
accordance with the Government Performance and Results Act of
1993.[Footnote 23] These assessments can be used to refocus the
direction of R&D programs periodically, as necessary, to ensure that
resources are most effectively utilized. Identified best practices are
discussed in the following sections.
Setting Clear, Quantifiable Goals and Measuring Progress toward These
Goals:
We have previously reported that, to be effective, any R&D program must
be directed toward a clear, measurable goal.[Footnote 24] Such goals
help ensure a direct linkage between R&D program efforts and an
agency's overall performance goals and mission. Applied research
programs, such as OPS's R&D program, are directed toward achieving
specific useful outcomes, such as the development of new technologies,
which can help accomplish agency performance goals. The Committee on
Science, Engineering, and Public Policy recommended in a 1999 report
that agencies operating applied research programs measure progress
toward practical outcomes and noted that such measurement can usually
be performed annually using milestones.[Footnote 25]
Similarly, in May 2002 the Office of Management and Budget established
investment criteria for federal R&D programs that require these
programs to clearly define goals and track progress toward these goals
using appropriate outcome measures and interim milestones. Indicators
that have been used to measure the outcomes of R&D include the
achievement of specific targets for developing new or improved
technologies and patent applications filed and granted.[Footnote 26]
However, measuring research outcomes can be challenging. For example,
outcomes may not occur for a number of years and may be difficult to
track.
Using Expert Review to Evaluate the Quality of Research Outcomes:
In its 1999 report and again in 2001, the Committee on Science,
Engineering, and Public Policy recommended the use of expert review,
supplemented by quantitative methods, to evaluate research
regularly.[Footnote 27] Expert review can be a useful addition to
performance measures because of the value of the reviewers' deep
knowledge of the field. Such review can be performed on a somewhat
longer term basis, rather than annually, and does not require that the
final impact of the research be known. Peer review, a form of expert
review, includes an independent assessment of the technical and
scientific merit or quality of research by peers with essential subject
matter expertise and perspective equal to that of the researchers. In
1999, we reported that some federal agencies, such as the Department of
Agriculture, the National Institutes of Health (NIH), and DOE, use peer
review to help them evaluate the performance of programs and determine
whether to continue or renew research projects.[Footnote 28]
The Committee on Science, Engineering, and Public Policy reported in
2001 on the use of expert review, including peer review, by NIH, DOE,
the National Science Foundation, the Department of Defense, and the
National Aeronautics and Space Administration to evaluate the quality
of their research programs. These agencies used varying methods for
carrying out this review, including convening panels of experts who use
defined evaluation processes and obtaining the views of external
advisory committees. The Committee on Science, Engineering, and Public
Policy has also noted that expert evaluation of applied research
programs requires the input of potential users of the results of the
research, since the ultimate usability of these results is an important
factor in determining the worth of the research. Similarly, key experts
and stakeholders we interviewed noted that the degree to which new
technologies are actually used would be a good indication of the
effectiveness of OPS's R&D program. One industry association
representative we interviewed noted that a "constant theme" raised by
pipeline companies is the need for R&D efforts to produce new
technologies that they can actually use in operating their pipelines.
Reporting Periodically on Evaluation Results:
Periodic reporting by applied research programs on results can help
keep key stakeholders--including oversight organizations and potential
users of new technologies--up-to-date on program accomplishments.
According to the Committee on Science, Engineering, and Public Policy,
applied research programs can usually report annually on progress in
meeting milestones. In addition, a retrospective analysis over several
years is needed to evaluate outcomes that take more than 1 year to
emerge. The committee also has recommended that agencies demonstrate
the value of their review processes by publicly describing them to
oversight groups, the potential users of research results, and the
general public. One expert we interviewed stressed the importance of
periodic public reporting by OPS on research goals and outcomes and on
the method for evaluating outcomes, in order to disseminate research
results and build support for its R&D program.
Conclusions:
OPS has made significant progress in establishing a pipeline safety
research agenda that is aligned with its mission and goals and that
incorporates the views of experts and stakeholders. However, without a
systematic process for evaluating the outcomes of its R&D program, the
agency is not able to demonstrate that it is effectively using its
increased resources for R&D to foster new and improved technologies
that can enhance pipeline safety. Identified best practices for
evaluating federal R&D programs--including setting clear quantifiable
R&D goals and measuring progress toward these goals, using expert
review to evaluate the quality of research outcomes, and reporting
periodically on evaluation results--can guide OPS as it moves forward
in developing an evaluation process for its program. By following such
practices, the agency can help ensure that it develops a systematic
evaluation process that will enable it to determine and demonstrate the
results of its investment in pipeline safety R&D. OPS could use such an
evaluation process to periodically refocus the direction of its program
in order to make the most effective use of resources.
Furthermore, although the Pipeline Safety Improvement Act's requirement
for annual reports on pipeline R&D, starting in December 2003, does not
specify in detail what information should be included in these reports,
this requirement provides an opportunity for the agency to keep
Congress informed about the results of evaluations of its R&D program.
In addition, such reporting, along with other communication methods
already in use by the agency, can keep other interested parties--
including the pipeline industry, state pipeline safety agencies,
pipeline safety advocates, and researchers--up-to-date on the program's
progress in advancing the most promising pipeline safety technologies.
Recommendations for Executive Action:
To improve OPS's ability to demonstrate the effectiveness of its R&D
program and make the most effective use of program funds, we recommend
that the Secretary of Transportation direct OPS to:
* develop a systematic process for evaluating the outcomes of its R&D
program that incorporates identified best practices and:
* include in the annual reports to Congress, which are required by the
Pipeline Safety Improvement Act, information on the results of R&D
evaluations.
Agency Comments:
We provided DOT with a draft of this report for review and comment. DOT
officials, including OPS's Director of Program Development, provided
oral comments on the draft on June 13, 2003. The officials generally
agreed with the report's findings and conclusions. They emphasized that
they are starting to develop a framework for evaluating the
effectiveness of their pipeline safety R&D program and that they intend
to finalize this framework by December 2003 by documenting it in the 5-
year plan and first annual report on pipeline R&D that DOT is required
to submit to Congress, jointly with DOE and the National Institute of
Standards and Technology. They also noted that they agree with and
intend to implement our recommendations and provided some technical
clarifications, which we have incorporated as appropriate.
We are sending copies of this report to the Secretary of
Transportation, the Administrator of RSPA, RSPA's Associate
Administrator for Pipeline Safety, the Director of the Office of
Management and Budget, and appropriate congressional committees. We
will make copies available to others upon request. In addition, this
report will be available at no charge on the GAO Web site at http://
www.gao.gov.
If you or your staffs have any questions about this report, please
contact me at (202) 512-2834 or guerrerop@gao.gov. Individuals making
key contributions to this report are listed in appendix III.
Signed by:
Peter Guerrero
Director,
Physical Infrastructure Issues:
[End of section]
Appendixes:
Appendix I: Experts' Views on R&D Priorities and OPS's R&D Funding, by
Type of R&D:
We asked selected experts to review the following descriptions of
specific types of pipeline safety research and development (R&D) and
assign a funding priority to each, based on its importance in achieving
the Office of Pipeline Safety's (OPS) mission of ensuring the safe,
reliable, and environmentally sound operation of the nation's pipeline
transportation system. Experts used the following scale: 1=little or no
funding, 2=some funding priority, 3=moderate funding priority, 4=high
funding priority, and 5=very high funding priority. Experts could also
indicate that they did not know or had no basis to judge the funding
priority for a particular type of R&D. The following table shows, for
each type of R&D, the number of experts who assigned it a high or very
high funding priority and OPS's current and planned allocation of
funding to it. A total of 49 experts completed our questionnaire.
Type of R&D: Damage Prevention and Leak Detection:
Type of R&D: In-line inspection for damage and defects: Improve in-line
inspection techniques, including "smart pigs" and other technologies,
for detecting and measuring damage, cracking, and defects in pipe
walls; Number of experts who assigned it a high or very high funding
priority: 39; OPS's current and planned allocation of
funding to this type of R&D[A]:
Allocated $592,500 to five projects in November 2002 for periods of 9
to 24 months.
Type of R&D: Nondestructive evaluation: Develop new approaches or
technologies, such as the innovative application of ultrasonics, that
can be used for the nondestructive evaluation of operational pipelines;
Number of experts who assigned it a high or very high funding priority:
38; OPS's current and planned allocation of funding to
this type of R&D[A]: Allocated
$500,000 to one project in November 2002 for a period of 24 months.
Type of R&D: Real-time monitoring using sensors attached to pipe:
Develop and test real-time sensors applied or attached to the pipe that
can detect possible third-party contact, leaks, or other signs of
damage; Number of experts who assigned it a high or very high funding
priority: 27; OPS's current and planned allocation of
funding to this type of R&D[A]:
Allocated $182,000 to one project in April 2001 for period of 12
months. Requested proposals in March 2002 but did not fund any of those
received. Requested additional proposals in December 2002 and plans to
make funding decisions in July 2003.
Type of R&D: Small leak detection: Improve technologies for quickly
detecting small pipeline leaks; Number of experts who assigned it a
high or very high funding priority: 22; OPS's
current and planned allocation of funding to this type of R&D[A]:
Requested proposals in December
2002 and plans to make funding decisions in July 2003.
Type of R&D: Pipe location: Develop better techniques or materials to
locate steel and plastic pipelines, including determining their depth;
Number of experts who assigned it a high or very high funding priority:
17; OPS's current and planned allocation of funding to
this type of R&D[A]: Allocated
$534,521 to two projects in July 2002 for periods of 23 to 24 months.
Type of R&D: Encroachment monitoring using satellites: Develop
satellite monitoring for encroachment and ground movement; Number of
experts who assigned it a high or very high funding priority: 16;
OPS's current and planned allocation of funding to
this type of R&D[A]: Requested
proposals in March 2002 but did not fund any of those received.
Requested additional proposals in December 2002 and plans to make
funding decisions in July 2003.
Type of R&D: Improved directional drilling: Improve directional
drilling techniques to avoid accidental damage to other underground
utilities; Number of experts who assigned it a high or very high
funding priority: 14; OPS's current and planned
allocation of funding to this type of R&D[A]: Requested proposals in
March 2002 but did not fund any
of those received. Requested additional proposals in December 2002 and
plans to make funding decisions in July 2003.
Type of R&D: Real-time right-of-way monitoring without pipe contact:
Develop fiber optic lines that can be buried above or alongside
pipeline to detect nearby movement; Number of experts who assigned it
a high or very high funding priority: 12; OPS's
current and planned allocation of funding to this type of R&D[A]:
Requested proposals in March 2002
but did not fund any of those received. Requested additional proposals
in December 2002 and plans to make funding decisions in July 2003.
Type of R&D: Airborne chemical mapping: Develop approaches using aerial
surveillance with optical technologies for right-of-way monitoring or
other pipeline safety concerns; Number of experts who assigned it a
high or very high funding priority: 11; OPS's
current and planned allocation of funding to this type of R&D[A]:
Allocated $600,000 to one project
in April 2001 for a period of 12 months. Allocated an additional
$600,000 to this project in April 2002 for an additional 12 months.
Plans to allocate an additional $600,000 to this project in May 2003
for an additional 12 months.[B].
Type of R&D: Enhanced Operations, Controls, and Monitoring:
Type of R&D: Direct assessment: Improve alternative inspection
techniques for "unpiggable" pipelines to identify internal and
external corrosion, third-party damage, and other pipe defects; Number
of experts who assigned it a high or very high funding priority:
Damage Prevention and Leak Detection: 37; OPS's current and planned
allocation of funding to this type of R&D[A]: Allocated
$572,000 to two projects in January 2003 for periods of 12 to 26
months.
Type of R&D: External corrosion control: Improve techniques for
characterizing, detecting, and preventing external corrosion damage;
Number of experts who assigned it a high or very high funding priority:
30; OPS's current and planned allocation of funding to
this type of R&D[A]: Allocated
$297,000 to one project in January 2003 for a period of 26 months.
Type of R&D: Internal corrosion control: Improve techniques for
characterizing, detecting, and preventing internal corrosion damage;
Number of experts who assigned it a high or very high funding priority:
30; OPS's current and planned allocation of funding to
this type of R&D[A]: Allocated
$275,000 to one project in January 2003 for a period of 12 months.
Type of R&D: Stress corrosion cracking detection: Improve techniques
for characterizing, detecting, and preventing stress corrosion
cracking; Number of experts who assigned it a high or very high funding
priority: 25; OPS's current and planned allocation of
funding to this type of R&D[A]:
Allocated $675,281 to four projects in May and July 2002 for periods of
12 to 24 months.
Type of R&D: Enhanced repair techniques: Develop enhanced repair
techniques that can be implemented without shutdown of pipeline; Number
of experts who assigned it a high or very high funding priority: 25;
Detection: OPS's current and planned allocation of funding to
this type of R&D[A]: [Empty].
Type of R&D: Risk assessment: Enhance techniques to integrate and
evaluate risk data to define pipe susceptibility to various threats;
Number of experts who assigned it a high or very high funding priority:
25; OPS's current and planned allocation of funding to
this type of R&D[A]: Allocated
$97,737 to three projects in May 2002 for a period of 12 months.
Allocated $70,000 to an additional project in January 2003 for a period
of 24 months. Requested additional proposals in December 2002 and plans
to make funding decisions in July 2003.
Type of R&D: Pipe strength: Improve methods for characterizing
remaining pipe strength; Number of experts who assigned it a high or
very high funding priority: 17;
OPS's current and
planned allocation of funding to this type of R&D[A]: Requested
proposals in December 2002 and plans to make funding decisions in
July 2003.
Type of R&D: Human factors: Study human factors, such as operator
fatigue, that influence pipeline integrity and develop technologies or
procedures to minimize operator error; Number of experts who assigned
it a high or very high funding priority: 7; OPS's
current and planned allocation of funding to this type of R&D[A]:
Requested proposals in June 2002
but did not fund any of those received. Requested additional proposals
in December 2002 and plans to make funding decisions in July 2003.
Type of R&D: Trenchless pipe installation: Develop trenchless pipe
installation and replacement techniques, including techniques that use
directional drilling or robotics; Number of experts who assigned it a
high or very high funding priority: 5; OPS's
current and planned allocation of funding to this type of R&D[A]:
[Empty].
Type of R&D: Improved Materials Performance:
Type of R&D: Damage-and defect-resistant materials: Develop materials
that better withstand third-party damage, corrosion, and cracking;
Number of experts who assigned it a high or very high funding priority:
25; OPS's current and planned allocation of funding to
this type of R&D[A]: Requested
proposals in December 2002 and plans to make funding decisions in July
2003.
Type of R&D: Pipe coatings: Develop enhanced field-and factory-applied
coatings, methods for testing coatings, and methods to improve coating
choices; Number of experts who assigned it a high or very high funding
priority: 15; OPS's current and planned allocation of
funding to this type of R&D[A]: Requested proposals in December 2002
and plans to make funding decisions in July 2003.
Type of R&D: Higher grade/strength steels: Develop higher grade/
strength steels, evaluate their performance, and develop methods for
determining when to use them; Number of experts who assigned it a high
or very high funding priority: 14; OPS's current and
planned allocation of funding to this type of R&D[A]: Requested
proposals in December 2002 and plans to make funding decisions in
July 2003.
Type of R&D: Welding and joining: Develop enhanced welding and joining
techniques and improved methods for assessing performance of welds and
joints; Number of experts who assigned it a high or very high funding
priority: 13; OPS's current and planned allocation of
funding to this type of R&D[A]:
Requested proposals in December 2002 and plans to make funding
decisions in July 2003.
Type of R&D: Higher design pressure: Develop materials that facilitate
pipelines operating at higher design pressures and methods for
determining when to use higher pressure designs; Number of experts who
assigned it a high or very high funding priority: 11;
OPS's current and planned allocation of funding to this type of R&D[A]:
Requested proposals in December
2002 and plans to make funding decisions in July 2003.
Type of R&D: Composite pipe: Develop pipe made of, or layered with,
materials other than steel that may exceed current performance
standards or allow greater flexibility or lower cost in challenging
installation conditions; Number of experts who assigned it a high or
very high funding priority: 11;
OPS's current and
planned allocation of funding to this type of R&D[A]: Allocated $98,680
to one project in November 2002
for a period of 6 months. Requested additional proposals in December
2002 and plans to make funding decisions in July 2003.
Type of R&D: Plastic pipe: Develop new or improved plastic pipe for
local distribution company systems; Number of experts who assigned it
a high or very high funding priority: 11; OPS's
current and planned allocation of funding to this type of R&D[A]:
Requested proposals in December
2002 and plans to make funding decisions in July 2003.
Type of R&D: Arctic and Offshore Technologies.
Type of R&D: Leak detection: Develop approaches to detect, verify, and
respond to leaks; Number of experts who assigned it a high or very high
funding priority: 17; OPS's current and planned
allocation of funding to this type of R&D[A]: Allocated $7,781 to one
project in May 2002 for a period of 12 months.
Type of R&D: Inspection and maintenance procedures: Develop alternative
inspection and maintenance technologies and procedures; Number of
experts who assigned it a high or very high funding priority: 8; OPS's
current and planned allocation of funding to this type of R&D[A]:
Allocated $50,000 to one project in May 2001 for a period of 12 months.
Type of R&D: Enhanced performance: Develop materials and fabrication
techniques to enhance low temperature performance; Number of experts
who assigned it a high or very high funding priority: 8;
OPS's current and planned allocation of funding to this type of R&D[A]:
Allocated $59,955 to one project
in May 2002 for a period of 12 months.
Type of R&D: Site evaluation: Develop improved techniques for site
evaluation; Number of experts who assigned it a high or very high
funding priority: 5; OPS's current and planned
allocation of funding to this type of R&D[A]: [Empty].
Type of R&D: Other Pipeline Safety Improvements.
Type of R&D: Inspection tools: Evaluate and quantify, where possible,
the strengths, limits, and performance of current inspection tools;
Number of experts who assigned it a high or very high funding priority:
28; OPS's current and planned allocation of funding to
this type of R&D[A]: [Empty].
Type of R&D: Pipeline modeling enhancements: Develop better
mathematical or computational modeling techniques to improve ability to
detect defects, including growth defects and small leaks; Number of
experts who assigned it a high or very high funding priority: 24;
Detection: OPS's current and planned allocation of funding to
this type of R&D[A]: Requested
proposals in December 2002 and plans to make funding decisions in July
2003.
Type of R&D: Impact of multiple utilities: Characterize impact of
multiple utilities in common right-of-way on integrity management
practices, such as cathodic protection; Number of experts who assigned
it a high or very high funding priority: 18; OPS's
current and planned allocation of funding to this type of R&D[A]:
[Empty].
Type of R&D: Higher stress levels: Evaluate potential for current
piping to operate at higher stress levels; Number of experts who
assigned it a high or very high funding priority: 17;
OPS's current and planned allocation of funding to this type of R&D[A]:
[Empty].
Type of R&D: Reduction of rupture impact: Explore means to reduce the
impact of a pipeline rupture and explosion, for example, through
additives to gas/liquid or enhanced shutoff capability; Number of
experts who assigned it a high or very high funding priority: 15;
Detection: OPS's current and planned allocation of funding to
this type of R&D[A]: [Empty].
Type of R&D: Impact of past releases: Research the impact of past
pipeline releases on their surrounding areas and provide information
that could be used to support local zoning decisions; Number of experts
who assigned it a high or very high funding priority: 7;
OPS's current and planned allocation of funding to this type of R&D[A]:
[Empty].
Sources: OPS data; GAO analysis.
Note: The pipeline safety R&D categories were identified as part of
OPS's 2001 R&D Planning Workshop. The descriptions of the categories
are based on materials from this workshop as well as OPS's 2002 and
2003 announcements soliciting R&D proposals. The information on OPS's
funding of each category is based on GAO's analysis of information
provided by OPS.
[A] This column describes OPS's funding of R&D projects, by category,
in fiscal years 2001 and 2002 and planned allocation of funding in
fiscal year 2003. Some projects that are applicable to more than one
category of R&D appear more than once.
[B] In conference reports accompanying appropriations bills for fiscal
years 2001, 2002, and 2003, Congress expressed its intent that the
Research and Special Programs Administration devote $600,000 of its
pipeline safety R&D budget to this project in each of these fiscal
years.
[End of table]
[End of section]
Appendix II: Scope and Methodology:
To perform our work, we reviewed Office of Pipeline Safety (OPS)
documentation on its research and development (R&D) funding and
analyzed this information to identify trends; reviewed pertinent
legislation and agency documents pertaining to the R&D program; and
interviewed OPS officials regarding their R&D funding, agenda-setting
processes, and processes for evaluating the outcomes of their R&D
program. We also interviewed key experts and stakeholders concerning
OPS's management of its R&D program, including the alignment of the
agency's research agenda with its mission and goals, and their views on
R&D priorities and gaps. These individuals included officials of the
Department of Energy (DOE) and the Department of the Interior's
Minerals Management Service (MMS) who are responsible for pipeline R&D;
representatives of pipeline industry associations and leading pipeline
research organizations; and several key experts in pipeline safety.
Also, we identified best practices for evaluating the outcomes of
federal R&D through a review of relevant literature and compared the
agency's processes with these practices.
To determine the views of experts on pipeline safety R&D priorities, we
sought to identify experts considered to be very knowledgeable about
the development of new pipeline safety technologies or pipeline safety
issues. To identify appropriate experts, we obtained recommendations on
individuals to contact from key organizations, contacted those
individuals, and obtained further recommendations from them on
additional individuals to contact. We identified initial individuals to
contact through prior work on pipeline safety issues or through
recommendations from OPS. These initial contacts included officials in
DOE and MMS, representatives of four industry associations, a former
head of a state agency that regulates gas pipelines, the heads of two
leading pipeline R&D organizations, two technical experts in pipeline
safety, and an environmentalist active in pipeline safety. Six of these
individuals have been or are members of OPS advisory committees or R&D
planning or review panels. We obtained recommendations from these
individuals on experts who could provide us with views on pipeline
safety R&D priorities.
We based our final selection of experts on the criteria of knowledge,
balance, and independence. We considered indications of their extent of
knowledge of pipeline safety R&D, as evidenced by the number of times
they had been recommended, their participation in OPS's R&D planning
and review activities, or other relevant factors. We included
individuals from a variety of groups in order to achieve a balanced
representation of experts, including some who are relatively
independent of OPS and the pipeline industry. We included individuals
from federal and state agencies, pipeline safety advocacy groups,
industry associations, pipeline companies, technical and consulting
organizations, and research organizations. We also provided our list of
identified experts to the National Academy of Sciences and OPS for
their review and comment.
We contacted 55 individuals whom we had identified as appropriate
experts for our review and asked them to complete a questionnaire
indicating their views on pipeline safety R&D priorities. Forty-nine
individuals responded, for an 89 percent response rate. Our results
pertaining to experts' views on R&D priorities represent the views of
only the experts who responded to our questionnaire. In a number of
cases, these individuals collaborated with others in their
organizations in completing their questionnaires. Listed below are the
organizational affiliations of experts who completed our
questionnaire.[Footnote 29]
Government and Public Interest Organizations:
Federal Agencies Federal Energy Regulatory Commission Minerals
Management Service, Department of the Interior National Institute of
Standards and Technology, Department of Commerce National
Transportation Safety Board Office of Fossil Energy, Department of
Energy:
State Agencies and Associations National Association of Pipeline Safety
Representatives National Association of Regulatory Utility
Commissioners New York State Department of Public Service Railroad
Commission of Texas Virginia State Corporation Commission Washington
Utilities and Transportation Commission:
Pipeline Safety Advocacy Groups Common Ground Alliance Cook Inlet
Keeper Safe Bellingham:
Pipeline Industry and Technical/Consulting Organizations:
Industry Associations American Gas Association American Petroleum
Institute Association of Oil Pipelines Interstate Natural Gas
Association of America Offshore Operators Committee:
Pipeline Companies BP Pipelines, North America ConocoPhillips CMS
Panhandle Companies Duke Energy El Paso Corporation Enbridge Pipelines
Enron Explorer Pipeline Company ExxonMobil Pipeline Company KeySpan
Energy Peoples Energy Shell Pipeline Company:
Technical/Consulting Organizations Accufacts, Inc. Batten and
Associates, Inc. Duckworth Pipeline Integrity Services, Inc. HSB
Solomon Kiefner and Associates, Inc. National Association of Corrosion
Engineers:
Research Organizations:
Advantica, Inc. Battelle CFER Technologies Edison Welding Institute Gas
Technology Institute Ohio State University, Fontana Corrosion Center
Pipeline Research Council International, Inc. Southwest Research
Institute Texas A&M University, Department of Mechanical Engineering
University of Florida, Department of Chemical Engineering:
In the questionnaire, we asked respondents to review descriptions of
various main categories of pipeline safety R&D as well as specific
types of R&D within these main categories and indicate what funding
priority they would assign to each.[Footnote 30] (See table 1 for
descriptions of the main R&D categories. See app. I for descriptions of
the types of R&D within these main categories.) We based the R&D
categories and descriptions on materials prepared as part of an R&D
planning workshop held by OPS in 2001, in which a variety of experts
and stakeholders participated; on announcements the agency subsequently
issued soliciting proposals for R&D in various areas; and on other OPS
documents related to pipeline safety R&D.
We compiled the scores obtained from the questionnaires to produce a
ranking of R&D priorities representing the views of the experts who
completed our survey. We also analyzed our results to determine whether
any differences existed in the responses of experts from the three
subgroups: government and public interest organizations, industry and
technical and consulting organizations, and research organizations. In
addition, we identified organizations that had bid on R&D funding from
OPS in fiscal year 2002 and conducted a separate analysis of the
responses of experts from these organizations to determine how they
compared with those of other experts who completed our questionnaire.
Seven of the experts who completed our questionnaire are from
organizations that had bid on OPS R&D funding within this time
frame.[Footnote 31]
We conducted our work from January 2003 through June 2003 in accordance
with generally accepted government auditing standards.
[End of section]
Appendix III: GAO Contacts and Staff Acknowledgments:
GAO Contacts:
Peter Guerrero, (202) 512-2834 Susan Fleming, (202) 512-4431:
Acknowledgments:
In addition to those named above, Sharon Dyer, Etana Finkler, Judy
Guilliams-Tapia, Brandon Haller, Bert Japikse, Nancy Kingsbury, Donna
Leiss, Gary Stofko, Ron Stouffer, and Stacey Thompson made key
contributions to this report.
(545025):
FOOTNOTES
[1] Figures do not add to total due to rounding.
[2] In general, OPS retains full responsibility for inspecting and
enforcing regulations on interstate pipelines but certifies states to
perform these functions for intrastate pipelines. In 2003, 49 state
agencies, the District of Columbia, and Puerto Rico were certified for
inspecting and enforcing regulations on intrastate pipelines. In
addition, OPS has agreements with 11 states to inspect segments of
interstate pipelines within their boundaries.
[3] We have previously reported on OPS's implementation of this new
regulatory approach. See U.S. General Accounting Office, Pipeline
Safety and Security: Improved Workforce Planning and Communication
Needed, GAO-02-785 (Washington, D.C.: Aug. 26, 2002), and Pipeline
Safety: The Office of Pipeline Safety Is Changing How It Oversees the
Pipeline Industry, GAO/RCED-00-128 (Washington, D.C.: May 15, 2000).
[4] These figures are based on accidents reported to OPS. For hazardous
liquid pipelines, they include accidents involving any fatality or
injury, a fire or explosion, total costs of $50,000 or more, or
releases of 50 or more barrels of hazardous liquids or 5 or more
barrels of highly volatile liquids. For natural gas pipelines, they
include accidents involving any fatality or injury, total costs of
$50,000 or more, or the emergency shutdown of a liquified natural gas
facility, as well as any accidents considered to be significant by the
pipeline operator.
[5] In March 2002, OPS requested proposals related to damage prevention
and leak detection. It received 82 proposals in response and, in
November 2002, funded 7 of them. In June 2002, the agency requested
proposals related to enhanced pipeline operations, controls, and
monitoring. It received 57 proposals in response and, in February 2003,
funded 3 of them, based on the availability of funding. OPS intends to
fund 3 more of these proposals in June 2003. OPS has provided
approximately 50 percent of the cost of the projects to awardees. In
December 2002, the agency requested proposals related to improved
performance of pipeline materials and other pipeline safety
improvements. It expects to make funding decisions about these
proposals in summer 2003.
[6] These figures have been adjusted to account for inflation. They are
in constant fiscal year 2003 dollars.
[7] In addition, RSPA's budget submission for fiscal year 2003 noted
that the proposed pipeline safety R&D budget would consolidate into
RSPA a pipeline infrastructure R&D program operated by DOE. However,
according to DOE and OPS officials, no transfer of funding or projects
between the two programs actually took place.
[8] These amounts represent OPS's planned expenditures in each area.
However, the agency's actual expenditures in an area depend on the
approval of R&D proposals received and may therefore differ from
planned expenditures. Figures do not add to total due to rounding.
[9] "Third parties" are people or companies not associated with a
pipeline company or its contractors. Damage to pipelines can result
from such people or companies digging in the vicinity of buried
pipelines without realizing that the pipelines are there. For example,
excavating equipment can accidentally strike a pipeline and cause a
leak or rupture, either immediately or over time, which poses a hazard
to life and property.
[10] Another area of pipeline R&D--the development of technologies to
support Arctic and offshore pipeline operations--was identified as a
main area of R&D at OPS's 2001 workshop. However, the agency did not
include this as a main area of funding in its R&D agenda because it was
not identified as a high-priority area at the workshop and because the
Department of the Interior's MMS funds some R&D in this area. OPS has
recently cofunded with MMS several projects and a workshop in this
area, at a cost of almost $148,000.
[11] Figures have been adjusted to account for inflation. They are in
constant fiscal year 2003 dollars.
[12] Smart pigs are devices that run inside a pipeline to detect
anomalies, such as corrosion, metal loss, or damage from excavation.
[13] Direct assessment involves several steps, including digging holes
at intervals along a pipeline to examine suspected problem areas. In a
notice of proposed rulemaking, OPS has proposed integrity management
regulations for gas transmission pipelines that would allow operators
to use direct assessment techniques. See 68 Fed. Reg. 4278, 4318 (Jan.
28, 2003). We have previously reported on the challenges faced by OPS
in ensuring that operators use these techniques appropriately. See GAO-
02-785.
[14] Committee on Science, Engineering, and Public Policy, Evaluating
Federal Research Programs: Research and the Government Performance and
Results Act (Washington, D.C.: National Academy of Sciences, February
1999).
[15] Committee on Science, Engineering, and Public Policy, Evaluating
Federal Research Programs: Research and the Government Performance and
Results Act, 11.
[16] DOT and the Department of the Interior have a memorandum of
understanding in place to coordinate their regulatory efforts regarding
outer continental shelf pipelines; this memorandum states that that the
two departments will coordinate their respective R&D projects
concerning these pipelines. In addition, OPS and MMS have an
interagency agreement to jointly fund R&D projects related to offshore
pipelines.
[17] Experts assigned a funding priority to each category and specific
type of R&D using the following scale: 1=little or no funding, 2=some
funding priority, 3=moderate funding priority, 4=high funding priority,
and 5=very high funding priority. Experts could also indicate that they
did not know or had no basis to judge the funding priority for a
particular R&D category.
[18] We also examined results for experts from those organizations that
had bid on OPS R&D funding in fiscal year 2002 to see how they compared
to those of other experts who completed our questionnaire. Seven of the
experts who completed our questionnaire are from organizations that had
bid on OPS R&D funding within this time frame. Of these, all seven
assigned scores of high or very high funding priority to the Damage
Prevention and Leak Detection category; six assigned such scores to the
Enhanced Operations, Controls, and Monitoring category; three assigned
such scores to the Improved Materials Performance category; and four
assigned such scores to the Arctic and Offshore Technologies category.
[19] We have recently reported that RSPA has not fulfilled a DOT
requirement for overseeing and developing ways to improve research
evaluation efforts throughout the department. See U.S. General
Accounting Office, Transportation Research: Action Needed to Improve
Coordination and Evaluation of Research, GAO-03-500 (Washington, D.C.:
May 1, 2003).
[20] For example, in a notice of proposed rulemaking, OPS proposed
integrity management regulations for natural gas transmission pipelines
that would allow pipeline operators to assess the integrity (structural
soundness) of their pipelines using a new technique called direct
assessment. See 68 Fed. Reg. 4278, 4318 (Jan. 28, 2003). OPS has funded
and is currently funding R&D to develop and validate this assessment
method.
[21] OPS has issued requirements for hazardous liquid pipeline
operators to develop such programs, which are aimed at assessing the
integrity (structural soundness) of pipelines and identifying and
addressing risks to segments where a leak or rupture could have
significant consequences, such as near highly populated areas. See 49
CFR § 195.452. In a notice of proposed rulemaking, the agency has
proposed such requirements for operators of natural gas transmission
pipelines. See 68 Fed. Reg. 4278 (Jan. 28, 2003).
[22] U.S. General Accounting Office, Highway Research: DOT's Actions to
Implement Best Practices for Setting Research Agendas and Evaluating
Outcomes, GAO-03-640T (Washington, D.C.: Apr. 10, 2003); Highway
Research: Systematic Selection and Evaluation Processes Needed for
Research Program, GAO-02-573 (Washington, D.C.: May 24, 2002); and
DOE's Success Stories Report, GAO/RCED-96-120R (Washington, D.C.: Apr.
15, 1996).
[23] The Government Performance and Results Act requires all federal
agencies to measure and report on the results of their activities
annually.
[24] U.S. General Accounting Office, Research and Development: Lessons
Learned from Previous Research Could Benefit FreedomCAR Initiative,
GAO-02-810T (Washington, D.C.: June 6, 2002).
[25] Committee on Science, Engineering, and Public Policy. Evaluating
Federal Research Programs: Research and the Government Performance and
Results Act.
[26] See U.S. General Accounting Office, Measuring Performance:
Strengths and Limitations of Performance Indicators, GAO/RCED-97-91
(Washington, D.C.: Mar. 21, 1997), and Intellectual Property: Federal
Agency Efforts in Transferring and Reporting New Technology, GAO-03-47
(Washington, D.C.: Oct. 31, 2002). Also, see Committee on Science,
Engineering, and Public Policy, Evaluating Federal Research Programs:
Research and the Government Performance and Results Act.
[27] Committee on Science, Engineering, and Public Policy, Evaluating
Federal Research Programs: Research and the Government Performance and
Results Act, and Implementing the Government Performance and Results
Act: A Status Report (Washington, D.C.: National Academy of Sciences,
2001).
[28] U.S. General Accounting Office, Federal Research: Peer Review
Practices at Federal Science Agencies Vary, GAO/RCED-99-99 (Washington,
D.C.: Mar. 17, 1999).
[29] Two of the individuals who responded are former officials of these
organizations.
[30] Experts assigned a funding priority to each category and
subcategory of R&D using the following scale: 1=little or no funding,
2=some funding priority, 3=moderate funding priority, 4=high funding
priority, and 5=very high funding priority. We also asked respondents
if they wished to identify any additional R&D categories and, if so,
what score they would assign to these categories.
[31] Of the 49 experts who completed our questionnaire, we identified 7
from organizations that had submitted R&D proposals in response to
announcements issued by OPS in March and June 2002, based on
information provided by OPS. Of these 7, 5 were from organizations that
received funding from OPS. In December 2002, OPS issued another
announcement soliciting R&D proposals. However, because OPS has not yet
made funding decisions about proposals received in response to this
announcement, officials preferred not to provide us with information
about these proposals.
GAO's Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW,
Room LM Washington,
D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.
General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.
20548:
