Aviation Research
Airport Cooperative Research Program Addresses Many Needs but Could Enhance Transparency and Clarify Scope of Research Role
Gao ID: GAO-10-729 July 15, 2010
Airports are a vital part of the nation's air transportation system and face many similar challenges. In 2003, the Airport Cooperative Research Program (ACRP) was authorized to conduct applied research to help airport operators solve shared challenges that are not addressed by other federal research. As requested, this report addresses (1) the extent to which ACRP's processes reflect criteria for conducting a high-quality research program and (2) ACRP's results to date and their usefulness for the aviation community. GAO reviewed ACRP documentation and compared ACRP processes to criteria previously developed by GAO that can be applied to research programs. These criteria identify three phases of the applied research process and steps to help produce high-quality results. GAO also reviewed ACRP projects and publications and interviewed ACRP stakeholders and airport officials.
In each of the three phases of applied research that GAO has identified, ACRP conducts its research with processes that align with many of GAO's criteria for producing high-quality research, but some gaps exist. (1) Selecting projects: ACRP has established a governing board, the ACRP Oversight Committee (AOC), which is composed of airport executives and other key industry stakeholders, and processes to determine the research needs of users and to select specific projects for funding. However, one organization that participates on the board--the Airport Consultants Council--and the consensus approach used to make project selection decisions are not included in the program's documented operating procedures. ACRP stakeholders commended the council's participation and the consensus approach, but their omission from documentation potentially diminishes program transparency. (2)Implementing projects: ACRP's processes for establishing a project panel to manage research projects, selecting a researcher, and overseeing projects are well documented and include quality control steps. However, product dissemination efforts may miss some potential users, particularly staff at smaller airports and mid-level staff. The AOC has initiated a project to improve research dissemination to better serve these groups, although the project's scope and time frame is still being determined. (3) Evaluating projects and the program overall: ACRP maintains considerable information on ongoing and completed projects that are used by program managers and the AOC to review project progress. The program, however, does not currently have a systematic process for evaluating the impact of individual projects or implementing continuous improvements to the program's overall performance. Two initiatives--the dissemination project and a project initiated to review ACRP processes--could address current gaps in project and program evaluation, though the scope and time frames of these projects are still being determined. Through 2009, ACRP approved 169 projects, about half of which have been completed, and published 66 products on topics such as environmental impacts, policy and planning, and administration. Airport operators and other ACRP stakeholders consistently told GAO that the program provides the industry with useful and unique research that individual airports, particularly smaller airports, have neither the time nor budget to conduct. However, ACRP's role in conducting security research is unclear. ACRP materials, such as its annual solicitation of project ideas, include security as a potential topic within the scope of the program. However, the AOC has not recently funded security projects, in part because of differing views about whether ACRP should do this research. The Federal Aviation Administration, as a member of the AOC, indicated that the Department of Homeland Security is a better venue for such research. Conversely, other AOC members told GAO that ACRP could address some unmet security research needs. The AOC has the authority to determine what role, if any, is appropriate for ACRP in this area. By not doing so, over time, user satisfaction with the program could decline. GAO recommends that the Secretary of Transportation (1) ensure ACRP documentation reflects all participants and governance practices and (2) clarify ACRP's role in conducting security research. The Department of Transportation generally agreed with the report, provided technical comments, and is considering the recommendations. The Department of Homeland Security and the Transportation Research Board did not provide any comments on the draft report.
Recommendations
Our recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director:
Susan A. Fleming
Team:
Government Accountability Office: Physical Infrastructure
Phone:
(202) 512-4431
GAO-10-729, Aviation Research: Airport Cooperative Research Program Addresses Many Needs but Could Enhance Transparency and Clarify Scope of Research Role
This is the accessible text file for GAO report number GAO-10-729
entitled 'Aviation Research: Airport Cooperative Research Program
Addresses Many Needs but Could Enhance Transparency and Clarify Scope
of Research Role' which was released on July 15, 2010.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Committee on Science and Technology, House of
Representatives:
United States Government Accountability Office:
GAO:
July 2010:
Aviation Research:
Airport Cooperative Research Program Addresses Many Needs but Could
Enhance Transparency and Clarify Scope of Research Role:
GAO-10-729:
GAO Highlights:
Highlights of GAO-10-729, a report to the Committee on Science and
Technology, House of Representatives.
Why GAO Did This Study:
Airports are a vital part of the nation‘s air transportation system
and face many similar challenges. In 2003, the Airport Cooperative
Research Program (ACRP) was authorized to conduct applied research to
help airport operators solve shared challenges that are not addressed
by other federal research. As requested, this report addresses (1) the
extent to which ACRP‘s processes reflect criteria for conducting a
high-quality research program and (2) ACRP‘s results to date and their
usefulness for the aviation community. GAO reviewed ACRP documentation
and compared ACRP processes to criteria previously developed by GAO
that can be applied to research programs. These criteria identify
three phases of the applied research process and steps to help produce
high-quality results. GAO also reviewed ACRP projects and publications
and interviewed ACRP stakeholders and airport officials.
What GAO Found:
In each of the three phases of applied research that GAO has
identified, ACRP conducts its research with processes that align with
many of GAO‘s criteria for producing high-quality research, but some
gaps exist.
* Selecting projects: ACRP has established a governing board, the ACRP
Oversight Committee (AOC), which is composed of airport executives and
other key industry stakeholders, and processes to determine the
research needs of users and to select specific projects for funding.
However, one organization that participates on the board”the Airport
Consultants Council”and the consensus approach used to make project
selection decisions are not included in the program‘s documented
operating procedures. ACRP stakeholders commended the council‘s
participation and the consensus approach, but their omission from
documentation potentially diminishes program transparency.
* Implementing projects: ACRP‘s processes for establishing a project
panel to manage research projects, selecting a researcher, and
overseeing projects are well documented and include quality control
steps. However, product dissemination efforts may miss some potential
users, particularly staff at smaller airports and mid-level staff. The
AOC has initiated a project to improve research dissemination to
better serve these groups, although the project‘s scope and time frame
is still being determined.
* Evaluating projects and the program overall: ACRP maintains
considerable information on ongoing and completed projects that are
used by program managers and the AOC to review project progress. The
program, however, does not currently have a systematic process for
evaluating the impact of individual projects or implementing
continuous improvements to the program‘s overall performance. Two
initiatives”the dissemination project and a project initiated to
review ACRP processes”could address current gaps in project and
program evaluation, though the scope and time frames of these projects
are still being determined.
Through 2009, ACRP approved 169 projects, about half of which have
been completed, and published 66 products on topics such as
environmental impacts, policy and planning, and administration.
Airport operators and other ACRP stakeholders consistently told GAO
that the program provides the industry with useful and unique research
that individual airports, particularly smaller airports, have neither
the time nor budget to conduct. However, ACRP‘s role in conducting
security research is unclear. ACRP materials, such as its annual
solicitation of project ideas, include security as a potential topic
within the scope of the program. However, the AOC has not recently
funded security projects, in part because of differing views about
whether ACRP should do this research. The Federal Aviation
Administration, as a member of the AOC, indicated that the Department
of Homeland Security is a better venue for such research. Conversely,
other AOC members told GAO that ACRP could address some unmet security
research needs. The AOC has the authority to determine what role, if
any, is appropriate for ACRP in this area. By not doing so, over time,
user satisfaction with the program could decline.
What GAO Recommends:
GAO recommends that the Secretary of Transportation (1) ensure ACRP
documentation reflects all participants and governance practices and
(2) clarify ACRP‘s role in conducting security research. The
Department of Transportation generally agreed with the report,
provided technical comments, and is considering the recommendations.
The Department of Homeland Security and the Transportation Research
Board did not provide any comments on the draft report.
View [hyperlink, http://www.gao.gov/products/GAO-10-729] or key
components. For more information, contact Susan A. Fleming at (202)
512-2834 or flemings@gao.gov.
[End of section]
Contents:
Letter:
Background:
ACRP Has Many Processes in Place Aimed at Conducting High-Quality
Research and Is Taking Steps to Help Address Gaps in Dissemination and
Evaluation, but Program Documentation Does Not Reflect Some Practices:
ACRP's Products Are Generally Well Regarded, but Its Role in Security
Research Is Unclear:
Conclusions:
Recommendations for Executive Action:
Agency Comments:
Appendix I: Scope and Methodology:
Appendix II: Criteria for Assessing Applied Research Processes:
Appendix III: Additional Comments from Airport Officials:
Appendix IV: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Comparison of ACRP's Selection Processes to GAO's Criteria
for a High-Quality Research Program:
Table 2: Comparison of ACRP's Implementation Processes to GAO's
Criteria for a High-Quality Research Program:
Table 3: Comparison of ACRP's Evaluation Processes to GAO's Criteria
for a High-Quality Research Program:
Table 4: ACRP Projects Organized by Research Topic (Calendar Years
2005 through 2009):
Table 5: ACRP Publications Have Increased in Number and Type from 2007
through 2009:
Figures:
Figure 1: Summary of Phases and Steps in Applied Research Process:
Figure 2: Step 1--Establish a Research Investment Board 1:
Figure 3: Step 2--Determine the Research Needs of Users 1:
Figure 4: Step 3--Select Research Projects for Investment:
Figure 5: Step 4--Implement Research Projects:
Figure 6: Step 5--Maintain Information on the Research Program:
Figure 7: Step 6--Perform Postpublication Reviews of Projects and
Results:
Figure 8: Step 7--Use Evaluation Information to Improve the Overall
Performance of the Research Program 1:
Abbreviations:
ACC: Airport Consultants Council:
ACRP: Airport Cooperative Research Program:
AIP: Airport Improvement Program:
AOC: ACRP Oversight Committee:
DHS: Department of Homeland Security:
FAA: Federal Aviation Administration:
MOA: Memorandum of Agreement:
NASA: National Aeronautics and Space Administration:
NCHRP: National Cooperative Highway Research Program:
NSSA: National Safe Skies Alliance:
TCRP: Transit Cooperative Research Program:
TRB: Transportation Research Board:
TSA: Transportation Safety Administration:
Vision 100: Vision 100--A Century of Aviation Reauthorization Act:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
July 15, 2010:
The Honorable Bart Gordon:
Chairman:
The Honorable Ralph Hall:
Ranking Member:
Committee on Science and Technology:
House of Representatives:
Airports play an important role in the nation's extensive air
transportation network for both passengers and cargo, operating in a
complex environment with a wide range of challenges. The 3,400
airports that make up the national integrated airport system and are
eligible for Airport Improvement Program (AIP)[Footnote 1] grants are
a diverse group of independent entities--large and small; run by port
authorities, governments, and airport commissions; located in urban
and rural areas; and responsible for a mix of commercial and general
aviation operations--with loose organizational connections among them.
Despite this diversity, airports face similar challenges. For example,
many airports have an interest in better runway pavements and many
must deal with their environmental impacts, such as aircraft noise,
air pollution, and deicing chemical runoff. Airports, regardless of
size, are subject to federal, state, and local environmental
regulations and can face significant challenges in complying with
them, and airports are seeking cost-effective solutions to
strengthening the security of their airfields and terminals.
Federal aviation research plays an important part in helping airports
make sound decisions to address the challenges they face. The federal
government spends hundreds of millions of dollars each year on
aviation research--carried out by numerous federal agencies--that is
focused on research, development, and technology related to the broad
national aviation system, including the air traffic control system,
security, alternative runway materials and designs, safety hazards,
and airports, among other topics.[Footnote 2] However, this body of
research and development can overlook some important applied research
topics that can help airport operators improve the safety, capacity,
and efficiency of their facilities. Such research can be too expensive
for one airport to fund by itself, particularly if multiple topics
need to be addressed. To address this problem, the Airport Cooperative
Research Program (ACRP) pilot program was authorized in 2003 as part
of the Vision 100--A Century of Aviation Reauthorization Act (Vision
100) to carry out applied research on problems shared by airport
operators.[Footnote 3] ACRP is sponsored by the Federal Aviation
Administration (FAA) and managed by the Transportation Research Board
(TRB), a unit of the National Research Council within the National
Academy of Sciences, through a joint management agreement that went
into effect in 2005.[Footnote 4] The program began publishing the
results of its research in 2007. As you requested, we have reviewed
the progress ACRP has made in addressing airports' research needs,
focusing on the following questions: (1) To what extent does ACRP have
processes in place that reflect established criteria for conducting a
high-quality research program? (2) What are ACRP's results to date--
including research studies, practical applications, and other results-
-and how useful have these results been for the aviation community?
Additionally, at your request, we have described, in the background of
the report, the implementation of ACRP relative to the requirements
outlined in its authorizing legislation.
To assess ACRP's research processes, we compared ACRP's research
processes to criteria--developed in previous GAO work--for managing
information technology investments.[Footnote 5] The criteria, which
can be applied to research programs with some adaptation, contain
critical processes for selecting investments, implementing projects,
and evaluating program performance.[Footnote 6] To determine the
extent to which ACRP's processes align with these criteria, and to
identify gaps and potential actions to address those gaps, we analyzed
ACRP's program documentation and, as needed, we interviewed ACRP
officials. To describe ACRP's results to date and determine how useful
those results have been to airports, we reviewed ACRP's published
reports and other program documentation and interviewed a wide range
of officials to obtain a diversity of perspectives on the program.
These interviews included FAA and TRB officials responsible for
managing the program; selected industry and airport officials involved
in program oversight and direction; and other airport officials who
are the intended users of ACRP but who do not have vested interests in
managing the program. These officials were judgmentally selected to
provide a range of views, including those of small and large airports,
various staff levels, and different topical areas of responsibility.
The results of these interviews should not be generalized to each of
the groups the officials represent or to all users of ACRP. Finally,
we interviewed officials from the Science and Technology Directorate
within the Department of Homeland Security to understand the extent to
which its research activities address the needs of airports in the
security area. We conducted this performance audit from August 2009 to
July 2010 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objectives.
We believe that the evidence obtained provides a reasonable basis for
our findings and conclusions based on our audit objectives. See
appendix I for more information on our scope and methodology and
appendix II for further details on the criteria we used to assess
ACRP's processes.
Background:
In 2003, ACRP was authorized as a pilot program in Vision 100 to
address problems shared by airport operators that could be solved
through applied, cooperative research and were not addressed by other
federal research programs. ACRP has continued to function under short-
term extensions of FAA's authorizing legislation, and as called for in
Vision 100, in 2008 the Secretary of Transportation transmitted to
Congress a report on ACRP that included a recommendation for the
program to be made permanent.[Footnote 7] ACRP addresses the shared
research needs of a large and diverse target audience of 3,400
airports located across the country, including approximately 600
commercial-service airports and 2,800 smaller general aviation
airports. Congress has appropriated a total of $59.9 million for ACRP
for fiscal years 2006 through 2010.[Footnote 8] Beginning in fiscal
year 2009, Congress increased the program's annual appropriation from
$10 million to $15 million, and FAA designated the additional funds
primarily for research related to airport environmental issues.
ACRP's structure and many of its operating procedures were established
in 2005 in a Memorandum of Agreement (MOA), as required in Vision 100,
between the Department of Transportation, FAA, and the National
Academy of Sciences, operating through TRB. Under this agreement, TRB
is responsible for managing ACRP and overseeing the program's daily
operations. The structure of the program was largely modeled after
other cooperative research programs managed by TRB, such as the
National Cooperative Highway Research Program (NCHRP) and the Transit
Cooperative Research Program (TCRP).[Footnote 9]
Vision 100 stipulates that the Secretary of Transportation appoint an
independent governing board for ACRP. The MOA designates that the ACRP
Oversight Committee (AOC) will function as the program's governing
board and specifies the initial size and composition of the AOC. In
2005, the Secretary appointed 13 voting members to the board drawn
from a diverse cross section of the airport industry. The AOC includes
representatives of a variety of perspectives within the airport
industry, including airport executives and senior management officials
from airports of different sizes; senior officials from FAA and the
National Safe Skies Alliance (NSSA); and industry consultants and
academics. Additionally, the MOA identifies seven nonvoting ex-officio
members to serve on the AOC. The nonvoting ex-officio members
represent four national associations, TRB, and two other federal
agencies.[Footnote 10] The AOC's responsibilities include (1)
prioritizing research needs identified through problem statements
solicited from airport operators and others in the industry, (2)
selecting research projects to address those needs, and (3) evaluating
program effectiveness.
The responsibility for implementing projects selected by the AOC is
handled by project panels and TRB staff assigned to the program.
Project panels are appointed by TRB from nominations solicited from
airports, universities, consultants, and airport associations. Panels
generally consist of airport practitioners and other individuals with
expertise in the subject area of the project. Panel responsibilities
include (1) finalizing the scope of the project; (2) developing
Requests for Proposals and selecting a researcher, often a consultant,
to perform the work; (3) providing project oversight while the
research is performed; and (4) reviewing interim reports and draft
products. TRB staff assist the project panels in all of these tasks,
perform many of the day-to-day functions required for managing the
research projects, and are responsible for the final editing and
production of ACRP reports.
Most federal aviation research is done outside of ACRP by a variety of
federal agencies. FAA conducts research, evaluation, and development
in a number of areas related to the national air transportation
system, including the air traffic control system, pavement design and
runway surface technologies, and other topics related to airports and
aviation. The Aeronautics Research Mission Directorate in NASA
performs fundamental aeronautical research in areas such as
propulsion. NSSA, a nonprofit organization jointly funded by FAA and
the Department of Homeland Security (DHS), conducts testing and
evaluation of airport security technologies. DHS conducts research on
a wide variety of security topics, including passenger and baggage
screening at airports, and the Department of Defense conducts research
on topics relevant to military needs, a few of which have applications
in civil aviation. TRB staff and FAA representatives work with the AOC
to help prevent ACRP's research from duplicating any of these other
efforts.
ACRP Has Many Processes in Place Aimed at Conducting High-Quality
Research and Is Taking Steps to Help Address Gaps in Dissemination and
Evaluation, but Program Documentation Does Not Reflect Some Practices:
ACRP has implemented many practices and procedures that help to assure
the production of high-quality applied research, but gaps exist in
some areas of its research processes. Specifically, ACRP's selection
processes include steps to identify research needs and evaluate
potential projects, but the program's documented operating procedures
do not include one current participant on the AOC or accurately
reflect the enhanced role of ex-officio members in project selection
decisions. In addition, ACRP has established well-documented policies
for managing, overseeing, and reviewing research projects, and TRB
maintains considerable information about the program's projects to
assist ACRP decision makers. However, ACRP's dissemination practices
may not reach all potential users of the program's research, and the
program does not have systematic processes in place to evaluate its
overall performance, although the AOC has recently approved two
special projects to examine how to address these gaps.
Including Certain Practices and Procedures in Research Processes Can
Help Applied Research Programs Produce High-Quality Research:
Applied research programs can help assure, though not guarantee, the
production of high-quality research by establishing certain procedures
and adhering to certain practices in their research processes. We
identified these procedures and practices by adapting criteria that we
previously developed for managing information technology investments
to provide guidance for applied research programs.[Footnote 11] We
applied the adapted criteria to ACRP's research processes (see
appendix II for additional details about the criteria we applied to
ACRP). These criteria identify three key phases of the applied
research process--selection, implementation, and evaluation--that
encompass seven general steps and a number of specific practices
needed to conduct applied research. These steps represent the key
actions that programs should take within each phase of the applied
research process (such as determining research needs, selecting and
implementing research projects, monitoring results, and improving
program performance) to support the production of high-quality
research. Figure 1 presents these seven steps within the three phases
of the applied research process.
Figure 1: Summary of Phases and Steps in Applied Research Process:
[Refer to PDF for image: illustration]
Phase 1: Selection:
Step 1. Establish a research investment board;
Step 2. Determine the research needs of users;
Step 3. Select research projects.
Phase 2: Implementation:
Step 4. Implement research projects by:
a) establishing a project panel to manage the project;
b) selecting a researcher to conduct the research;
c) providing oversight;
d) reviewing draft products;
e) disseminating final publications to users.
Phase 3: Evaluation:
Step 5. Maintain information on the research program;
Step 6. Perform postpublication reviews of projects and results;
Step 7. Use evaluation information to improve the overall performance
of the research program.
Source: GAO.
[End of figure]
Within each of these seven steps there are a wide variety of specific
practices that collectively contribute to a program's overall ability
to conduct high-quality research. The extent to which ACRP has
procedures and practices in place that help to assure high-quality
research is summarized below for each phase of the applied research
process.
Many ACRP Research Selection Processes Include Practices to Assure
High Quality, but Some Beneficial Program Practices Are Not Reflected
in Documentation:
ACRP's processes to establish the AOC, determine research needs, and
select research projects include many of the practices that help
assure high-quality work, but the program's documentation does not
accurately reflect some practices followed during the Selection phase
(see table 1).
Table 1: Comparison of ACRP's Selection Processes to GAO's Criteria
for a High-Quality Research Program:
Phase 1: Selection: Step 1. Establish a research investment board;
Key strengths in ACRP's processes:
* The AOC is composed of individuals from the airport industry who are
responsible for making project selection decisions;
Key gaps in ACRP's processes:
* Program documentation does not include one group that participates
on the AOC.
Phase 1: Selection: Step 2. Determine the research needs of users;
Key strengths in ACRP's processes:
* Research needs are identified and research topics are organized
through a systematic process;
Key gaps in ACRP's processes:
* None.
Phase 1: Selection: Step 3. Select research projects for investment;
Key strengths in ACRP's processes:
* Potential research projects are evaluated and prioritized following
documented policies;
Key gaps in ACRP's processes:
* Program documentation does not accurately reflect the enhanced role
of the AOC's ex-officio representatives, who are allowed to
participate in project selection decisions through a consensus-driven
decision-making approach;
* Potential security research projects may not be selected for reasons
discussed later in the report.
Source: GAO analysis of FAA and TRB documentation and interviews with
ACRP stakeholders.
[End of table]
As required in Vision 100, the AOC was established to govern ACRP and
select research projects. The MOA established the initial size,
composition, and role of the AOC, but the Secretary of Transportation
is responsible for appointing individual members to the board and is
authorized by the MOA to adjust the size of the AOC, including the
number of voting and nonvoting ex-officio members. Most significantly,
the AOC is responsible for making project selection decisions, and its
13 voting members include airport executives and other individuals
connected to the airport industry who collectively contribute a
variety of perspectives and expertise to the board's operations. As a
result, several AOC members stated that they consider the board to be
a proxy for the diverse membership of the airport industry.
However, the program's documented operating procedures do not reflect
the involvement of the Airport Consultants Council (ACC) on the AOC,
which reduces the transparency of the board's operations and could
potentially jeopardize the group's continued participation. As
previously discussed, the MOA specifies that certain national
associations and federal agencies serve as nonvoting ex-officio
members on the board. In practice, however, ACC, which was not
identified in the MOA, has also been invited to participate on the
AOC. Unlike other prominent industry associations that were involved
during the establishment of ACRP and now serve as ex-officio members
on the board, ACC was not involved during the program's establishment
and was not included on the AOC. Instead, after the program was
established and operating, the AOC, TRB, and FAA decided that the
program would benefit from ACC's participation due to its role
representing airport consultants, the entities that perform much of
ACRP's research. In lieu of formally including ACC as an ex-officio
member, program officials and FAA have allowed the group to
participate informally on the AOC, in effect, largely treating ACC as
an ex-officio member. Several AOC members we spoke with described
ACC's contributions as beneficial to the board. As a result of ACC not
being designated as an ex-officio member in the MOA or other program
documentation, the transparency of the AOC's operations is diminished
and ACC's continued participation on the AOC is not guaranteed.
According to a senior FAA official, the FAA Administrator has
initiated action to formalize ACC's membership on the AOC by sending a
memorandum to the Secretary of Transportation in April 2010
recommending the Secretary appoint ACC as a nonvoting ex-officio
member on the AOC. As of July 7, 2010, FAA reported that this
nomination was under consideration by the Secretary. FAA officials
told us that they do not plan to revise the MOA to reflect this
change, should the Secretary of Transportation approve it, because
Vision 100 provided the Secretary with broad authority to appoint new
members to the board, and in their view, no further steps are required
to document any new members in the MOA. Additionally, FAA officials
pointed out that because the MOA expressly authorizes the Secretary to
adjust the size of the AOC, the Secretary may appoint an additional
nonvoting ex-officio member without amending the MOA. Furthermore,
officials stated that if the Secretary approves ACC's nomination to
the board, a signed appointment letter from the Secretary would
formalize ACC's participation as a nonvoting ex-officio member and
help to improve the transparency of the board's operations. However,
without also revising the MOA to reflect this appointment, the
transparency of the program's documented operating procedures would
remain diminished because the ACC would be the only ex-officio member
not expressly identified in the MOA.
Determining the research needs of users is necessary to select and
conduct useful research, and ACRP has taken various steps to identify
and organize topics in need of research. For example, ACRP opens its
annual solicitation of problem statements to all airport operators and
other industry stakeholders to help identify specific current research
needs. The AOC ultimately decides which problem statements should be
approved for funding, but the process of soliciting problem statements
helps the AOC remain aware of the current research needs confronting
airport operators.[Footnote 12] Additionally, in 2008, ACRP convened
focus groups and workshops with airport industry stakeholders and
industry groups to help identify and organize research topics and
critical issue areas viewed as important to the airport industry. The
results of this effort included, among others, (1) the identification
of 58 current and emerging topics for potential research that were
organized into 10 critical issue areas,[Footnote 13] and (2) the
inclusion of a strategic emphasis area in subsequent problem statement
solicitations to encourage submissions targeting particular subjects
of interest identified by the AOC, such as airport maintenance and
operations for the fiscal year 2011 submissions.[Footnote 14]
The program's project selection process also follows a systematic
approach to evaluate and prioritize potential research projects. For
example, multiple groups of knowledgeable individuals prescreen the
problem statements by evaluating and providing written feedback on the
submissions in advance of being reviewed by the AOC. Subsequently, AOC
members prioritize the problem statements using an online survey,
based on their view of a project's importance and the input provided
by the various prescreening groups. The results of this survey are
used to determine the order of the agenda for considering and
selecting projects at the AOC's annual summer meeting. Several AOC
members told us that this project prioritization procedure works
reasonably well in helping the board select projects to serve the
research needs of airports, although some AOC members reported that
reviewing all of the problem statements--ACRP received 219 problem
statement submissions for fiscal year 2010--is a time-intensive
process.
ACRP's documented operating procedures do not reflect the AOC's
current consensus-driven approach to selecting projects, which several
AOC members praised. The MOA states that the AOC should reach project
selection decisions through the affirmative vote of a majority of the
board's voting members. In practice, the AOC makes decisions on the
basis of consensus among its members, which essentially gives
nonvoting ex-officio members (and the ACC) the same role as the formal
voting members in project selection.[Footnote 15] Several AOC members
complimented the use of this consensus approach, and the ACRP program
manager said it would be detrimental to not include the views of ex-
officio members in the selection process, noting that consensus
decision making is fundamental to TRB's research process.
Additionally, one AOC member expressed appreciation for the input
provided by ex-officio members during project selection discussions,
and other board members highlighted the effectiveness with which the
AOC Chair facilitates the board's operations using the consensus
approach. However, the transparency of ACRP's project selection
processes is diminished since the program's documentation does not
reflect the AOC's consensus-based decision-making approach, which
enhances the role of ex-officio members.
ACRP's Processes for Implementing Projects Help Produce High-Quality
Research, but Product Dissemination May Miss Some Potential Users:
ACRP's policies and procedures for managing, overseeing, and reviewing
research projects include many of the processes needed to produce high-
quality research, but current product dissemination techniques may
miss some potential users of ACRP's research in the airport industry
(see table 2).
Table 2: Comparison of ACRP's Implementation Processes to GAO's
Criteria for a High-Quality Research Program:
Phase 2: Implementation:
Step 4. Implement research projects by:
a) establishing a project panel to manage the project;
b) selecting a researcher to conduct the research;
c) providing oversight;
d) reviewing draft products;
e) disseminating final publications to users;
Key strengths in ACRP's processes:
* ACRP follows documented policies and procedures for establishing
project panels to manage research projects;
* ACRP follows documented policies and procedures to select and
provide oversight of researchers conducting ACRP projects;
* ACRP's quality control process includes reviews of draft products by
project panels;
Key gaps in ACRP's processes:
* ACRP's current product dissemination mechanisms may miss some
potential users of the program's research. However, ACRP officials
initiated a special project in July 2009 to improve their
dissemination processes, increase ACRP's awareness among airport
practitioners, and improve feedback from users of ACRP products. The
details of this project are still under development.[A]
Source: GAO analysis of FAA and TRB documentation and interviews with
ACRP stakeholders.
[A] ACRP Project #11-05, Dissemination of ACRP Research Results.
[End of table]
ACRP follows well-documented policies for establishing project panels
to manage research projects, selecting and providing oversight of
researchers, and reviewing draft products prior to publication.
Project panels are assembled by TRB and generally consist of six to
eight members selected from the airport industry, government,
academia, and elsewhere to provide subject-area expertise and guidance
to a project. Several AOC members we spoke with stated that TRB did a
good job putting panels together, but one member expressed a need for
greater transparency in how TRB forms panels to better assure the AOC
that all of the needed perspectives are well represented and reflected
in the final report. Once established, the panels work closely with
TRB staff in many aspects of project management and oversight. For
example, project panels are responsible for developing Requests for
Proposals, selecting contractors to perform the research, and
reviewing interim reports and draft products prepared by the
contractor before approving final reports for publication. External
review of ACRP's products is rare, but occasionally panels have held
workshops or distributed surveys about an ongoing project to
participants at industry conferences. As part of the program's quality
control process, after the project panel approves the final content of
a report, TRB's editorial staff edits the text and manages final
publication. One board member we spoke with noted that the program's
research processes contain practices for ensuring quality that, while
lengthy at times, are important for the success of the program.
Overall, several AOC members told us that the program's project
management and oversight processes function well, and some members
stated that it would not be feasible or beneficial for the AOC to
assume a more direct role in project oversight. Likewise, the airport
operators we spoke with who have also served on ACRP project panels
generally spoke highly of their experience, and several AOC members
praised the ability of the panels and TRB staff to effectively oversee
ACRP's research projects.
ACRP disseminates its publications through a variety of mechanisms,
but its current methods may not effectively reach some potential users
of the program's research. Practices used to disseminate ACRP's
reports include:
* providing publications for free download on the TRB Web site;
[Footnote 16]
* including notices about new ACRP products in TRB newsletters;
* enabling people to sign up to receive electronic notification of new
ACRP publications;
* automatically notifying recipients of FAA's electronic mailing list
when ACRP reports are posted on the FAA Web site;
* distributing reports at industry meetings and conferences; and:
* working with various airport industry associations to promote ACRP
publications through the associations' established distribution
systems.
Although these dissemination mechanisms target many airport operators
and other potential users of ACRP's research, several airport
officials expressed concern that they may not effectively reach some
segments of the airport industry that could benefit from the program's
research. Many of the airport officials and AOC members we spoke with
stated that ACRP's visibility within the industry has increased over
time, but some expressed concerns that awareness of the program was
lower among some groups of airport operators than others. In
particular, some airport officials and AOC members suggested that
smaller airports and mid-level staff throughout the industry may be
less familiar with the program's research than large airports or
senior staff and managers. Various airport officials, AOC members, and
TRB staff with whom we spoke noted that staff from smaller airports
and mid-level staff are often less likely to be involved in national
associations and attend fewer national conferences than staff from
larger airports and high-level staff and managers. Some senior-level
officials mentioned that they make an effort to share ACRP reports
with their staff, but it is not clear how widely this practice is
replicated by other airport executives and managers in the industry.
As a result, smaller airports and mid-level airport staff may be less
likely to be among the direct recipients of ACRP's product
dissemination efforts. Maintaining an ACRP presence at industry
conferences was viewed by several airport officials and AOC members as
an important promotional tool for the program, and one official
suggested that participating in smaller regional conferences could
help ACRP connect with more officials from smaller airports. Several
airport officials also stressed the importance of electronic
publication of ACRP products, and one official emphasized that an
easily accessible and well organized online library of ACRP resources
is important for users, particularly as the program issues more
reports.
In recognition of some of these gaps in ACRP's dissemination
practices, the AOC approved a special project in July 2009 to (1)
develop and implement a strategic dissemination process and (2)
increase awareness of ACRP among airport practitioners.[Footnote 17]
Although the specific details and overall time frame of this project
are under development, the potential scope of work for this project
might include an examination of practices, used both within and
outside the airport industry, for disseminating and implementing
research findings to determine their applicability to ACRP. Whereas
most ACRP projects are concluded after a report is published, this
special project may develop into an ongoing internal effort by the
program to continually evaluate and revise ACRP's dissemination
practices as the program advances into the future. Given the early
stage of this project, it is not yet clear to what extent small
airport staff or mid-level staff will be targeted, but some ideas
being considered may serve these groups. For example, TRB staff are
considering methods for presenting ACRP research results at industry
conferences that TRB staff are unable to attend. TRB staff told us
that they expect to present some initial results from these efforts to
the AOC in July 2010, with additional dissemination initiatives
anticipated to begin under this project later in 2010.
TRB Maintains Considerable Information on ACRP's Projects, but the
Program Is Still in the Early Stages of Evaluating Project Results and
Assessing Program Performance:
TRB maintains considerable information about ACRP's completed and
ongoing research projects to support management decisions, but ACRP is
in the initial stages of evaluating project results and the program's
overall performance (see table 3).
Table 3: Comparison of ACRP's Evaluation Processes to GAO's Criteria
for a High-Quality Research Program:
Phase 3: Evaluation: Step 5. Maintain information on the research
program;
Key strengths in ACRP's processes:
* Information about research projects is identified and collected
following documented procedures;
* The collected information is easily accessible and understandable to
the program's decision makers;
Key gaps in ACRP's processes:
* None.
Phase 3: Evaluation: Step 6. Perform postpublication reviews of
projects and results;
Key strengths in ACRP's processes:
* Project panel members are surveyed about their experience at the
conclusion of a project and program officials have obtained some
anecdotal information from users of the research. The AOC approved two
special ACRP projects in July 2009 to explore, among other things,
options for conducting postpublication reviews and assessing project
results from the perspective of users[A];
Key gaps in ACRP's processes:
* ACRP does not have a formal process for reviewing the results of its
research projects and it does not have a systematic process for
obtaining feedback from users of the research about ACRP products. The
two special ACRP projects initiated in July 2009 are taking steps to
help address these gaps, although the specific details of the projects
have not been finalized.
Phase 3: Evaluation: Step 7. Use evaluation information to improve the
overall performance of the research program;
Key strengths in ACRP's processes:
* According to TRB officials and preliminary project documentation,
the two special projects approved by the AOC in July 2009 may include,
among other things, (1) an evaluation of ACRP's impact and usefulness
to airport operators and (2) a review of ACRP's processes and
examination of ways to improve ACRP's performance[A];
Key gaps in ACRP's processes:
* ACRP has not yet developed a process to systematically evaluate the
program, ascertain how its publications are used by the airport
industry, or obtain systematic feedback from users on how the program
can be improved. The two special projects initiated in July 2009 are
taking steps to help address these gaps, although the specific details
of the projects have not been finalized.
Source: GAO analysis of FAA and TRB documentation and interviews with
ACRP stakeholders.
[A] These options are being explored as part of ACRP Project #11-05,
Dissemination of ACRP Research Results and ACRP Project #11-06,
Evaluating ACRP Processes.
[End of table]
TRB maintains an internal database that includes considerable
information--such as contract costs, staffing, and timeline data--
collected on ACRP's completed and ongoing research projects. TRB
provides summaries of this information to the AOC at its semiannual
meetings, and shares other information as needed with program
stakeholders to help them monitor project performance. Additionally,
TRB prepares quarterly progress reports outlining program activities
for each project. Several AOC members told us they generally receive
the right amount of information from TRB staff to fulfill their
responsibilities on the board, and some members commented that any
additional information they need about a project is readily available
upon request.
However, ACRP does not have policies in place to systematically review
completed projects, assess project results from the perspective of the
research users, or to evaluate the overall performance of its research
portfolio, but the AOC recently approved two special projects that may
begin to address these gaps. Evaluating program performance and
reviewing the results of individual projects are necessary to gauge
user satisfaction, and high-quality research programs should analyze
the results of these reviews to identify areas for improvement and
implement changes to the program's processes. Although ACRP has not
systematically performed these evaluative steps, the program has
developed a survey for project panel members to solicit information
about their experiences upon the completion of a project, and board
members have received some informal anecdotal feedback from airport
operators about the program. Additionally, in April 2010, ACRP added a
feature to its Web site to allow airport operators to describe how
they have used ACRP's publications. To date, this effort has resulted
in the submission of more than one dozen examples of how ACRP
publications have been used by members of the airport industry. FAA
and TRB officials stated that in the early years of the program, too
few projects had been completed to conduct extensive program
evaluation, but now they believe that ACRP has grown to the point
where evaluating the program's results and overall performance would
be appropriate. Consequently, in July 2009, the AOC approved two
special projects that, among other goals, are intended to evaluate
ACRP's processes and improve the program's performance and usefulness
to airport operators.[Footnote 18] Initial recommendations and ideas
for the scope of work of these projects include several tasks that
align with practices for program evaluation in a high-quality research
program. These initial recommendations and ideas include the following:
* Conducting an internal program evaluation of key elements of ACRP's
process.
* Developing outcome measures and collecting additional program
information to benchmark ACRP's performance against the performance of
NCHRP and TCRP.
* Examining the processes followed by NCHRP and TCRP in several
important areas, including methods used by these programs to evaluate
program performance. These programs have already addressed some of the
same evaluative questions being looked at by the AOC, and ACRP may be
able to learn from practices adopted by the other cooperative research
programs to improve its own evaluation processes.[Footnote 19]
* Assessing the value and usefulness of publications to the research
user community through readership surveys or other means.
Initial results from some of these tasks are expected to be presented
to the AOC in July 2010, including findings from workshops with AOC
members convened to examine ACRP's processes, and the results from
comparing ACRP to NCHRP and TCRP for selected performance metrics.
Although the scope and time frames of these projects are still under
development, these continued efforts, including solicitation of
actionable feedback from users of the research about specific projects
and the program's overall performance, could help address gaps in
ACRP's current project and program evaluation.
ACRP's Products Are Generally Well Regarded, but Its Role in Security
Research Is Unclear:
ACRP Has Produced a Wide Range of Products That Are Generally Well
Regarded by Selected Airport Officials and ACRP Stakeholders:
From January 2006 through December 2009, ACRP approved 169 projects
for funding, completed approximately half of those projects, and
published 66 products, covering a wide range of topics.[Footnote 20]
Once approved, most projects are organized into topical categories, as
shown in table 4. According to the ACRP program manager, projects are
assigned to a single category based on the main topic area for
research, though some projects address multiple topic areas.
Table 4: ACRP Projects Organized by Research Topic (Calendar Years
2005 through 2009):
Research field: Environment;
Projects: 34.
Research field: Policy and planning;
Projects: 28.
Research field: Administration;
Projects: 20.
Research field: Operations;
Projects: 15.
Research field: Safety;
Projects: 14.
Research field: Design;
Projects: 7.
Research field: Human resources;
Projects: 3.
Research field: Maintenance;
Projects: 3.
Research field: Construction;
Projects: 2.
Research field: Security;
Projects: 1.
Research field: Other projects[A];
Projects: 42.
Research field: Total projects[B];
Projects: 169.
Source: GAO analysis of TRB data.
[A] Includes legal research projects and quick response studies to
address special needs.
[B] Table shows completed and ongoing projects approved as of December
31, 2009.
[End of table]
Since ACRP began publishing products in 2007, the program has
increased the diversity of product types and the number of in-depth
reports published. ACRP products include a variety of different
publication types, including project reports, such as guides on
airport strategic planning or environmental performance; synthesis
reports that summarize already available information; research results
digests that offer quick summaries on topics; legal research digests
that summarize case law and other legal research; and Web-only
documents that typically augment other published products (see table
5). Project reports tend to take longer to complete than other
products and result in more detailed, substantive publications. ACRP
products have targeted the needs of various airports and user groups.
For example, ACRP has published a guide for managing small airports
that focuses specifically on the needs of small airports.[Footnote 21]
Some products benefit only those airports that operate in certain
conditions, such as ACRP products on aircraft deicing, which target
airports in geographic regions that operate in winter conditions where
icing is a concern. Other products--such as the recently completed
airport strategic planning guide--address issues faced by all
airports.[Footnote 22]
Table 5: ACRP Publications Have Increased in Number and Type from 2007
through 2009:
Type of publication: Project reports (guides and other in-depth
studies);
2007: 1;
2008: 9;
2009: 16.
Type of publication: Synthesis reports (summarize already available
information);
2007: 4;
2008: 8;
2009: 5.
Type of publication: Research results digests (offer quick summaries
of research results);
2007: 3;
2008: 0;
2009: 4.
Type of publication: Legal research digests (summarize case law,
airport regulations, and other legal research);
2007: 0;
2008: 4;
2009: 4.
Type of publication: Web-only documents (supplement other
publications);
2007: 0;
2008: 0;
2009: 8.
Type of publication: Total publications;
2007: 8;
2008: 21;
2009: 37.
Source: GAO analysis of TRB data.
[End of table]
This increase in products is due to several factors. First, the
program received a steadily increasing number of research proposals,
from 68 for consideration for fiscal year 2007 to 219 for fiscal year
2010. Second, recent funding increases enabled the program to fund 30
projects in 2010, up from 23 in 2007. Finally, the AOC has approved
projects faster than work has been completed. Specifically, while some
projects can take more than 2 years to complete, the AOC approves most
projects on an annual basis. Due to all of these factors, officials
familiar with ACRP expect a large body of work, particularly in-depth
reports, to be issued in 2010 and 2011 that will greatly increase the
breadth and depth of ACRP's library of research.
ACRP has generally produced high-quality and useful products,
according to all 10 airport officials with whom we spoke. Officials
generally indicated that ACRP products are consistently high quality,
provide an appropriate level of detail for a given topic and audience,
place an issue within its overall context, and explain the methodology
used to reach its conclusions. For example, one official commented
that ACRP's two reports on safety management systems clearly explained
the principles of a safety approach and how it might change
operations.[Footnote 23] Officials also commended ACRP products for
the number of different ways they could be used. For example, two
officials noted that ACRP reports have provided reassurance that the
practices used at their airport are consistent with industry
practices. Other ACRP reports have been used to help educate airport
staff, elected officials, and airport board members about issues
facing airports. ACRP reports have also helped airports understand the
implications of new regulations or proposals. For example, multiple
officials mentioned that the ACRP review of airport rescue and
firefighting standards was a valuable tool in understanding the cost
and staffing implications of a proposed regulatory change.[Footnote
24] Officials offered few specific examples of ACRP reports having a
quantifiable financial impact on airport practices, but some indicated
that ACRP products have shaped their approach to issues with financial
implications. For example, officials at two airports mentioned that an
ACRP report informed their strategy for managing parking facilities--
an important revenue source for airports.[Footnote 25] One of these
officials explained that the ACRP report helped him ask detailed,
probing questions of vendors when considering options for a capital
project--without first hiring a consultant--and likely saved the
airport money. See appendix III for additional comments on ACRP
provided by airport officials.
Airport officials told us that ACRP provides airports with a unique
source for research that airports have neither the time nor budget to
fund individually, noting that the program's research can be
particularly useful to small airports. One official commented that
information from industry groups and peers--which airport operators
typically rely on--may not be as in-depth as an ACRP report. Officials
also commented that while large and small airports can benefit from
ACRP's research, small airports might benefit more than larger
airports because small airports tend to have fewer staff with
experience in technical subjects. Officials said they expect the
program to become increasingly useful as more products are released,
and they commented that many ACRP reports are likely to remain
relevant and applicable for multiple years, so they could find
themselves using the reports in the future as issues arise at their
airports.
ACRP stakeholders--FAA, industry groups, and AOC members--have also
generally been pleased with the work ACRP has produced to date. FAA
officials told us that the program has strongly met the agency's
expectations, finding the work produced to be well done and useful to
the agency and the industry. For example, according to FAA officials
familiar with ACRP, FAA environmental staff will sometimes refer to
ACRP products when developing advisory circulars or other guidance.
Most of the industry groups involved with the AOC--namely the
nonvoting ex-officio members of the board and the ACC--also indicated
that the program is generally doing a good job meeting the needs of
the airport industry. One group that regularly holds training seminars
for its members now uses some ACRP products as part of its course
materials, and another group that serves state government officials
that have airport management responsibilities, including small
airports, has found ACRP to be a helpful resource for its membership.
The AOC voting members we spoke with also find the program is
producing work that benefits industry practitioners. For example, one
board member explained that ACRP's research can be particularly
helpful to officials at small airports, such as his own, helping them
keep up with federal requirements and providing new information on
issues they may not otherwise have the resources to address. Lastly,
ACRP stakeholders agreed that the program also provides some
intangible benefits to the airport industry. For example, serving on
an ACRP project panel can help build the technical expertise and
professional networks of the participants, and ACRP's graduate student
award program aids future workforce development by encouraging student
interest and expertise in airport-related issues.
ACRP's Role in Conducting Security Research Projects Is Unclear:
ACRP includes security as within the scope of its designated research
topics in program materials--such as its annual report, Web site, and
annual solicitation for research project ideas--but to date, the
program generally has not conducted security research. Airport
operators have direct responsibility for day-to-day airport
operations, including the security of airport perimeters, access
controls, and workers, as well as implementing Transportation Security
Administration (TSA) security requirements.[Footnote 26] To the extent
that airports share research needs in these areas that are not
adequately addressed by other federal research, ACRP's pursuit of this
research would be consistent with the broad authority provided to the
program in Vision 100, the legislation that authorized ACRP. Airport
and AOC officials we interviewed, including two airport officials
directly responsible for security, consistently told us that the
security of airport operations is a shared concern for airports and
that applied research could be beneficial in addressing this concern.
For example, those officials told us that research could help airports:
* coordinate airports' terminal design and layout decisions with new
baggage and passenger security screening requirements and technologies
to ensure smooth passenger flows in terminals, and:
* implement new strategies for securely managing the delivery of
supplies to concessionaires inside the secure area of the terminal.
Despite potential shared needs such as these and the legislative
authority to address unmet security research needs, ACRP has issued
one product it categorized as related to airport security as of May
2009.[Footnote 27]
Though the federal government conducts a large amount of research on
transportation security, some security research needs for airports may
not be fully addressed by federal research programs.[Footnote 28] DHS,
which has regulated airport security since 2003 through TSA, conducts
research through its Science and Technology Directorate.[Footnote 29]
The directorate's research activities are intended to support all of
DHS's wide-ranging responsibilities, including transportation security
for all modes such as aviation, border security, emergency
preparedness and disaster recovery, immigration, and counterterrorism.
In the case of aviation, the directorate's research projects are
prioritized based on security needs identified by TSA and focus on
technology development to aid TSA's screening operations at airports.
For example, the directorate conducts research in explosive detection
technologies that can be used to screen passengers and baggage at
airports. To the extent that the research needs of airports do not
align with TSA's security priorities, these needs would not be
addressed by the directorate. For example, as also noted earlier in
this report by an AOC member, a directorate official similarly
indicated that airports might need research into managing the secure
flow of supplies to concessionaires inside the secure area. In our
prior work, we have reported that while TSA has taken some steps to
coordinate with airport operators in developing and implementing new
screening technologies, TSA does not have a systematic process to
coordinate with airports in implementing new screening
technologies.[Footnote 30] Further, we have previously reported that
while TSA has taken some steps to develop biometric airport worker
credentialing and the agency has conducted some outreach to airport
operators to understand their priorities, it is unclear to what extent
TSA plans to address these needs.[Footnote 31] Additionally, NSSA,
which is represented on the AOC, has done some security research at
the request of the AOC. However, as an organization focused on testing
security technologies and not on research, it is not equipped to
conduct security research in all of the areas that airports might
benefit from, according to NSSA officials. For example, NSSA recently
declined a project referred by ACRP on the basis that NSSA does not
have the technical capacity to complete the research. Further, NSSA
does not generally publish reports for widespread dissemination like
ACRP.
Members of the AOC differ in their views about the appropriate role of
ACRP in conducting security research. In our interviews, 7 of the 11
AOC participants that discussed the program's role in conducting
security research expressed a willingness to fund proposed security
research projects if those projects were meritorious and not addressed
elsewhere.[Footnote 32] Conversely, four AOC members, including FAA,
believed that ACRP should not conduct security research, noting that
DHS or NSSA are more appropriate venues for this research. FAA has
also raised concerns about ACRP funding security research because FAA
does not regulate airport security. In its 2008 report to Congress on
ACRP, FAA stated that, in its view, ACRP should not fund security
research, a position the agency continues to hold.[Footnote 33]
However, as noted earlier in this report, project selection decisions
are the responsibility of and are made by the AOC as a whole, not just
by FAA or any of the other members of the AOC, based on the authority
granted to the governing board in Vision 100 and the operating
procedures delineated in the MOA. Further, with respect to DHS's views
on this issue, DHS, including TSA and the Science and Technology
Directorate, have not been involved in ACRP to date. Science and
Technology Directorate officials did not state an opinion about
whether ACRP should or should not address security research because
these officials were not familiar with the program.
AOC members have been reluctant to pursue security research given
FAA's position and the large number of nonsecurity research proposals
available to fund, and also because the lack of coordination with DHS
makes it difficult for the ACRP to select and execute needed and
credible security research. During the January 2010 AOC meeting,
members discussed ACRP's role in addressing security research and many
members voiced an interest in improving ACRP's handling of security
issues. Members expressed differing views about whether or not they
should communicate that ACRP is a venue for security research to
outside parties interested in submitting security proposals. The TRB
staff that manages ACRP told us that the program could do more
security work, at the direction of the AOC, and that TRB has policies
and procedures to properly handle secure, sensitive information. In
fact, other TRB cooperative research programs regularly publish
security-related products. For example, NCHRP has issued reports on
blast-resistant highway bridges, vulnerability assessment guidelines,
and other topics. Given the varying views of ACRP program officials
and other stakeholders about whether and to what extent the program
should address potentially unmet security research needs, users of
ACRP's work may expect the program to produce security research that
is not likely to be forthcoming. The inclusion of security as one of
the research topics within the scope of the program on its Web site
and in program materials, such as the annual solicitation of research
ideas from the airport industry, reinforces this expectation. Over
time, if expectations remain unmet, user satisfaction with the program
could decline.
Conclusions:
ACRP is regarded by the officials we interviewed as a generally
valuable resource for addressing the shared challenges faced by
airport operators, but improving some aspects of its processes could
further enhance its effectiveness. To the program's credit, ACRP's
research processes include many of the practices necessary to select
and implement high-quality applied research. However, some gaps in
ACRP's research processes hinder the program's ability to fully meet
the research needs of airports. In particular, the transparency of
some aspects of the AOC's operations is incomplete because the
program's documentation does not accurately reflect certain program
practices. For example, the involvement of ACC on the board is not
accurately reflected in the MOA or other documentation. Although FAA
told us it has initiated steps to formalize ACC's membership on the
board, it does not plan to revise the MOA that governs the program's
operating procedures to reflect this change. If the Secretary of
Transportation approves FAA's request to appoint ACC to the board,
obtaining a signed appointment letter from the Secretary to formalize
ACC's role as a nonvoting ex-officio member would be a positive step
toward increasing the program's transparency. However, the
transparency of the board's operations would still remain diminished
because the ACC would be the only ex-officio member not expressly
identified in the MOA. Likewise, the AOC's documented decision-making
process--which specifies voting procedures--differs from the board's
practice of selecting projects on the basis of consensus among its
members, including those designated as nonvoting ex-officio members.
AOC members and other program stakeholders generally found the
participation of ACC and the consensus-based decision-making approach
to be beneficial, but since the program's documentation does not fully
align with these practices, there is little formal assurance they will
be sustained in the future.
Additionally, levels of awareness of ACRP vary throughout the airport
industry, and ACRP's current research dissemination practices may miss
some potential beneficiaries of the program's research, according to
the program stakeholders and airport officials we interviewed. In
particular, mid-level staff and small airport officials may be less
familiar with the program and more likely to not be included as direct
recipients of some of ACRP's current dissemination mechanisms. ACRP is
in the early stages of developing systematic processes to conduct
postpublication reviews of its products, assess satisfaction with the
program among the user community, and evaluate the program's overall
impact and performance. These are important steps in the applied
research process that ACRP can improve upon. The AOC's approval of two
special projects holds promise that it will begin addressing the
program's dissemination and evaluation gaps. It is an encouraging sign
that ACRP's processes are on track to more fully reflect practices
that help to assure the production of high-quality applied research,
including project and program evaluation.
To date, ACRP has produced a variety of high-quality and useful
results for the airport industry, according to the officials we
interviewed. However, the program needs to clarify its scope to the
airport industry by explaining what role, if any, it has in addressing
the security research needs of airports. The program has the
legislative authority to address security research topics, provided
that the research needs are shared by airports and not addressed by
other federal research. Airport security is certainly a concern shared
by airport operators. Additionally, some security-related research
topics, such as the use of biometric technology, may be of value to
airports and are not being fully addressed by other federal research.
Given the significant amount of aviation security research conducted
by DHS, ACRP may or may not be an appropriate venue to address unmet
needs of airports in this area. Although FAA has stated that ACRP
should not conduct security research, it is the responsibility of the
AOC as a whole, of which FAA is a part, to determine what role, if
any, the program has in this area. The AOC has recently discussed the
program's role in addressing security, but it is not clear what
actions, if any, it plans to take to resolve the current lack of
clarity. The present lack of agreement among ACRP stakeholders about
the program's role in addressing security research-
-and the program's lack of coordination with DHS--has left the
program, in effect, not addressing security research, but holding out
the possibility that it could to the airport industry. Clarification
of the program's role in this area would better communicate to the
industry what research can be expected from ACRP.
Recommendations for Executive Action:
We recommend that the Secretary of Transportation take the following
two actions:
(1) To better align key program documentation with ACRP program
practices as implemented and to increase the transparency and
stability of the program over time, we recommend the Secretary of
Transportation take steps to revise the Memorandum of Agreement (MOA)
between the Department of Transportation and the National Academy of
Sciences, and other appropriate documentation, such that:
* all organizations, including ex-officio members, that are involved
in ACRP Oversight Committee (AOC) proceedings are included in program
documentation, and:
* project selection procedures documented in the MOA are supplemented
to include, as an option, a consensus-based approach in addition to
voting procedures, and a more explicitly defined role for ex-officio
members in project selection.
(2) To clarify the role of ACRP in conducting security research, we
recommend that the Secretary of Transportation take steps to encourage
the AOC--in collaboration with other key federal agencies and
stakeholders--to clearly articulate ACRP's role, if any, in conducting
security research and, subsequently, to ensure that ACRP's program
documentation clearly and accurately reflects this role, such that
airport operators and others can readily understand what to expect of
the program in this area.
Agency Comments:
We provided a draft of this report to DOT, DHS, and TRB for review and
comment prior to finalizing the report. DOT generally agreed with the
information in the report and provided technical clarifications, which
we incorporated as appropriate. DOT officials also said that they are
considering the recommendations. DHS and TRB did not provide any
comments on the draft report.
We are sending copies of this report to the Secretary of the
Department of Transportation, the Administrator of the Federal
Aviation Administration, the Director of Cooperative Research Programs
at the Transportation Research Board, the Secretary of the Department
of Homeland Security, and appropriate congressional committees. This
report is also available at no charge on the GAO Web site at
[hyperlink, http://www.gao.gov].
If you or your staff have any further questions about this report, or
wish to discuss these matters further, please contact me at (202) 512-
2834 or flemings@gao.gov. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last
page of this report. Key contributors to this report are listed in
appendix IV.
Signed by:
Susan A. Fleming:
Director, Physical Infrastructure Issues:
[End of section]
Appendix I: Scope and Methodology:
To assess the Airport Cooperative Research Program's (ACRP) research
processes, we compared ACRP's research processes to criteria--
developed in previous GAO work--for managing information technology
investments.[Footnote 34] The criteria, which can be applied to
research programs with some adaptation, contains critical processes
for identifying user needs, selecting investments, monitoring results,
and improving performance over time, among other steps. Similar to
when an organization undertakes information technology investments,
the applied research processes of ACRP are conducted by a third party
for a group of diverse users or beneficiaries, the topics are
technical in nature, and funding constraints require making decisions
between competing projects. To adapt the information technology
framework as criteria for assessing ACRP processes, we (1) simplified
it to focus on the most critical processes, without the attention to
details about levels of organizational maturity and capacity that were
extensively addressed in the original framework, (2) changed the
terminology to refer to applied research, rather than information
technology, and (3) added criteria on product dissemination that we
determined are important to research, but are not in the original
framework. See appendix II for further information about the adapted
criteria we used in our study. To determine the extent to which ACRP's
processes align with these criteria, and to identify gaps and
potential actions to address those gaps, we conducted a content
analysis of a variety of program documentation and interviewed
officials involved with ACRP.
To describe ACRP's results to date and determine how useful those
results have been to airports, we reviewed ACRP's published reports
and other program documentation and interviewed a wide range of
officials to obtain a diversity of views on the program. These
interviews included senior-level Federal Aviation Administration (FAA)
and Transportation Research Board (TRB) officials responsible for
managing the program; selected industry and airport officials involved
in program oversight and direction; and other airport officials who
are the intended users of ACRP, but who do not have vested interests
in managing the program. These officials were judgmentally selected to
provide a range of views, including those of small and large airports,
various staff levels, and different topical areas of responsibility.
Officials included top executives and mid-level managers with
expertise and responsibilities in a range of topical areas such as
environmental management, security, and finance, among others. Some of
these officials had also participated on ACRP project panels and,
therefore, could comment on their experience as a panelist. To
understand views of ACRP among stakeholders of the program and get an
overall view of how well the program was meeting their expectations
and to identify their areas of concern, we interviewed selected
members of the ACRP Oversight Committee (AOC)--including selected
voting members of the AOC and all of the industry associations
participating as ex-officio members on the AOC--and representatives
from the Airport Consultants Council, TRB, and FAA. Although our
interviews with airport officials and ACRP stakeholders provide a
reasonable view on the perspectives and views of these groups, our
findings based on these interviews should not be generalized to the
groups overall. Finally, we interviewed officials from the Science and
Technology Directorate within the Department of Homeland Security to
understand the extent to which its research activities address the
needs of airports in the security area.
We conducted this performance audit from August 2009 to July 2010 in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.
[End of section]
Appendix II: Criteria for Assessing Applied Research Processes:
To assess ACRP's research processes, we adapted an existing GAO
framework for managing information technology investments to make it
applicable to the processes followed by applied research programs to
produce high-quality research.[Footnote 35] This adapted framework
served as our criteria for evaluating ACRP's processes, and allowed us
to identify the elements of ACRP's processes that help the program
produce high-quality research, as well as identify gaps in ACRP's
processes that can be addressed to improve the program. Figures 2
through 8 present the specific practices (categorized as
"organizational commitments," "prerequisites," and "activities") in
each step of the applied research process that programs follow to help
facilitate the production of high-quality applied research. We
analyzed the extent to which ACRP's processes align with each of the
practices contained in these figures, and we summarized the results of
this analysis for key elements of ACRP's processes in the body of this
report.
Within each step of the applied research process, "purpose" refers to
the primary function of the step and states the desired outcome of
completing the step; "organizational commitments" refer to the
management actions and practices that ensure necessary processes to
perform the step are established and will endure; "prerequisites"
refer to the conditions that must exist within an organization to
implement the necessary processes successfully; and "activities" refer
to the key practices necessary to implement the critical processes and
successfully complete the step.
Figure 2: Step 1--Establish a Research Investment Board:
[Refer to PDF for image: illustration]
Purpose:
To define and establish a management structure and process for
selecting, controlling, and evaluating research project investments
Organizational commitments:
1. The investment board defines and implements its structure and
processes and is composed of individuals qualified to make decisions
on project selection, execution, oversight, and evaluation.
2. The processes used to guide research investment decisions and
organization operations are documented, including the role of the
board, the process for appointing members to the board, and the
board‘s decision-making process.
Prerequisites:
1. Adequate resources, including managerial and staff support,
funding, and tools to analyze operations, are provided to support the
board‘s operations.
2. Board members understand the procedures and tools used to make
decisions. Trainings or other guidance on the research process are
available for board members
Activities:
1. The board is responsible for developing and documenting the process
by which it works and the research investment process.
2. The board operates within its authority and responsibility.
3. Management controls are in place to ensure board decisions are
carried out by staff.
Source: GAO.
[End of figure]
Figure 3: Step 2--Determine the Research Needs of Users:
[Refer to PDF for image: illustration]
Purpose:
To ensure that research projects selected for investment support the
needs of users and are consistent with the mission of the organization.
Organizational commitments:
1. The organization follows documented policies and procedures to
systematically identify, classify, and organize users‘ research needs
and projects to address those needs.
2. Policies and procedures should include regularly identifying users‘
research needs as part of strategic planning, stating research needs
in terms of desired outcomes, linking similar research needs of
different users into a single project, and terminating projects or
proposals that do not target identifiable users.
Prerequisites:
1. The organization has a stated mission that includes goals and
objectives.
2. Adequate managerial time and attention are spent identifying users‘
research needs.
3. The organization has adequate staff support to carry out activities
to identify research needs.
Activities:
1. The organization defines and documents the need for approved
research projects.
2. The organization identifies specific users and beneficiaries of
research projects.
3. Users are involved in project management throughout the research
process, including project identification, prioritization, selection,
execution, and evaluation.
4. The board periodically evaluates how well research investments
align with the goals and objectives of the program, including how well
projects meet users‘ needs, and takes corrective action as needed.
Source: GAO.
[End of figure]
Figure 4: Step 3--Select Research Projects for Investment:
[Refer to PDF for image: illustration]
Purpose:
To ensure a well-defined and disciplined process is used to select new
research projects and reselect ongoing projects.
Organizational commitments:
1. The organization follows documented policies and procedures to
identify, evaluate, prioritize, and select new and ongoing research
projects.
2. Policies and procedures should (a) include documented criteria for
use in project selection to ensure common understanding of the
selection process, (b) include documentation that defines the roles
and responsibilities of each group participating in the project
selection process, (c) document the predefined data required to make
selection decisions, (d) document that reinvestments in completed or
ongoing projects are based on their previous success and likely
contribution to current needs, and (e) integrate project selection
with available funding.
Prerequisites:
1. Adequate resources, including managerial and staff support,
funding, and tools to organize and analyze proposals, are dedicated to
identifying and selecting research projects.
2. Criteria for analyzing, prioritizing, and selecting projects are
established that allow proposals to be compared with one another based
on qualitative and quantitative measures such as benefit-cost, project
longevity, customer needs, risk, and technical difficulty.
3. A mechanism exists to ensure project selection criteria remain
appropriate to organizational goals and are revised as the needs of
users change.
Activities:
1. The organization follows its defined project selection process,
using pre-established selection criteria, to select projects for
funding. The selection process should align with the budget process.2.
The organization‘s leadership makes the final investment decisions
based on information and analysis from prior steps in the selection
process, with less meritorious projects being denied funding.
Source: GAO.
[End of figure]
Figure 5: Step 4--Implement Research Projects:
[Refer to PDF for image: illustration]
Purpose:
To conduct approved research, monitor progress, and produce research
products that are disseminated to users.
Organizational commitments:
1. The organization follows documented policies and procedures for
management oversight of research projects that specify the
responsibilities of entities involved in providing oversight and the
procedures for decision making they are to follow.
2. The organization establishes threshold criteria for assessing
project performance (e.g., cost and schedule measures) that trigger
when remedial action may be needed.
3. Documented procedures exist to ensure that the board and project
teams are involved and oversee decisions to change projects to meet
cost, schedule, or other considerations.
4. The organization has documented procedures for elevating
significant issues from project teams to program management and the
board, and establishes conditions and a process for project
termination.
Prerequisites:
1. Adequate resources, including managerial and staff support,
funding, and tools, are provided for project oversight.
2. Project teams maintain plans for project execution that include
expected cost and schedule milestones and measurable benefit and risk
expectations.
Activities:
1. Selected projects are completed and products are produced.
2. Verified data on actual performance (including cost and schedule
performance) are provided to the project team and the board, and are
used to review the performance of projects against their stated
expectations.
3. Appropriate actions are taken to correct or terminate
underperforming projects in accordance with documented policies and
procedures.
4. Expert review of draft products is completed as part of the quality
control process and changes are made as appropriate.
5. Products are disseminated to users through appropriate mechanisms.
Source: GAO.
Note: Step 4 involves implementing research projects by (1)
establishing a project panel to manage the project, (2) selecting a
researcher to conduct the research, (3) providing oversight, (4)
reviewing draft products, and (5) disseminating final publications to
users.
[End of figure]
Figure 6: Step 5--Maintain Information on the Research Program:
[Refer to PDF for image: illustration]
Purpose:
To make available to decision makers information to understand the
impacts and opportunities created by completed, current, proposed, or
potential research in certain areas.
Organizational commitments:
1. The organization has documented policies or procedures for
identifying and collecting information about its research projects.
2. An official is responsible for ensuring project information is
collected that meets the needs of the research investment process.
Prerequisites:
1. Adequate resources, including managerial and staff support,
funding, and tools, are dedicated to identifying and collecting
information about research projects.
Activities:
1. The organization‘s projects are identified and specific information
is collected about the projects to support decisions.
2. The collected information is easily accessible and understandable
to decision makers.
3. The collected information is used by the board in making future
investment decisions.
4. The collected information is used to inform decisions about how to
increase the effectiveness of the organization‘s research portfolio.
Source: GAO.
[End of figure]
Figure 7: Step 6--Perform Postpublication Reviews of Projects and
Results:
[Refer to PDF for image: illustration]
Purpose:
To compare actual results and usefulness of projects to users with the
expectations that were set for projects when initially selected and
develop process improvements.
Organizational commitments:
1. The organization has documented policies and procedures for
conducting postpublication reviews and assessing results of projects
from the perspective of users.
2. Policies and procedures should document the types of projects for
which postpublication reviews are conducted, what information is
included in the reviews, and how and to whom the conclusions of the
review are disseminated.
3. Postpublication reviews should include project expectations and
results (e.g., user satisfaction and impact), cost and schedule
deviations, and overall recommendations and lessons learned.
Prerequisites:
1. Adequate resources, including managerial and staff support,
funding, and tools, are provided for conducting postpublication
reviews.
2. Individuals conducting postpublication reviews should be familiar
with the policies and procedures for conducting such reviews.
Activities:
1. The board uses specified criteria to determine which projects will
have postpublication reviews.
2. Quantitative and qualitative data related to performance
expectations and actual outcomes, including those experienced by users
of the publication, are collected, evaluated for reliability, and
analyzed.
3. Quantitative and qualitative data related to cost, schedule, and
timeline objectives are collected, evaluated for reliability, and
analyzed.
4. Quantitative and qualitative data related to the contribution of
the publication to achieving the organization‘s mission are collected,
evaluated for reliability, and analyzed.
5. Quantitative and qualitative data obtained from surveys and
interviews of users, project staff, and contractors are collected,
evaluated for reliability, and analyzed.
6. Quantitative and qualitative data obtained from interviews with
officials involved in initial project selection.
7. The organization develops, documents, and distributes lessons
learned and recommendations for improving project processes and
management to better address the needs of users.
Source: GAO.
[End of figure]
Figure 8: Step 7--Use Evaluation Information to Improve the Overall
Performance of the Research Program:
[Refer to PDF for image: illustration]
Purpose:
To assess and improve the impact and usefulness of the overall
research portfolio and improve the research investment management
process.
Organizational commitments:
1. The organization has documented policies and procedures for
evaluating and improving the impact and usefulness of its research
portfolio.
2. Policies and procedures should (a) document the responsibilities of
each party involved in evaluating impact and usefulness, (b) document
the time frame for conducting assessments and implementing
improvements, (c) include the measures used to assess the performance
of the research portfolio, and (d) include a mechanism for reporting,
and acting on, the results of the evaluation.
Prerequisites:
1. Adequate resources are provided for evaluating and improving the
performance of the research portfolio.
2. Board members and other staff responsible for evaluating and
improving portfolio performance are qualified to do so and receive
training as needed.
Activities:
1. Comprehensive performance measurement data are defined and
collected which show if (a) products are consistently cost-effective,
(b) projects are managed well and on budget, (c) users are satisfied
with the products or if other research areas would be more useful, and
(d) the products meet general industry standards.
2. The organization analyzes aggregate performance data and trends.
3. Recommendations for improvements to the investment process and
research portfolio are developed and implemented as a result of the
self-evaluation.
Source: GAO.
[End of figure]
[End of section]
Appendix III: Additional Comments from Airport Officials:
As part of our assessments of ACRP's research processes and ACRP's
results to date, we conducted interviews with selected airport
officials--the primary intended beneficiaries of ACRP's work--to learn
their views on, among other topics, areas in which ACRP might consider
actions to improve the quality or usefulness of its products. The
major themes that came out of these interviews are presented in the
body of this report. Additionally, a number of other comments were
made by these officials, some of which are summarized below. Their
inclusion should not be construed as a GAO endorsement or
recommendation for action, but instead are included because they may
be of interest to officials involved in ACRP.
* Some ACRP products are very general in nature. As the program
produces more work, future products may need to explore some issues
more in-depth.
* Over time, some ACRP products will need to be updated as the
information in the original reports become outdated. Reliance on the
normal project submission process may not effectively assure updates
are completed.
* An executive summary of longer reports may be helpful for some
users. A summary might help readers understand the major points of a
report without reading all the details, some of which may not apply to
all readers.
* Some airport officials may not understand that ACRP produces quick-
response projects, in addition to conducting longer term research. As
a result, some users with research needs may not seek out ACRP's help
for problems that require a quick response. ACRP may be a more useful
resource to airports if the diversity of its products were better
understood.
* Use of social media may help promote ACRP to some members of the
airport industry, particularly younger staff, and enhance
accessibility of ACRP products.
* Organizing ACRP products into logical categories in an online
library that takes only a click or two to navigate would make it
easier to access reports.
* An interactive, open-architecture Web site that allows users of ACRP
reports to post comments, questions, and read what other users think
about reports might enhance the impact of the reports and be a
mechanism for TRB to get feedback from the industry.
* ACRP distribution through the TRB newsletter--which includes
information on research in transportation modes other than aviation--
can be unwieldy, even with the ability to customize the TRB newsletter
to include only topics of interest to the recipient.
* Airport officials already get lots of surveys and completing these
surveys can be burdensome. Efforts to streamline and consolidate the
surveys ACRP conducts during its research might ease this burden.
* Additional formal guidance and clarification for project panels on
precisely what final ACRP products are intended to accomplish, and how
to specify a Request for Proposal to do so, might help panels execute
their oversight of research more effectively. Panelists are often
airport officials or other industry members that may not be
experienced in research methodologies, costs of research, or other
relevant issues.
* Nonairport specific professional organizations--such as bar
associations--may be a channel to reach airport officials within these
professions.
[End of section]
Appendix IV: GAO Contact and Staff Acknowledgments:
GAO Contact:
Susan A. Fleming, (202) 512-2834 or f [Hyperlink, flemings@gao.gov]
lemings@gao.gov:
Acknowledgments:
In addition to the contact named above, Rita Grieco (Assistant
Director), Lauren Calhoun, Patrick Dudley, Joel Grossman, Delwen
Jones, Stephen M. Lord, Faye Morrison, Madhav Panwar, Laura Shumway,
John W. Stambaugh, and Joshua Wiener made key contributions to this
report.
[End of section]
Footnotes:
[1] Inclusion in the National Plan of Integrated Airport Systems
allows an airport to access AIP funds. 49 U.S.C. §§ 47103, 47105(b)(2).
[2] These agencies include the Federal Aviation Administration (FAA),
the National Aeronautics and Space Administration (NASA), the
Department of Defense, the Department of Homeland Security (DHS), and
others.
[3] Pub.L. No. 108-176, § 712, 117 Stat. 2490, 2586 (2003), codified
at 49 U.S.C. § 44511(f).
[4] The National Research Council's mission is to improve government
decision making and public policy, increase public education and
understanding, and promote the acquisition and dissemination of
knowledge in matters involving science, engineering, technology, and
health. The Transportation Research Board--which is a division of the
National Research Council--promotes innovation in transportation
through research.
[5] GAO, Executive Guide: Information Technology Investment
Management: A Framework for Assessing and Improving Process Maturity,
[hyperlink, http://www.gao.gov/products/GAO-04-394G] (Washington,
D.C.: March 2004).
[6] To adapt the information technology framework as criteria for
assessing ACRP processes, we (1) simplified it to focus on the most
critical processes, rather than on organizational maturity which was
extensively addressed in the original framework; (2) changed the
terminology to refer to applied research, rather than information
technology; and (3) added criteria on product dissemination that we
determined are important to research, but are not in the original
framework.
[7] Federal Aviation Administration, Airport Cooperative Research
Program, 2005-2007: Report of the Secretary of Transportation to the
United States Congress (Washington, D.C., June 1, 2008). In addition,
the proposed FAA reauthorization bill would make the program
permanent. Aviation Safety and Investment Act of 2010, H.R. 1586, §
907, 111th Cong. (2009).
[8] In fiscal year 2005, FAA used $3 million from its Facilities and
Equipment Account to fund ACRP, as requested by H.R. Rep. No. 108-671
(2004). Since fiscal year 2006, ACRP appropriations have been drawn
from FAA's Airport Improvement Program funds.
[9] TRB has a long history of administering industry-driven
cooperative research programs focused on different transportation
sectors, having managed NCHRP since 1962 and TCRP since 1992. TRB also
began administering the National Cooperative Freight Research Program
and the Hazardous Materials Cooperative Research Program in 2006,
after ACRP was established. A cooperative research program for rail is
currently being established.
[10] The four national associations are the Air Transport Association,
Airports Council International--North America, American Association of
Airport Executives, and National Association of State Aviation
Officials. The Administrator of NASA is identified as one of the AOC's
ex-officio members in the MOA, but to date, NASA has not actively
participated on the AOC. The Environmental Protection Agency is also
listed as an ex-officio member and has participated on the AOC.
[11] See [hyperlink, http://www.gao.gov/products/GAO-04-394G].
[12] As part of its project selection responsibilities, the AOC
decides whether to fund ACRP's synthesis and legal research
subprograms, but it generally does not approve specific projects to be
covered in these subprograms. Instead, individual synthesis and legal
research projects are typically identified and approved by separate
panels established to oversee each of the subprograms.
[13] Transportation Research Board, Current and Emerging Issues Facing
the Airport Industry, ACRP Research Results Digest 5 (Washington,
D.C., January 2009) presents the 58 topics for potential research, 10
critical issue areas, and recommendations for identifying targeted
focus areas in problem statement solicitations.
[14] Previous emphasis areas have included information technology for
fiscal year 2009 and future finance and business strategies for fiscal
year 2010.
[15] Ex-officio members on the AOC explicitly represent the views of
their respective organizations. It is not clear what impact, if any,
their role in decisions reached through consensus has on the board's
independence, which is a Vision 100 requirement, because no clear
guidance or legislative history is available to conclusively determine
the meaning of an independent governing board in the context of ACRP.
[16] ACRP publications are available at [hyperlink,
http://www.trb.org/Publications/PubsACRPPublications.aspx].
[17] Transportation Research Board, Dissemination of ACRP Research
Results, ACRP Project #11-05 (ongoing).
[18] Transportation Research Board, Evaluating ACRP Processes, ACRP
Project #11-06 (ongoing) and Transportation Research Board,
Dissemination of ACRP Research Results, ACRP Project #11-05 (ongoing).
[19] For example, TCRP occasionally surveys its user community to
learn how its research results have been used and to gauge
satisfaction with TCRP's products, whereas NCHRP surveys past project
panel members every few years to discover what they have heard about
ways in which the projects they worked on have been used by their
peers.
[20] Some projects have resulted in more than one product, such as a
published report and a Web-only supplement. For a current list of
ACRP's projects, including published reports, see [hyperlink,
http://www.trb.org/ACRP/ACRPProjects.aspx].
[21] James H. Grothaus, Thomas J. Helms, Shaun Germolus, Dave Beaver,
Kevin Carlson, Tim Callister, Robert Kunkel, and Ann Johnson, ACRP
Report 16: Guidebook for Managing Small Airports, for the
Transportation Research Board of the National Academies (Washington,
D.C.: Airport Cooperative Research Program, 2009).
[22] Ricondo & Associates, Inc., Booz Allen Hamilton, Inc., George
Mason University, and National Service Research, ACRP Report 20:
Strategic Planning in the Airport Industry, for the Transportation
Research Board of the National Academies (Washington, D.C.: Airport
Cooperative Research Program, 2009).
[23] Duane A. Ludwig, Cheryl R. Andrews, Nienke R. Jester-ten Veen,
and Charlotte Laqui, ACRP Report 1: Safety Management Systems for
Airports, Volume 1: Overview, for the Transportation Research Board of
the National Academies (Washington, D.C.: Airport Cooperative Research
Program, 2007) and Manuel Ayres Jr., Hamid Shirazi, Samuel Cardoso,
Jeffrey Brown, Richard Speir, Olga Selezneva, Jim Hall, Tara Puzin,
Jeff Lafortune, Fernando Caparroz, Robert Ryan, and Edward McCall,
ACRP Report 1: Safety Management Systems for Airports, Volume 2:
Guidebook, for the Transportation Research Board of the National
Academies (Washington, D.C.: Airport Cooperative Research Program,
2009).
[24] Richard Golaszewski, Gregson Helledy, Benedict Castellano, and
Robert E. David, ACRP Web-only Document 7: How Proposed ARFF Standards
Would Impact Airports, for the Transportation Research Board of the
National Academies (Washington, D.C.: Airport Cooperative Research
Program, June 17, 2009). ACRP conducted this project to assess the
impact on airports of more closely aligning airport rescue and
firefighting regulations with certain National Fire Protection
Association standards or International Civil Aviation Organization
firefighting standards.
[25] Jacobs Consultancy, Walker Parking Consultants, Mannix Group, and
DMR Consulting, ACRP Report 24: Guidebook for Evaluating Airport
Parking Strategies and Supporting Technologies, for the Transportation
Research Board of the National Academies (Washington, D.C.: Airport
Cooperative Research Program, 2009).
[26] Airport operators implement security requirements in accordance
with their TSA-approved security programs.
[27] Hollis Stambaugh, Daryl Sensenig, Teresa Copping, Maria
Argabright, Joseph Ockershausen, and Lisa Spencer, ACRP Report 12: An
Airport Guide for Regional Emergency Planning for CBRNE Events, for
the Transportation Research Board of the National Academies
(Washington, D.C.: Airport Cooperative Research Program, 2009).
[28] Transportation Research Board, Airport Research Needs:
Cooperative Solutions, TRB Special Report 272 (Washington, D.C.,
2003). This TRB report recommended the creation of ACRP and identified
airport security as a topic, among others, in which cooperative
research could be beneficial for airport operators and complement the
research conducted by federal agencies, such as DHS, in support of
their mission.
[29] The directorate is responsible for conducting basic and applied
research and advanced development, including developmental testing and
evaluation. TSA is responsible for conducting operational testing and
evaluation, operational integration, procurement, and deployment of
new technologies, including checkpoint screening technologies. GAO,
Aviation Security: DHS and TSA Have Researched, Developed, and Begun
Deploying Passenger Checkpoint Screening Technologies, but Continue to
Face Challenges, [hyperlink, http://www.gao.gov/products/GAO-10-128]
(Washington, D.C.: Oct. 7, 2009).
[30] [hyperlink, http://www.gao.gov/products/GAO-10-128].
[31] GAO, Aviation Security: A National Strategy and Other Actions
Would Strengthen TSA's Efforts to Secure Commercial Airport Perimeters
and Access Controls, [hyperlink,
http://www.gao.gov/products/GAO-09-399] (Washington, D.C.: Sept. 30,
2009).
[32] ACRP received 8 security related project proposals--out of 219
total--for consideration for fiscal year 2010 funding. A program
official commented that ACRP does not receive many security proposals,
but it is unclear whether this is due to a lack of need for security
research, a lack of program awareness among individuals working in
airport security, or as hypothesized by this official, because ACRP
may have developed a reputation for not funding security research.
[33] Federal Aviation Administration, Airport Cooperative Research
Program 2005-2007, Report of the Secretary of Transportation to the
United States Congress (Washington, D.C.: June 2008). This report was
prepared pursuant to Section 44511 to Title 49, U.S. Code.
[34] GAO, Executive Guide: Information Technology Investment
Management: A Framework for Assessing and Improving Process Maturity,
[hyperlink, http://www.gao.gov/products/GAO-04-394G] (Washington,
D.C.: March 2004).
[35] [hyperlink, http://www.gao.gov/products/GAO-04-394G].
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548: