Education Research
Education Should Improve Assessments of R&D Centers, Regional Labs, and Comprehensive Centers Gao ID: GAO-02-190 January 24, 2002Research and Development (R&D) Centers, Regional Labs, and Comprehensive Centers support the Department of Education's research agenda to various degrees. Because statutes define different missions and activities for these programs, the amount and focus of the research and other research-based activities they support varies. Education shapes the priorities that guide the research done by the R&D Centers and targets the technical assistance provided by the Comprehensive Centers through requirements in agreements with these entities. However, Education has little control over the activities of the Regional Labs because, unlike most federal education programs, neither federal nor state governments have oversight responsibility for their programs. The R&D Centers, Regional Labs, and Comprehensive Centers reported collaborating and coordinating with each other and Education and cited various factors that have either facilitated or hindered such activities. They said that they were most likely to engage in these activities when they shared a common interest in a specific student population, such as English language learners, or in a specific topic, such as assessment. Current evaluation practices for assessing the R&D Centers, Regional Labs, and Comprehensive Centers have provided only limited information about the performance of these organizations and have not been useful for making future funding decisions.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-02-190, Education Research: Education Should Improve Assessments of R&D Centers, Regional Labs, and Comprehensive Centers
This is the accessible text file for GAO report number GAO-02-190
entitled 'Education Research: Education Should Improve Assessments of
R&D Centers, Regional Labs, and Comprehensive Centers' which was
released on January 24, 2002.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
United States General Accounting Office:
GAO:
Report to Congressional Requesters:
January 2002:
Education Research:
Education Should Improve Assessments of R&D Centers, Regional Labs, and
Comprehensive Centers:
GAO-02-190:
Contents:
Letter:
Results in Brief:
Background:
Activities Reflect Legislative Mandates and Support Education‘s Agenda,
but Education Has Limited Control over Regional Labs:
Collaboration and Coordination among the R&D Centers, Regional Labs,
Comprehensive Centers, and Education Is Ongoing:
Education Needs to Improve Its Assessments of the R&D Centers, Regional
Labs, and Comprehensive Centers:
Conclusions:
Matters for Congressional Consideration:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I GAO Contacts And Staff Acknowledgments:
GAO Contacts:
Acknowledgments:
Tables:
Table 1: Key Features of the R&D Centers, Regional Labs, and
Comprehensive Centers:
Table 2: Primary Focus and Mandated Activities of the R&D Centers,
Regional Labs, and Comprehensive Centers:
Figures:
Figure 1: R&D Centers Reported Spending Most Core Funds from OERI on
Research:
Figure 2: Comprehensive Centers Reported Spending Most Core Funds from
OESE on Technical Assistance:
Figure 3: Regional Labs Reported Spending Core Funds from OERI on a
Variety of Activities:
Figure 4: Education Offices with Responsibility for Education Research,
Research-Based Activities, and Technical Assistance:
Figure 5: Locations of the R&D Centers:
Figure 6: Regions of the Regional Labs:
Figure 7: Regions of the Comprehensive Centers:
Figure 8: Organizational Structure of an R&D Center and Its
Affiliations with Other R&D Centers:
Figure 9: Regional Lab Parent Organization, In-House Programs, and
Services Provided to Partners:
Abbreviations:
ESEA: Elementary and Secondary Education Act:
OBEMLA: Office of Bilingual Education and Minority Languages Affairs:
OERI: Office of Education Research and Improvement:
OESE: Office of Elementary and Secondary Education:
R&D: Research and Development:
[End of section]
United States General Accounting Office:
Washington, DC 20548:
January 24, 2002:
The Honorable John Boehner:
Chairman, Committee on Education and the Workforce:
House of Representatives:;
The Honorable Mike Castle:
Chairman, Subcommittee on Education Reform:
Committee on Education and the Workforce:
House of Representatives:
The Honorable Pete Hoekstra:
Chairman, Subcommittee on Select Education:
Committee on Education and the Workforce:
House of Representatives:
The Congress and the administration have emphasized the importance of a
scientifically sound research base to improve teaching and learning in
the nation‘s schools. To develop this research base and promote the use
of research-based practices, the Department of Education (Education) is
charged with sponsoring research and developing and disseminating
research-based activities, such as technical assistance. Education
administers over a thousand grants, cooperative agreements, [Footnote
1] and contracts that fund educational research, development of
materials, new methods of instruction and practices in teaching,
dissemination of research results, and technical assistance activities.
Some of these grants and contracts are administered through the Office
of Educational Research and Improvement (OERI), but others are
administered elsewhere in Education.
The number of programs these grants, cooperative agreements, and
contracts support, as well as changes in the laws authorizing them, have
raised many questions about the activities the programs perform, the
extent to which the programs collaborate and coordinate with each other
and Education, and the usefulness of the evaluations that have assessed
their performance.
Consequently, you asked us to conduct a series of studies on
Education-funded research, including this examination of three programs:
Research and Development Centers (R&D Centers), Regional Educational
Laboratories (Regional Labs), and Regional Comprehensive Assistance
Centers (Comprehensive Centers). Specifically, we agreed to answer:
1. To what extent are the activities of the R&D Centers, Regional Labs,
and Comprehensive Centers consistent with their legislative mandates
and to what extent can Education shape and control these activities to
support its research agenda?
2. To what extent do the R&D Centers, Regional Labs, and Comprehensive
Centers collaborate and coordinate with each other and with Education
and what factors foster or hinder collaboration and coordination?
3. To what extent are Education‘s practices for assessing the R&D
Centers, Regional Labs, and Comprehensive Centers useful for evaluating
their performance?
To answer these questions, we sent questionnaires to the 12 R&D Centers,
10 Regional Labs, and 15 Comprehensive Centers; reviewed documents from
the R&D Centers, Regional Labs, Comprehensive Centers, and Education;
examined relevant statutes; and interviewed Education officials. We
also conducted 14 site visits to R&D Centers, Regional Labs, and
Comprehensive Centers to gather illustrative examples and to more fully
understand the processes involved in their funding and operations. In
addition, we examined the standards Education uses to assess R&D
Centers and Regional Labs and the most recent Education-funded
assessments of the R&D Centers, Regional Labs, and Comprehensive
Centers. We performed our work between January and December 2001 in
accordance with generally accepted government auditing standards. We
did not independently verify survey information reported by the R&D
Centers, Regional Labs, and Comprehensive Centers.
Results in Brief:
Activities conducted by the R&D Centers, Regional Labs, and
Comprehensive Centers reflect their legislative mandates and support
Education‘s research agenda to various degrees. Because statutes define
different missions and activities for these programs, the amount and
focus of the research and other research-based activities they support
varies. For example, law mandates that R&D Centers engage in national
research, development, and dissemination activities. In a manner
consistent with this mandate, in fiscal year 2000 they reported
spending 87 percent of the $35.5 million in core funding they received
from OERI on these activities. (See figure 1)
Figure 1: R&D Centers Reported Spending Most Core Funds from OERI on
Research:
[Refer to PDF for image]
This figure is a pie-chart depicting the following data:
R&D Centers Reported Spending Most Core Funds from OERI on Research:
Research: 73%;
Dissemination: 9%;
Development: 5%;
Collaboration: 4%;
Other: 4%;
Evaluation: 3%;
Technical Assistance: 1%.
[End of figure]
In contrast, legislation authorizes the Comprehensive Centers to provide
technical assistance and training to help state and local educational
agencies implement federal programs established by the Elementary and
Secondary Education Act (ESEA). Accordingly, in fiscal year 2000,
Comprehensive Centers reported spending 83 percent of the $28.6 million
of their core funds from Office of Elementary and Secondary Education
(OESE) on technical assistance. (See figure 2)
Figure 2: Comprehensive Centers Reported Spending Most Core Funds from
OESE on Technical Assistance:
[Refer to PDF for image]
This figure is a pie-chart depicting the following data:
Comprehensive Centers Reported Spending Most Core Funds from OESE on
Technical Assistance:
Technical Assistance: 83%;
Dissemination: 5%;
Collaboration: 4%;
Other: 3%;
Evaluation: 2%;
Development: 2%;
Research: 1%.
[End of figure]
The law requires Regional Labs to conduct a wide range of activities
that reflects primarily the needs of the regions they serve. In line
with this requirement, in fiscal year 2000 the Regional Labs reported
spending fairly equal amounts as compared to the R&D Centers and
Comprehensive Centers of their $65.2 million in core funds from OERI on
research, development, dissemination and technical assistance
activities. (See figure 3)
Figure 3: Regional Labs Reported Spending Core Funds from OERI on a
Variety of Activities:
[Refer to PDF for image]
This figure is a pie-chart depicting the following data:
Regional Labs Reported Spending Core Funds from OERI on a Variety of
Activities:
Research: 25%;
Technical Assistance: 21%;
Development: 17%;
Dissemination: 16%;
Collaboration: 8%;
Other: 7%;
Evaluation: 6%.
[End of figure]
To a large extent, Education shapes the priorities that guide the
research conducted by the R&D Centers and targets the technical
assistance provided by the Comprehensive Centers through requirements in
agreements with these entities. However, Education has limited control
over the activities of the Regional Labs because regional governing
boards, mandated by the legislation authorizing Regional Labs,
establish regional priorities and are responsible for directing the
Regional Labs in fulfilling the terms of their contracts with
Education. Therefore, the Regional Labs are unlike most federal
education programs because neither the federal government nor state
governments have oversight responsibility for their programs. As the
Congress reauthorizes the Regional Labs, it may wish to consider giving
Education responsibility for the agenda of the Regional Labs and the
quality of the products and services they produce or giving states the
responsibility by providing these funds to each state for subsequent
distribution.
The R&D Centers, Regional Labs, and Comprehensive Centers reported
collaborating and coordinating with each other and Education and cited a
variety of factors that facilitated and hindered such activities. These
programs provided a variety of examples of collaboration and
coordination. They said that they were most likely to engage in these
activities when they shared a common interest in a specific student
population, such as English language learners, or in a specific topic,
such as assessment. Relationships between the R&D Centers, Regional
Labs, and Comprehensive Centers facilitated collaboration and
coordination. Moreover, Education played a proactive role in
encouraging such activities by requiring joint activities such as
conferences as part of their funding agreements, and by identifying
areas where collaboration would be beneficial.
Current evaluation practices for assessing the R&D Centers, Regional
Labs, and Comprehensive Centers have provided only limited information
about the performance of these organizations and have not been useful
for making future funding decisions. The law requires that Education
use peer review a process that relies on knowledgeable individuals to
make independent assessments of research‘s technical and scientific
merit to evaluate OERI activities, including those undertaken by the
R&D Centers and Regional Labs. Although peer review is well accepted
and widely used throughout the government to assess the merit of
research proposals and the scientific soundness of research, it does
not directly assess research usefulness, outcomes, or effects.
Moreover, Education‘s procedures for peer reviews had a potential for
bias and were cumbersome, which limited the usefulness of their
findings. Finally, peer review is of limited value for Regional Labs
because most of their activities do not involve research. With regard
to the Comprehensive Centers, Education‘s 2000 evaluation of the
Comprehensive Centers provided useful information about the
Comprehensive Centers as a network but limited information on the
performance of individual centers. Thus, the information from the
evaluation could not be used to inform decisions pertaining to funding
of individual Comprehensive Centers or to improve ongoing practices.
Because of the shortcomings of the evaluations of R&D Centers, Regional
Labs, and Comprehensive Centers, we are suggesting that the Congress
consider directing Education to use other accepted evaluation techniques
to assess applied research, development, dissemination, and technical
assistance activities. Moreover, we are recommending to Education that
it revise its peer review standards to allow for division of labor and
greater concentration on assessing the quality of projects, services
and products, rather than reviewing procedural materials. We are also
recommending that it design future evaluations of the Comprehensive
Centers to provide information on individual centers.
Background:
The R&D Centers, Regional Labs, and Comprehensive Centers share
responsibility with other programs created by the Congress for education
research, research-based activities, and technical assistance.[Footnote
2] Many of these programs are located in six different offices
throughout Education. (See figure 4)
Figure 4: Education Offices with Responsibility for Education Research,
Research-Based Activities, and Technical Assistance:
[Refer to PDF for image]
This figure is an organizational chart indicating education offices
with responsibility for education research, research-based activities,
and technical assistance, as follows:
Secretary of Education:
* Office of Educational Research and Improvement (OERI):
- R&D Centers;
- Regional Labs;
- Eisenhower Regional Mathematics and Science Consortia;;
- Regional Technology in Education Consortia;
- Field Initiated Studies.
* Office of Elementary and Secondary Education (OESE):
- Comprehensive Centers.
* Office of Bilingual Education and Minority Languages Affairs
(OBEMLA):
- Comprehensive Centers;
- Bilingual Education Research.
* Office of Postsecondary Education (OPE):
- American Oversees Research Centers;
- International Research and Studies;
- Centers for International Business Education.
* Office of Vocational and Adult Education (OVAE):
- National Centers for Career and Technical Education;
- Adult Education-National Leadership Activities.
* Office of Special Education and Rehabilitative Services (OSERS):
- National Institute on Disability and Rehabilitation Research;
- Special Education Technical Assistance and Dissemination;
- Special Education Studies and Evaluations;
- Special Education Research and Innovation to Improve Services and
Results for Children with Disabilities;
- Early Childhood Technical Assistance;
- Rehabilitation Services Demonstration and Training Special
Demonstration.
[End of figure]
For example, the Individuals with Disabilities Education Act
established a special education research and innovation program as well
as technical assistance centers to improve services and results for
children with disabilities. The Rehabilitation Act established a
National Institute for Disability and Rehabilitation Research. The
Office of Special Education and Rehabilitative Services administers
these programs. The amount of funding received by these programs in
fiscal year 2000 ranged from $70,000 for 11 American Overseas Research
Centers to $86.5 million for the National Institute on Disability and
Rehabilitation Research.
OERI is Education‘s lead office for educational research and
development. Its goals are to promote quality and equity in education
by funding research; developing new learning materials, teaching
techniques, and methods of organizing schools; demonstrating and
evaluating promising educational practices; disseminating research-
based information; and collecting data related to schools in the United
States and other nations. Unlike other Education offices, OERI‘s
activities span all grade levels, from preschool through adult
education, and all major content areas of instruction.
The R&D Centers, established in the 1960s to increase fundamental
knowledge in education, are administered by OERI. Over the years,
legislative changes have repositioned their placement in Education. Most
recently, the Educational Research, Development, Dissemination and
Improvement Act of 1994 reorganized OERI, implementing measures that
changed the way the R&D Centers related to Education. First, the act
created the National Educational Research Policy and Priorities Board to
work with the Assistant Secretary of OERI to establish a long-term
national agenda for research, development, and dissemination activities.
Unlike previous boards, this board was charged with improving research
priorities and developing standards for evaluating OERI research,
including that done by the R&D Centers. Second, the act established five
national research institutes[Footnote 3] within OERI, each with its own
research focus, and placed the R&D Centers, as well as field-initiated
studies and other research-related programs, under the appropriate
institute. The Congress is expected to start the reauthorization of
OERI in 2002; the last reauthorization was in 1994 and was intended to
be on a 5-year cycle. Figure 5 shows the locations of the R&D Centers.
Figure 5: Locations of the R&D Centers:
[Refer to PDF for image]
This figure is a map of the United States depicting locations of the
R&D centers, as follows:
Center for the Study of Teaching and Policy (CTP):
Location: Washington state.
National Center for Postsecondary Improvement (NCPI):
Location: Northern California.
Center for Research on Evaluation, Standards, and Student Testing
(CRESST):
Location: Southern California.
Center for Research on Education, Diversity and Excellence (CREDE):
Location: Southern California.
National Center for Early Development and Learning (NCEDL):
Location: North Carolina.
Center for Research on the Education of Students Placed At-Risk
(CRESPAR):
Location: Maryland.
Consortium for Policy Research in Education (CPRE):
Location: Pennsylvania.
National Research Center on the Gifted and Talented (NRC/GT):
Location: Connecticut.
National Center for Study of Adult Learning and Literacy (NCSALL):
Location: Massachusetts.
National Research and Development Center on English Learning and
Achievement (CELA):
Location: New York.
Center for the Improvement of Early Reading Achievement (CIERA):
Location: Michigan.
National Center for Improving Student Learning and Achievement in
Mathematics and Science (NCISLA):
Location: Wisconsin.
[End of figure]
Legislation has changed the focus of the Regional Labs over time.
Originally established to resemble the labs funded by the Atomic Energy
Commission, such as the Los Alamos National Laboratory, the Congress
created the Regional Labs in 1965 to conduct long-term activities to
address national educational problems. However, funding to support these
activities was never made available, limiting the scope of the Regional
Labs to smaller-scale projects. Further, in the late 1970s, the
Congress‘s negative reactions to federally supported curriculum
projects prompted Education and the Regional Labs to discontinue all
large-scale nationally oriented curriculum projects. As a result,
Regional Labs developed an increasingly regional agenda. In 1994, the
Congress gave the governing board of each Regional Lab sole
responsibility for determining if the Regional Lab fulfilled the terms
of its contract with Education and its regional agenda. The Congress
mandated that each governing board reflects a balanced representation
of states in the region, as well as interests and concerns of regional
constituencies, including teachers and education researchers. Figure 6
shows the states included in the regions of the 10 Regional Labs.
Figure 6: Regions of the Regional Labs:
[Refer to PDF for image]
This figure is a map of the United States depicting the states
contained within the borders of the ten Regional Labs, as follows:
Northeast and Islands Laboratory at Brown University (LAB):
Connecticut;
Maine;
Massachusetts;
New Hampshire;
New York;
Rhode Island;
Vermont.
Laboratory Student Success (LSS):
Delaware;
Maryland;
New Jersey;
Pennsylvania.
Appalachia Educational Laboratory (AEL):
Kentucky;
Tennessee;
Virginia;
West Virginia.
Southeastern Regional Vision for Education (SERVE):
Alabama;
Florida;
Georgia;
Mississippi;
North Carolina;
South Carolina.
Southwestern Educational Development Laboratory (SEDL):
Arkansas;
Louisiana;
New Mexico;
Oklahoma;
Texas.
North Central Regional Educational Laboratory (NCREL):
Illinois;
Indiana;
Iowa;
Michigan;
Minnesota;
Ohio;
Wisconsin.
Mid-continent Regional Educational Laboratory (McREL):
Colorado;
Kansas;
Missouri;
Nebraska;
North Dakota;
South Dakota;
Wyoming.
Northwest Regional Educational Laboratory (NWREL):
Alaska;
Idaho;
Montana;
Oregon;
Washington.
WestEd:
Arizona;
California;
Nevada;
Utah.
Pacific Resources for Education and Learning (PREL):
American Samoa;
Commonwealth of the Northern Mariana Islands;
Federated States of Micronesia (Chuuk, Kosrae, Pohnpei, and Yap);
Guam;
Hawaii;
Republic of Marshall Islands;
Republic of Palau.
[End of figure]
Created in 1994, the Comprehensive Centers were established more
recently than the R&D Centers and the Regional Labs. The Improving
America‘s Schools Act of 1994 consolidated the functions of 48
categorical technical assistance centers[Footnote 4] that supported
programs authorized under the ESEA, including Title I, Migrant
Education, and Indian Education, into 15 Comprehensive Centers. The
Congress created the Comprehensive Centers to support comprehensive,
cross-program assistance as envisioned in the law, and placed them
under the Office of Elementary and Secondary Education (OESE) and the
Office of Bilingual Education and Minority Languages Affairs (OBEMLA).
Figure 7 show the states included in the regions served by the
Comprehensive Centers.
Figure 7: Regions of the Comprehensive Centers:
[Refer to PDF for image]
This figure is a map of the United States depicting the states
contained within the borders of the Regional of the Comprehensive
Centers as follows:
Region 1:
Connecticut;
Maine;
Massachusetts;
New Hampshire;
Rhode Island;
Vermont.
Region 2:
New York;
Region 3:
Delaware;
Maryland;
New Jersey;
Ohio;
Pennsylvania.
Region 4:
Kentucky;
North Carolina;
South Carolina;
Tennessee;
Virginia;
West Virginia.
Region 5:
Alabama;
Arkansas;
Georgia;
Louisiana;
Mississippi.
Region 6:
Iowa;
Michigan;
Minnesota;
North Carolina;
South Carolina;
Wisconsin.
Region 7:
Illinois;
Indiana;
Kansas;
Missouri;
Nebraska;
Oklahoma.
Region 8:
Texas.
Region 9:
Arizona;
Colorado;
Nevada;
New Mexico;
Utah.
Region 10:
Idaho;
Montana;
Oregon;
Washington;
Wyoming.
Region 11:
Northern California;
Region 12:
Southern California.
Region 13:
Alaska.
Region 14:
American Samoa;
Commonwealth of the Northern Mariana Islands;
Federated States of Micronesia (Chuuk, Kosrae, Pohnpei, and Yap);
Guam;
Hawaii;
Republic of Marshall Islands;
Republic of Palau.
[End of figure]
The amount of resources allocated to the R&D Centers, Regional Labs, and
Comprehensive Centers differs, but the overall investment in these
programs is modest. For example, to operate in fiscal year 2000, each
R&D Center received from $1.5 million to $6.6 million, each Regional Lab
received from $3.8 million to $8.6 million, while each Comprehensive
Center received from $0.9 million to $2.7 million. The core budget for
all these programs totaled about $130 million. Table 1 shows the key
features of the R&D Centers, Regional Labs, and Comprehensive Centers.
Table 1: Key Features of the R&D Centers, Regional Labs, and
Comprehensive Centers:
Program: R&D Centers;
Responsible Education office: Office of Educational Research and
Improvement;
FY 2000 core funding[A]: $35.5 million[B];
FY 2000 average and range of funding per unit to support core program:
Average: $3 million; Range: $1.5 million to $6.6 million;
Allowable award recipients: University partnerships consisting of
universities and other not-for-profit organizations.
Program: Regional Labs;
Responsible Education office: Office of Educational Research and
Improvement;
FY 2000 core funding[A]: $65.2 million[B];
FY 2000 average and range of funding per unit to support core program:
Average: $6.5 million; Range: $3.8 million to $8.6 million;
Allowable award recipients: Not-for-profit organizations and
universities.
Program: Comprehensive Centers;
Responsible Education office: Office of Elementary and Secondary
Education and Office of Bilingual Education and Minority Languages
Affairs;
FY 2000 core funding[A]: $28.6 million;
FY 2000 average and range of funding per unit to support core program:
Average: $1.9 million; Range: $0.9 million to $2.7 million;
Allowable award recipients: Public or private not-for-profit
organizations, universities, and consortia of these institutions.
[A] The core funding figures do not include supplemental funds that R&D
Centers, Regional Labs and Comprehensive Centers may have received
during fiscal year 2000 from Education or other agencies for special
activities.
[B] These figures are based on amounts reported to us by R&D Center,
Regional Lab, and Comprehensive Center officials. Education, however,
provided somewhat different figures, reporting $34.9 million in core
funding for the R&D Centers and $65 million for the Regional Labs.
[End of table]
Because of the importance of education research, research-based
activities and technical assistance in improving schools, many studies
have focused on the R&D Centers, Regional Labs, and Comprehensive
Centers. Education has funded various assessments of the R&D Centers
and the Regional Labs and has recently conducted an evaluation of the
Comprehensive Centers. In addition, the R&D Centers and the Regional
Labs have been studied and discussed by numerous independent
organizations, including the National Academy of Sciences, the Brookings
Institution, the RAND Corporation, and the National Educational
Research Policy and Priorities Board.[Footnote 5] Several studies,
including the 1992 National Academy of Sciences report and a 2000
Brookings Institute report, have concluded that the funds available to
OERI to support research have been, and continue to be, insufficient to
support long-term, large-scale efforts.
Activities of the Three Programs Reflect Legislative Mandates and
Support Education‘s Agenda, but Education Has Limited Control over
Regional Labs:
Laws define different missions and roles for the R&D Centers, Regional
Labs, and Comprehensive Centers, and these differences are reflected in
how these programs spend their money. R&D Centers focus on national
research priorities, such as student assessment. Although both Regional
Labs and Comprehensive Centers have a regional orientation, Regional
Labs focus on meeting the needs of the regions. Comprehensive Centers
focus on assisting customers in their regions implement federal
education agendas, such as helping to close the achievement gaps for
federally targeted groups like disadvantaged students. Education uses
funding documents and program oversight to direct and prioritize the
activities of the R&D Centers, Regional Labs, and Comprehensive Centers
and shape the agendas of the R&D Centers and Comprehensive Centers.
Unlike R&D Centers and Comprehensive Centers, the Regional Labs have
governing boards. Because these governing boards determine regional
agendas and oversee Regional Lab activities, Education has limited
ability to shape the agendas of Regional Labs or ensure accountability
for their products and services.
Laws Mandate Missions and Activities of the R&D Centers, Regional Labs,
and Comprehensive Centers:
The Congress created a separate primary focus for the R&D Centers,
Regional Labs, and Comprehensive Centers and gave them the
responsibility of performing specific activities. Because of
differences in their mandates, the R&D Centers, Regional Labs, and
Comprehensive Centers have different roles in supporting Education‘s
research agenda. The Educational Research, Development, Dissemination,
and Improvement Act of 1994 places the R&D Centers under education
research institutes in OERI, each of which addresses a specific content
area, and requires them to carry out the purposes for which the
institutes were created by conducting research and development. In
contrast, rather than requiring the Regional Labs to address a
particular content area, the act requires them to use research-based
knowledge to address the issues in the regions they serve and assigns
them an expansive array of activities to perform. Like the Regional
Labs, the Comprehensive Centers have extensive mandates that require
the Comprehensive Centers to focus on certain customers by giving
priority to schools with schoolwide[Footnote 6] programs and the
highest number of children in poverty. The law, however, allows
Education to guide the general direction of the mandated activities.
Table 2 provides the primary focus of the R&D Centers, Regional Labs,
and Comprehensive Centers and summarizes their mandated activities.
Table 2: Primary Focus and Mandated Activities of the R&D Centers,
Regional Labs, and Comprehensive Centers:
Program: R&D Centers;
Primary focus: Address national research priorities on specified content
areas or student groups;
Mandated activities: Conduct basic research, applied research, and
dissemination, which may also include development.
Program: Regional Labs;
Primary focus: Serve needs of geographical regions regarding the
implementation of broad-based systemic school improvement strategies;
Mandated activities:
* Conduct applied research projects;
* Develop and disseminate educational research products;
* Develop a plan for identifying and serving the needs of its region;
* Serve the educational development needs of the region;
* Facilitate communication between education experts, school officials,
teachers, parents, and librarians;
* Provide support, training, and technical assistance;
* Collaborate and coordinate with other technical assistance providers;
* Bring teams of experts together to develop and implement school
improvement plans;
* Collaborate with the OERI institutes;
* Consult with state educational agencies and libraries;
* Develop strategies to use schools as components in reforming education
and reviving rural communities;
* Report and disseminate information on overcoming obstacles faced by
rural schools;
* Identify successful practices that have been developed by the Regional
Labs or other educational entities in the region.
Program: Comprehensive Centers;
Primary focus: Provide technical assistance and training related to the
administration and implementation of programs authorized under ESEA;
Mandated activities:
* Provide support, training, and assistance to state and local
educational agencies, tribal divisions of education, and other
recipients of Education funds;
* Improve the quality of instruction, curricula, assessments, and other
aspects of school reform;
* Implement effective schoolwide programs;
* Meet the needs of children served by programs funded by Education;
* Implement high quality professional development;
* Improve the quality of bilingual education;
* Create safe and drug free environments;
* Implement educational applications of technology;
* Evaluate programs;
* Expand the involvement of parents in the education of their children;
* Reform schools, school systems, and the governance and management
of schools;
* Meet the special needs of schools and children in urban and rural
areas;
* Provide technical assistance, and coordinate and cooperate with
Regional Labs, Eisenhower regional consortia, literacy centers, and
other entities engaged in research, development, dissemination, and
technical assistance.
[End of table]
The spending patterns of the R&D Centers, Comprehensive Centers, and
Regional Labs reflect their mandates and missions. For example, the
Congress authorized the R&D Centers to conduct research and development
in order to increase fundamental knowledge of central issues in
education. To support this mission, they reported spending 73 percent
of the $35.5 million in core funding they received from OERI in fiscal
year 2000 on research and an additional 14 percent on development and
dissemination.[Footnote 7] The Congress created the Comprehensive
Centers to help state educational agencies, school districts, and
schools within an assigned region implement federal elementary and
secondary school programs by providing technical assistance and
training. Accordingly, the Comprehensive Centers reported spending most
of their fiscal year 2000 $28.6 million core funds from OESE 83 percent
on technical assistance. Regional Labs were authorized to conduct a
wide range of research-based activities, including applied
research,[Footnote 8] development, dissemination, and technical
assistance. Their spending reflected these purposes: the Regional Labs
reported spending 25 percent of their fiscal year 2000 $65.2 million
core funds from OERI on research, 17 percent on development, 16 percent
on dissemination, and 21 percent on technical assistance.
Most R&D Centers, three Regional Labs, and two Comprehensive Centers
used funds from other organizations, including other federal agencies,
state and local educational agencies, and foundations, to expand work
they were performing for Education. For example, one R&D Center, the
National Center for Postsecondary Improvement, used funding from the
Pew Charitable Trust to extend the scope of a study examining the impact
of state and university admission standards on secondary schools and
students. Similarly, 10 R&D Centers reported that they leveraged
additional money from other federal and state agencies and not-for-
profit organizations to promote Education‘s research agenda. Two of
these R&D Centers reported receiving more funding from these other
sources of funding than from OERI. Three Regional Labs also reported
receiving funds from non-OERI sources, with these funds composing
between 1 to 27 percent of their budgets. Two Comprehensive Centers
reported receiving external funding and these funds accounted for 0.3
to 2 percent of those Centers‘ funding.
Education‘s Ability to Shape R&D Center, Regional Lab, and
Comprehensive Center Agendas Varies:
Education shapes the activities of the R&D Centers, Regional Labs, and
Comprehensive Centers through its funding documents and program
monitoring. Education uses funding documents, such as grant
announcements and statements of work, to guide and direct activities
included in the mandates and to obtain help in implementing department
activities. In addition, Education assigns program officers to oversee
the activities of the R&D Centers, Regional Labs, and Comprehensive
Centers.
Although priorities in R&D Center cooperative agreements are broad and
do not impose particular methods for researching a topic, they are
specific enough to shape the direction and breadth of the R&D Centers‘
research agenda. For example, in spelling out the priorities for the
R&D Center for enhancing young children‘s development and learning,
Education identified topics, theories, and research areas that the
Center should address. Similarly in spelling out the priorities for the
R&D Centers for meeting the needs of diverse student populations,
Education identified topics, theories, and student populations to be
studied.
In addition, the R&D Centers‘ cooperative agreements require that a
portion of R&D Centers‘ funds be spent on tasks defined by OERI. R&D
Centers‘ cooperative agreements require them to reserve 5 percent of
their core funds to carry out OERI initiated activities that assist
OERI in carrying out its responsibilities. For example, OERI may
require an R&D Center to write a briefing paper or conduct a research
project.
Contracts between Education and the Regional Labs give the Regional
Labs control over most of their activities, but also give Education the
opportunity to guide some of their work. Regional Labs identify the
critical issues in their region and develop plans to address these
issues. However, these contracts also allow Education to assign each
Regional Lab a broad specialty area for example, early childhood
education or educational technology”that reflects the national
education agenda and is aligned with a dominant theme of an OERI
research institute with which they are associated.[Footnote 9] In
making Regional Labs responsible for a particular specialty
area, the contracts required that they (1) conduct development, applied
research, and dissemination in that area; (2) keep abreast of
developments in their designated field; (3) provide subject area
expertise to other labs; and (4) work cooperatively with OERI
institutes as appropriate. Although Education defines the general type
of activities for Regional Labs in specialty areas, the governing
boards determine the focus of the activities and the extent to which
they met the requirements of the contract. Additionally, the contracts
require Regional Labs to work together on areas of concern to all
Regional Labs, such as how to effectively disseminate their products
and develop a telecommunications network. Regional Labs reported
spending about 25 percent of their funds on these required national
activities and on other activities with a national purpose. Unlike the
cooperative agreements for the R&D Centers, Regional Lab contracts do
not require them to do work to support OERI activities. Prior to 1995,
the contracts required that the Regional Labs spend 1 percent of their
core funds from OERI to support OERI. However, this requirement was
dropped because of objections from the Regional Labs. Currently,
Regional Labs may agree to perform work for OERI in return for
additional funding.
Although the Comprehensive Centers are not research entities, Education
shapes their activities by setting priorities for them in funding
documents. Such priorities include, for example, meeting with school
district officials to review and provide advice on district procedures
for meeting federal requirements and assisting school districts in the
development of student assessments. In addition, Comprehensive Centers
are required to engage in common activities identified by Education
such as conducting annual conferences on school improvement activities
that promote Education‘s agenda.
In addition to funding documents, program officers who administer grants
and contracts may play an important role in influencing activities of
the R&D Centers and Comprehensive Centers by ensuring that work
performed is consistent with work proposed and that funds are being used
as effectively and efficiently as possible. Program officers are
supposed to perform the following types of activities: (1) help to
develop funding documents and ensure that R&D Centers and Comprehensive
Centers are in compliance with these documents; (2) review progress
reports, financial reports, and products; and (3) approve dissemination
plans, staffing changes, and activities funded by other sources.
Program officers reported using these oversight functions to ensure
that the activities of R&D Centers and Comprehensive Centers are
consistent with their proposals. For example, Education program
officials reported that they identified activities that were
inconsistent with those in the R&D Centers‘ proposals and subsequently
negotiated alternative activities, indicated where collaboration
between some R&D Centers would be beneficial, and encouraged R&D
Centers to drop nonpromising lines of research. Program officers also
may play a role in determining supplemental and future funding
decisions.
Even though regional governing boards are responsible for the oversight
of Regional Labs, Education‘s program officers, nonetheless, may still
have the potential to influence the activities of these labs. A program
officer assigned to a Regional Lab described their role as that of
’critical friends“ who use their professional expertise and
interpersonal relationships with Regional Labs‘ staff to influence the
activities of the Regional Labs. In addition, Regional Lab program
officers, like R&D Center program officers, may play a role in
determining supplemental and future funding decisions.
Time and authority may limit program officers‘ ability to exercise their
influence. In OERI, one program officer is generally assigned to each
R&D Center and Regional Lab. The program officers we interviewed
reported spending about 50 percent of their time on monitoring
activities related to the R&D Centers and Regional Labs because of
other assigned responsibilities.[Footnote 10] Because the R&D Centers
and Regional Labs are complex organizations and prolific producers of
products and services, officers have to be very selective in targeting
their own time. Only one program officer is assigned to monitor all 15
Comprehensive Centers.
Regional Labs‘ Governance Structure Limits Education‘s Ability to
Provide Product Oversight and Shape Their Agenda:
The Congress has consistently given education program oversight to
either a federal agency”usually Education or to the states, since the
states are generally responsible for the education of their students.
The Regional Labs are unlike other federal education programs because
neither the federal government nor state governments have oversight
responsibility for programs. Specifically, Education has little control
over Regional Labs because regional boards govern them. This occurs in
spite of the fact that the Regional Labs get the largest share of the
federal dollars devoted to these three programs that conduct or support
research. The law requires that the Regional Labs establish governing
boards with regional representatives. The regional boards have sole
responsibility for determining the regional agenda and for determining
whether the Labs are fulfilling the responsibility of their contracts,
even though these contracts are funded by Education.[Footnote 11] Even
for the national specialty areas, Education sets only wide parameters
while the governing boards determine specific activities. Education
limits its communications with Regional Labs mainly to administrative
issues, according to a director we interviewed at a Regional Lab. In
addition, Education‘s Regional Lab program officers told us their
comments on Regional Lab work and products were only advisory.
The law requires each Regional Lab to establish a governing board that
reflects both a balanced representation of the states in the region and
the concerns of regional constituencies and includes teachers and
education researchers. In addition, Education requires that every chief
state school officer in the area served by the Regional Labs be offered
an opportunity to serve on the board or to designate a representative.
These safeguards, however, do not assure that priorities of each state
in the region will be met. A variety of factors potentially make
achieving balanced regional and state representation challenging.
First, Regional Labs are not given guidance on how to obtain balance
between states and regional interests, and regions, unlike states, have
no formal governing body to establish educational priorities. Second,
the governing boards, not the states, have the authority to determine
how members are nominated and selected. Finally, states within regions
vary substantially in size, population density, poverty levels, and
ethnic composition. These factors may determine state educational
priorities in a way that would make them vary widely. For example,
California”a diverse and large state in size and population”is included
with Utah, Nevada and Arizona in one region. New York also a diverse
and large state”is included with Vermont and other New England
states in another region.
Collaboration and Coordination among the R&D Centers, Regional Labs,
Comprehensive Centers, and Education Is Ongoing:
R&D Centers, Regional Labs, and Comprehensive Centers provided many
examples of collaborative and coordinated activities. R&D Centers,
Regional Labs, and Comprehensive Centers are required to collaborate
and coordinate with each other and Education. The programs reported
they are most likely to engage in collaborative and coordinated
activities when they share a common interest in a specific student
population, such as English language learners, or a specific topic,
such as assessment. Partnerships between programs, common memberships
in consortia, and staff members who are employed by more than one of
these organizations facilitate collaboration and coordination and help
leverage resources. Education plays a proactive role in promoting
collaboration and coordination by including requirements for certain
activities as part of its funding documents and in its ongoing
negotiations with these organizations. However, certain factors
differences in student populations, funding uncertainty, and
competition reduce opportunities for collaboration and coordination.
R&D Centers, Regional Labs, and Comprehensive Centers Collaborate and
Coordinate in Various Ways:
R&D Centers, Regional Labs, and Comprehensive Centers collaborate and
coordinate with each other and Education as required by law and the
documents that control their funding. Certain types of collaboration and
coordination flow naturally from overlapping needs, interests, and
resources. For example, Regional Labs and Comprehensive Centers with
populations of students with limited English proficiency would naturally
make use of the R&D Centers, Regional Labs, and Comprehensive Centers
that have expertise in that area.
R&D Centers, Regional Labs, and Comprehensive Centers reported a
variety of collaborative efforts for fiscal year 2000, including joint
projects and training. Joint projects included:
* Two Comprehensive Centers (the Northern California Comprehensive
Center and the Southern California Comprehensive Center) worked
together to produce teleconferences to help low performing schools.
* Ten Regional Labs and an R&D Center (the National Center for Early
Development and Learning) produced a training guide entitled Continuity
in Early Childhood: A Framework for Home, School, and Community
Linkages.
* An R&D Center (the Center for Research on the Education of Students
Placed At-Risk) and a Regional Lab (the Northwest Regional Lab)
produced a joint publication about parent involvement in schools.
* A Regional Lab (the Northwest Regional Lab) and an R&D Center (the
Center for Research on Evaluation, Standards, and Student Testing)
created the Classroom Assessment Tool Kit.
Examples of training included:
* A Comprehensive Center (the Southern California Comprehensive Center)
trained other Comprehensive Centers to teach instructors how to coach
children learning to read.
* An R&D Center (the Center for Research on Evaluation, Standards, and
Student Testing) trained a Regional Lab (WestEd) to use a data
collection tool.
* An R&D Center (the National Center for Improving Student Learning and
Achievement in Mathematics and Science) worked with a Comprehensive
Center (the Region VI Comprehensive Center) to conduct a professional
development project by teaching people to train math and science
teachers.
* A Regional Lab (the Southwest Educational Development Laboratory)
provided training to a staff member of a Comprehensive Center (the
Southeast Comprehensive Center) in the use of Flashlight and Compass: A
Collection of Tools to Promote Instructional Coherence a tool for
establishing teacher study groups.
R&D Center, Regional Lab, and Comprehensive Center staff also provided
many examples of coordination efforts meant to ensure that each was
aware of the others‘ projects. Some examples include the following:
* As a follow-up to Education‘s National Awards for Model Professional
Development, three Regional Labs, WestEd, the North Central Regional
Lab, and the Mid-Continent Regional Lab, studied how the awarded
districts supported districtwide teacher and student learning. An R&D
Center, the Center for the Study of Teaching and Policy, shared its data
on resource allocation among some of the same districts and contributed
to the research questions and design. The Regional Labs shared the
findings with Education.
* An R&D Center, Center for Research on Education, Diversity, and
Excellence, reviewed work on Spanish Writing Assessment done by a
Regional Lab, Northwest Regional Lab
Legislative Requirements and Common Interests Foster Collaboration and
Coordination:
Legislation requires that the Regional Labs and Comprehensive Centers
collaborate and coordinate with each other and with the R&D Centers, but
their mandates differ in the amount of collaboration and coordination
they require. The Regional Labs are required to collaborate and
coordinate with each other, Education-funded technical assistance
providers, and OERI institutes, and to share and plan joint activities
with other Education-funded state, and federal programs. They are also
required to establish a network for sharing information, planning
activities involving multiple regions, and working on national
projects. The Comprehensive Centers are required to share information,
coordinate services, and work cooperatively with the Regional Labs, R&D
Centers, Education‘s regional offices, state and local educational
agencies, and all other Education-funded research, development,
dissemination, and technical assistance programs. The R&D Centers do
not have specific legislative requirements to collaborate and
coordinate with other Education-funded programs; however, they are
required to do so in their cooperative agreements.
R&D Centers, Regional Labs, and Comprehensive Centers collaborate and
coordinate when they have an interest in the same student population.
The following examples illustrate how student populations provide a
focal point for collaboration and coordination.
* A member of the Mid-Continent Regional Lab, which includes states
with large numbers of Native Americans, sat on the steering committee
of the National Research Center on the Gifted and Talented, an R&D
Center. This committee oversaw the production of a publication on
talented American Indian and Alaskan Native students.
* Two R&D Centers, the Center for Research on Education, Diversity,
and Excellence and the National Center for Improving Student Learning
and Achievement in Mathematics and Science, published a newsletter on
issues related to how diverse students learn math and science.
* All 15 of the Comprehensive Centers created an Internet mailing list
about English-language learners to share information from their
regions, identify staff with proficiency in meeting the needs of
English-language learners, and disseminate information.
Interlacing Organizational Relationships Generally Facilitate
Collaboration and Coordination:
The structures of the R&D Centers, Regional Labs, and Comprehensive
Centers foster collaboration and coordination among them and other
entities. In some cases, partnerships may encourage collaboration. In
other cases, collaboration occurs because one single entity operates
both an R&D Center, Regional Lab, and/or Comprehensive Center.
Partnerships may encourage collaboration by establishing bridges
between programs. R&D Centers, Regional Labs, and Comprehensive
Centers are programs that operate through parent organizations
universities, not-for-profit organizations, and educational agencies.
These parent organizations create consortia”formal partnerships with
other universities, not-for-profit organizations, for-profit
corporations, and educational agencies”to run R&D Centers and
Comprehensive Centers. Participants in partnerships that run R&D
Centers range from 2 universities to 29 universities and not-for-profit
organizations. Figure 8 shows one R&D Center that is a consortium of 5
universities and the affiliations they have with other R&D Centers.
Figure 8: Organizational Structure of an R&D Center and Its
Affiliations with Other R&D Centers:
[Refer to PDF for image]
This figure is an illustration of the organizational structure of an
R&D Center and its affiliations with other R&D Centers, as follows:
U.S. Department of Education (administered by the Office of Educational
Research and Improvement):
Funds Granted:
Center for the Improvement of Early Reading Achievement:
* University of Michigan (Parent Organization);
* Michigan State University;
* University of Virginia;
* University of Southern California;
* University of Minnesota.
University of Michigan, affiliations With Other R&D Centers:
* Center for the Study of Teaching and Policy (CTP);
* Consortium for Policy Research in Education (CPRE);
* National Center for Postsecondary Improvement (NCPI).
University of Southern California, affiliations With Other R&D Centers:
* Center for Research on Education, Diversity and Excellence (CREDE).
[End of figure]
Unlike R&D Centers and Comprehensive Centers, the parent organizations
that run Regional Labs do not form partnerships with other
universities, not-for-profits, or educational agencies to run the
Regional Labs. They may, however, be asked by a Comprehensive Center or
R&D Center to enter into a partnership in order to provide specialized
services. We found four parent organizations that operated Regional
Labs that had formed partnerships with Comprehensive Centers. Figure 9
shows an example of such a partnership.
Figure 9: Regional Lab Parent Organization, In-House Programs, and
Services Provided to Partners:
[Refer to PDF for image]
This figure is an illustration of regional lab parent organization, in-
house programs, and services provided to partners, as follows:
Parent Organization: West Ed:
In House Programs:
* Northern California Comprehensive Center (Region XI):
- Funded by U.S. Department of Education (administered by Office of
Elementary and Secondary Education);
* Western Regional Lab;
- Funded by U.S. Department of Education (administered by Office of
Educational Research and Improvement);
* Eisenhower Regional Consortium:
- Funded by U.S. Department of Education (administered by Office of
Educational Research and Improvement);
* Other Federal, State, and Privately Funded Programs:
- Other Funding.
Services Provided to Partners:
* Operates Arizona office of Southwest Comprehensive Center (Region
IX):
- Funded by U.S. Department of Education (administered by Office of
Elementary and Secondary Education);
* Provides 3 technical assistance and training staff for New England
Comprehensive Center (Region I):
- Funded by U.S. Department of Education (administered by Office of
Elementary and Secondary Education).
[End of figure]
Participants in partnerships that run Comprehensive Centers range from
three to eight universities, not-for-profit organizations, and
educational agencies. Most parent organizations that run Comprehensive
Centers partner with other organizations to obtain expert services in
specialized areas, such as migrant education or Indian education.
Because each of these specialized partners may work with as many as
four Comprehensive Centers, these partnerships may establish bridges
that forge coordination in particular topical areas. For example,
ESCORT, a former categorical technical assistance center, specializes
in migrant education and partners with four Comprehensive Centers to
provide services in that area.
Collaboration also results when a single organization operates both an
R&D Center, Regional Lab, or a Comprehensive Center. For example, staff
from the Western Regional Lab and the Northern California Comprehensive
Center participated in a workgroup that developed a guide on how
schools could better obtain student perspectives and suggestions to
improve school planning. As shown in figure 9, both the Western
Regional Lab and the Northern California Comprehensive Center are run
by the same parent organization. Five of the 10 parent organizations
that run a Regional Lab also run a Comprehensive Center. One of the 12
parent organizations that operates an R&D Center also operates a
Comprehensive Center.
These interlacing organizational relationships allow R&D Centers,
Regional Labs, and Comprehensive Centers to leverage resources. For
example, an Education official working with the Comprehensive Centers
told us that through their parent organizations and partnerships
Comprehensive Centers are able to leverage the personnel and expertise
needed to perform their work. In those cases where parent organizations
run two programs, we were told that staff divide their time between
programs to leverage expertise. Similarly, a director of a parent
organization that runs a Regional Lab and a Comprehensive Center stated
that the Comprehensive Center draws upon experts assigned primarily to
other projects to obtain skills needed to implement particular
activities.
Individual relationships, such as staff members holding multiple
appointments within R&D Centers, Regional Labs, or Comprehensive
Centers also facilitate collaboration and coordination among R&D
Centers, Regional Labs, and Comprehensive Centers. For example, a
principal investigator for the Consortium for Policy Research in
Education, an R&D Center, also works on projects for the National Center
for Improving Student Learning and Achievement in Mathematics and
Science, another R&D Center.
Education Plays a Proactive Role in Encouraging Collaboration and
Coordination:
The funding agreements between Education and the R&D Centers, Regional
Labs, and Comprehensive Centers reflect Education‘s interpretation and
implementation of legislative requirements for collaboration and
coordination. The cooperative agreements for the R&D Centers require
them to collaborate and coordinate with Regional Labs, Comprehensive
Centers, other federal programs, policy institutions, and advocacy
groups. R&D Centers also agree, in their funding documents, to conduct
an annual meeting to share research with Education and other research
and development programs, and collaborate with OERI. Regional Lab
contracts with Education require that they participate in at least two
meetings a year convened to discuss issues related to Education-funded
programs. In addition, Education‘s contracts for Regional Labs require
Regional Lab representatives to meet with OERI annually and
chairpersons of the governing boards to meet with OERI when their
contract begins. Regional Labs also have an option in their contracts
with Education that allows them to earn supplemental funds by agreeing
to perform work in collaboration and coordination with OERI, including
sponsoring meetings and panels and writing briefs. Cooperative
agreements for the Comprehensive Centers require them to meet with
seven different Education-funded programs, including the Regional Labs,
to discuss collaboration and coordination; plan a national conference;
engage in a common project to improve teaching; and collaborate with
each other and local and state educational agencies.
Cooperative agreements for the R&D Centers and Comprehensive Centers
also outline Education‘s responsibilities to facilitate collaboration
and coordination. For example, the funding agreement for the
Comprehensive Centers specifies that Education officials will work with
Comprehensive Centers in planning conferences and identifying areas for
collaboration and coordination. The funding agreements that Education
has with Regional Labs do not outline Education‘s responsibilities for
facilitating collaboration and coordination.
Education officials may also identify appropriate areas for
collaboration and coordination. For example, an institute director told
us that she contacted an R&D Center in another institute to discuss
possible areas for collaboration and coordination with the R&D Center
in her institute. Similarly, some Comprehensive Center officials stated
that the program officer assigned to them identified areas for cross-
program collaboration and coordination and communicated with them
frequently. For example, the program officer suggested collaborating
and coordinating on the creation of a common framework for working with
low performing schools in Comprehensive Center regions.
Differences in Student Population and Topics, Funding Uncertainty, and
Competition Reduce Opportunities for Collaboration and Coordination:
Despite the efforts made by the R&D Centers, Regional Labs,
Comprehensive Centers, and Education in fostering collaboration and
coordination, barriers exist. R&D Centers, Regional Labs, and
Comprehensive Centers cited differences in student populations and
topics, uncertain funding, and competition as hindrances to
collaboration and coordination. R&D Centers have different research
focuses and conduct research on different topics. Regional Labs and
Comprehensive Centers serve diverse geographical areas with different
interests. These differences potentially reduce opportunities for
collaboration and coordination. For example, an R&D Center with a focus
on postsecondary education would have little or no reason to
collaborate or coordinate on substantive issues with other R&D Centers,
Regional Labs, and Comprehensive Centers that focus on research related
to elementary and secondary education. Funding uncertainties also make
collaboration and coordination difficult. Directors said they were
often reluctant to write collaborative activities into their proposals
because they did not know which organizations would win future funding
competitions. Competition also limits collaboration and coordination.
The education research and technical assistance business is a
competitive industry. Like others in competitive industries, R&D
Center, Regional Lab, and Comprehensive Center staff are protective of
ideas and practices that give them advantages over other organizations
that they perceive as competitors for future sources of funding.
Education Needs to Improve Its Assessments of the R&D Centers, Regional
Labs, and Comprehensive Centers:
The recently funded evaluations of the R&D Centers, Regional Labs, and
Comprehensive Centers provided limited information on outcomes of the
activities conducted by these programs. OERI is required to use peer
review to evaluate the R&D Centers and Regional Labs. Peer review is
well accepted and widely used for assessing the merit of research
proposals and the scientific soundness of research. Unlike the R&D
Centers, research is only a relatively small part of what Regional Labs
do. Their other activities dissemination and technical assistance would
have been more appropriately evaluated using methods other than peer
review. The peer review processes that Education used to evaluate the
R&D Centers and Regional Labs have shortcomings that limited the
usefulness of the findings. First, the peer review process used has the
potential for bias because the R&D Centers and the Regional Labs
selected most of the products that were reviewed. Second, the processes
were cumbersome. For example, Education required each member of a
review team to write an individual assessment report and to review all
contracts, contract modifications, progress reports, and annual updates
for a three-year period. In addition, with regard to the Comprehensive
Centers, the customer satisfaction survey evaluations of the
Comprehensive Centers did not provide information on individual
centers.
Recent Peer Reviews of the R&D Centers and Regional Labs Provided
Limited Information:
Traditionally, Education has used peer review to evaluate activities
carried out by OERI, including those conducted by the R&D Centers and
the Regional Labs. The Educational Research, Development, Dissemination,
and Improvement Act of 1994 codified this practice by requiring OERI to
develop peer review standards for evaluating and assessing the
performance of recipients of grants, cooperative agreements, and
contracts that exceed $100,000, as well as for selecting proposals for
funding and identifying exemplary and promising educational programs.
Historically, peer review has been used extensively in the selection of
proposed research projects and, to a lesser extent, to evaluate research
and development programs. Peer review entails an independent assessment
of the technical or scientific merit of research by peers who are
scientists with knowledge and expertise equal to that of the
researchers whose work they review. It is sometimes used to evaluate
research when the ultimate outcomes of the research are unpredictable.
Peer review may be appropriate for evaluating some activities conducted
by the R&D Centers but other evaluation techniques are better suited for
evaluating the many activities of the R&D Centers and Regional Labs.
According to the National Academy of Sciences, although peer review is
well suited for assessing theory-driven research with potential long-
term effects, it is less appropriate for assessing applied research,
development, technical assistance, and dissemination efforts in which
outcomes are anticipated and can be measured over a relatively short
period of time.[Footnote 12] The R&D Centers and Regional Labs engage
in many research activities that are designed to achieve practical
outcomes. Evaluation methods that measure outcomes customer surveys,
comparisons with similar programs, and controlled case studies may be
better suited for evaluating these activities.
Education‘s peer review processes have the potential for bias and were
cumbersome. Both of these conditions limited the usefulness of findings
in addressing key issues.
* The self-selection of materials for review by R&D Centers and Regional
Labs did not provide a representative cross section of their products
and services. R&D Center staff were involved in deciding which products
were to be reviewed and each Regional Lab nominated five or six major
products or services, two of which were selected for review. While this
approach allowed reviewers an in-depth look at major program
initiatives, it did not provide reviewers with a cross section of the
R&D Centers or Regional Labs‘ work, nor did it allow them to generalize
from these works to other activities.
* Selecting as reviewers for the R&D Centers and Regional Labs some
individuals who have previously evaluated the merits of the grant
applications or proposals raised questions about objectivity. It is
likely that the individuals who selected these organizations as grant or
contract recipients might want their original choices validated.
* Requiring peers to have a broad mix of skills made selection and
scheduling of reviewers difficult. Unlike other agencies that select
peers solely on the basis of their expertise in the area of work, OERI
requires that review panels include individuals with a broad range of
knowledge and experience. For example, OERI standards require peer
review panels to include individuals with in-depth knowledge of
education policy or practice and in-depth knowledge of theories and
methods of study related to the subject area. These requirements
complicated the identification of peer review panels.
* The amount of material to be reviewed was extensive in terms of the
short time frames of the assessments and the complexity of the
organizations. Over a short period of time”for example, 5 days on site
for the review of a Regional Lab, with half that time devoted to data
gathering and the other half to writing the reports”all reviewers were
required to read immense amounts of material, including funding
documents, statements of work, proposals, and progress reports, to
learn, in detail, how the programs performed their work and to write
individual reports. Some reviewers complained that they only had time
to ’scratch the surface,“ and that much of the material they had to
review was repetitive. If responsibilities could have been shared, peer
reviewers would have been able to gather more in-depth knowledge.
* In some cases, materials chosen for review by the R&D Centers were
incomplete. Reviewers of some programs noted that materials addressing
the rationale, hypotheses, controls, and usefulness of studies were
often insufficient for them to make informed judgments.
* Assessments took place midway in the funding cycles. The reviews
generally took place during the third year of a 5-year contract. As a
result, reviewers were hesitant to be critical in reports because many
studies were on going and therefore could improve by the end of the
contract.
* The dual purposes of the peer reviews inhibited candor. On one hand,
Education designed the reviews to be formative evaluations” evaluations
that were to focus on the performance of the programs in terms of their
missions and the technical quality of their products. In this regard,
reviews were to examine the overall quality of the work of the R&D
Centers and Regional Labs, the extent to which R&D Centers and Regional
Labs performed their work on time and met professional standards.
However, the reviews were also designed to assess the usefulness,
outcomes, and effects of their work to help OERI determine if the R&D
Centers and Regional Labs merited continued funding. The peer review
process depended exclusively upon expert opinion rather than directly
measuring how useful the research was or its effects. Reviewers
discussed ’the potential“ value of activities and were not able to
predict the ability of the entities to contribute substantially to the
field. Moreover, they did not believe their findings should have
influence over funding decisions, which affected what they wrote in
their reports.
Evaluation of the Comprehensive Centers Did Not Provide Comparable
Information on Individual Centers:
Education‘s evaluation of the Comprehensive Centers met the
requirements of the law but provided little information that would help
Education determine if each Comprehensive Center was meeting the
needs of its customers. The Elementary and Secondary Education Act
requires the Secretary of Education to collect information about the
availability and quality of services provided by the Comprehensive
Centers and to conduct surveys to determine if populations served by the
Comprehensive Centers are satisfied with their access to services and
the quality of those services. As part of the year 2000 evaluation, a
contractor surveyed the satisfaction of customers who had participated
in either of two activities offered by a Comprehensive Center. These
activities were selected from among all the activities offered by the
Comprehensive Center because they were long-term or intensive. The
contractors surveyed customers by randomly selecting them from a list
prepared by each Comprehensive Center. The contractor also surveyed a
nationally representative sample of state and district officials that
they identified as a likely pool of customers for the centers and a
sample of gatekeepers” individuals that had requested or negotiated for
services on behalf of their school or school district. In randomly
selecting customers to survey, the contractor did not choose a number
large enough from each Comprehensive Center‘s list to allow any
reliable generalizations to be made about a particular Comprehensive
Center. Likewise, the sample of potential customers and gatekeepers was
not suitable to draw conclusions about an individual Comprehensive
Center. The inclusion of representative activities and customers for
each Comprehensive Center would have increased the cost of the mail
survey. However, not drawing a sample that was representative at the
center level reduced the usefulness of the evaluation because Education
could not identify variation across Comprehensive Centers or obtain
information to improve practices at individual Comprehensive Centers.
Conclusions:
The Regional Labs are unlike most federal education programs because
neither the federal government nor state governments have oversight
responsibility for their programs. Under the current structure,
Education is accountable for the activities it funds through the
Regional Labs, but current laws limit its ability to oversee those
activities. Not only is federal oversight limited, states also have
limited control over the regional agenda. Although the requirement that
governing boards have a balanced representation of states in the region
may ensure state input for the agenda of the Regional Lab, the regional
priorities that the Regional Labs serve may not correspond to the
educational priorities of all states in the regions. Thus, neither the
states nor Education can ensure that the Regional Labs are meeting the
needs of the states.
Congressionally mandated peer reviews of the R&D Centers and Regional
Labs have produced limited information about the overall performance of
each organization, the services they provide, or the extent to which
teaching and learning are improved by the products R&D Centers and
Regional Labs produce. In part, this was because reviewers could not
divide the tasks among themselves, as each reviewer was required to do a
full, independent assessment. In addition, given the present practice
allowing the Regional Lab directors to choose products and services for
review, the potential exists for bias and therefore calls into question
the quality of the assessments. As a result, Education lacks
information that would be useful in making funding decisions or
improving the performance of each organization. Unless standard program
evaluation techniques, such as customer surveys or controlled case
studies, are introduced into the evaluation process of these entities,
these problems are likely to continue.
The value of the mandated year 2000 evaluations of the Comprehensive
Centers was limited. We recognize that addressing this problem would
involve expanding the sample sizes and, for the consumer survey, the
number of activities selected. However, currently, neither the
Comprehensive Centers themselves nor Education can determine if the
customers of a particular Comprehensive Center are satisfied with some
or all of its products and services. As a result, problems at any given
center could go unchecked. Moreover, Education cannot assess the
relative strengths and weakness of individual Comprehensive Centers so
it can improve the services in all centers and make better funding
decisions.
Matters for Congressional Consideration:
If the Congress wishes to ensure greater accountability to a
governmental entity for the Regional Labs, it could consider either
giving Education responsibility for determining the regional agenda and
overseeing the products and services of the Regional Labs or Education
could provide these funds to states, possibly as part of a larger
formula grant, for subsequent distribution by each state. This would
give states greater control in purchasing research-based educational
products and services.
If the Congress wants to increase the usefulness of the assessments of
the R&D Centers and Regional Labs, the Congress should consider allowing
Education to use methods other than peer review when such methods are
more appropriate than peer review for evaluating the activities of R&D
Centers and Regional Labs.
Recommendations for Executive Action:
To improve the assessments of the R&D Centers and Regional Labs, we
recommend that the Secretary of Education direct the Assistant Secretary
of OERI to:
* use random selection of projects, services and products to be reviewed
when conducting future evaluations of R&D Centers and Regional Labs,
and:
* revise the peer review standards to allow for division of labor and
greater concentration on assessing the quality of projects, services
and products.
To improve the performance and usefulness of the Comprehensive Centers,
the Secretary of Education should direct the Assistant Secretary of
Elementary and Secondary Education and the Assistant Secretary of
Bilingual Education and Minority Languages Affairs to:
* design future evaluations of the Comprehensive Centers to produce
findings pertaining to individual Comprehensive Centers.
Agency Comments:
We provided a draft of this report to the Department of Education for
comment. Education‘s Executive Secretariat confirmed by e-mail that
Department officials had reviewed the draft and had no comments except
for a few technical clarifications regarding funding.
We are sending copies of this report to the Secretary of Education, the
appropriate congressional offices, and other interested parties. If you
have any questions, please contact me at (202) 512-7015 or Eleanor L.
Johnson (202) 512-7209. Other contributors can be found in appendix I.
Signed by:
Marnie S. Shaul:
Director, Education, Workforce, and Income Security Issues:
[End of section]
Appendix I: GAO Contacts and Staff Acknowledgments:
GAO Contacts:
Eleanor L. Johnson (202) 512-7209:
Kathleen D. White (202) 512-8512:
Acknowledgments:
In addition to those named above, Malcolm Drewery, Jr., Tahra N.
Edwards, Richard B. Kelley, and Sarah Moorhead made key contributions
to this report.
[End of section]
Footnotes:
[1] A cooperative agreement is a type of grant used by agencies when
substantial involvement and interaction is expected of both the agency
and the recipient in carrying out the activities in the agreement.
[2] Education funds many additional research and support entities,
including Special Education and Early Childhood Regional Resource
Centers, Eisenhower Regional Math and Science Consortia, and National
Centers for Vocational Education.
[3] The 5 OERI institutes are relatively small. Each employed between 8
to 14 professional and support staff. In contrast, the National
Institute of Mental Health, one of the Department of Health and Human
Services 19 institutes, employed over 400 staff at its headquarters
office.
[4] Categorical programs typically permit funds to be used only for
specific, narrowly defined purposes and populations, such as migrant,
Indian, or economically disadvantaged children.
[5] The National Academy of Sciences, Center for Education, Committee
on Scientific Principles in Education Research, Science, Evidence, and
Inference in Education: Report of a Workshop (Washington, D.C.:
National Academy Press, 2001); The National Academy of Sciences,
Committee on Scientific Principles in Education Research, Research and
Educational Reform: Roles for the Office of Educational Research and
Improvement (Washington, D.C.: National Academy Press, 1992); The
Brookings Institution, Can We Make Education Policy on the Basis of
Evidence? What Constitutes High Quality Education Research and How Can
It Be Incorporated Into Policymaking?“ (Washington, D.C.: 1999); RAND
Corporation, Setting Standards for Education Research (forthcoming);
National Educational Research Policy and Priorities Board, U.S.
Department of Education, Investing in Learning : A Policy Statement
with Recommendations on Research in Education (Washington, D.C.: 1999)
and A Blueprint for Progress in American Education (Washington, D.C.:
2000).
[6] Schoolwide programs combine resources from various Education
programs, such as those authorized by the Individuals with Disabilities
Education Act and the Safe and Drug Free Schools and Communities Act,
to enhance teaching and learning for all students in a school.
[7] We pretested the survey with officials of the R&D Centers and
Comprehensive Centers. These officials agreed that the general
categories of activities”research, dissemination, technical assistance,
collaboration, development and evaluation”were sufficiently distinct.
However, in follow-up interviews with respondents, some noted the
difficulty in separating certain interrelated activities from one
another. For example, respondents reported difficulty in separating
technical assistance from dissemination, since both might take place
during the course of one activity. Consequently, percentages reported
are estimates.
[8] The terms ’basic research“ and ’applied research“ are found in the
authorizing legislation for the R&D Centers and ’applied research“ is
found in the authorizing legislation for the Regional Labs, but the
legislation‘s definition for research does not distinguish between
basic and applied research. Similarly, the congressionally mandated
National Education Research Policies and Priorities Board discusses
these terms in Investing in Learning: A Policy Statement with
Recommendations on Research in Education (Washington, D.C.: 1999) but
gives no formal definitions. Further, the Board concluded that Education
conducts no basic research. Rather, such research is conducted in other
federal agencies, most notably the National Institute for Child Health
and Human Development, the Office of Naval Research, and the National
Science Foundation. Education concentrates on applied research and,
according to the Board, is contributing some of the important applied
research aimed at comprehensive or standards-based reform, testing, and
assessment.
[9] The contracts awarded for December 1995 through December 2000
contained tasks for specialty areas. The contracts awarded for December
2000 through December 2005 replaced ’specialty area“ tasks with
’national leadership area“ tasks, such as educational leadership and
teacher development, and required that the leadership role include
synthesizing research, disseminating information, and providing
training to other Regional Labs.
[10] Program officers said they spend the other 50 percent of their
time on a wide range of activities including working on field initiated
studies and interagency research projects, writing statements of work
and cooperative agreement announcements, coordinating with a number of
Education offices, and monitoring other OERI programs.
[11] Education provides financial oversight, but does not oversee the
Regional Labs‘ products and services.
[12] National Academy of Sciences, Committee on Science, Engineering,
and Public Policy, Evaluating Federal Research Programs (Washington,
D.C.: National Academy Press, 1999).
[End of section]
GAO‘s Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO‘s commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO‘s Web site [hyperlink,
http://www.gao.gov] contains abstracts and full text files of current
reports and testimony and an expanding archive of older products. The
Web site features a search engine to help you locate documents using
key words and phrases. You can print these documents in their entirety,
including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ’Today‘s Reports,“ on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
[hyperlink, http://www.gao.gov] and select ’Subscribe to daily E-mail
alert for newly released products“ under the GAO Reports heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov:
(202) 512-4800:
U.S. General Accounting Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
