Department of Education
Improved Dissemination and Timely Product Release Would Enhance the Usefulness of the What Works Clearinghouse Gao ID: GAO-10-644 July 23, 2010In connection with the Omnibus Appropriations Act, 2009, GAO was required to study the What Works Clearinghouse (WWC), a federal source of evidence about effective education practices. Operating through a 5-year contract awarded by the U.S. Department of Education's Institute of Education Sciences (IES), the WWC reviews education research and disseminates its findings. GAO examined: (1) the extent to which the WWC review process meets accepted standards for research evaluation and how the WWC has responded to recommendations and criticism, (2) how WWC output and costs have changed over time and how its performance is measured, and (3) how WWC products are disseminated and how useful educators find them to be. To conduct its work, GAO reviewed WWC-related documents, analyzed cost and performance data, surveyed all states and a nationally representative sample of school districts, and interviewed IES officials, WWC contractors, researchers, and others.
GAO as well as a congressionallymandated panel of experts, found that the WWC's review process, which includes screening studies to determine if they meet WWC criteria, follows accepted standards for evaluating research on the effectiveness of education interventions. WWC is responding to recommendations made by the expert panel to further improve its review and reporting processes. For example, the panel recommended improvements in the way the WWC presents information to readers on the reasons why studies do not qualify for review. The WWC is revising a report template to include a table summarizing which studies met or did not meet WWC criteria for evaluating research. The WWC has also responded to researchers who have criticized the WWC for presenting limited information because its screening criteria exclude some rigorous research designs that may be appropriate for evaluating certain education programs, such as special education. The WWC responded to this criticism by creating new standards that include two additional study designs and by creating a new product, called a practice guide, which includes a wider range of research. WWC's report output and scope increased under the current contract. For example, the WWC increased its production of various reports, introduced new products, and developed new processes for evaluating research. However, IES had a substantial backlog in its product review process from January 2009 to May 2010. The backlog generally decreased the timeliness of WWC reports, with 20 reports being delayed by up to 6 months. To support the increases in output and scope, WWC's costs doubled from the previous contract to the current one. Both contracts designated about 60 percent of costs to production, while the other 40 percent of costs support other tasks, such as communications, dissemination, and process development. IES' performance goals for the WWC primarily relate to the number of reports produced. However, IES has not developed performance measures related to the cost or usefulness of WWC products. Education uses WWC contractors, Regional Educational Laboratories (RELS) and the Doing What Works (DWW) Web site to disseminate information about WWC products; however, awareness and use of the WWC varies among states, districts, teachers, and principals. WWC contractors disseminate product information in various ways including email alerts and presentations. The RELs host events featuring WWC products for state, district, and school officials and DWW provides resources to educators based on WWC products. Based on our survey, officials from 33 of 38 state education agencies that responded to our survey and an estimated 42 percent of school districts have heard of the WWC. Those states and school districts generally used the WWC to a small or moderate extent to inform decisions on effective practices. Based on our survey, states and school districts reported that they would likely increase their use of the WWC if it included a broader array of information or more timely information. GAO recommends that IES: develop and implement strategies to avoid backlogs in WWC product reviews; establish performance measures related to costs and usefulness; and improve dissemination efforts to promote awareness and use of the WWC. Education generally agreed with GAO's recommendations.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Cornelia M. Ashby Team: Government Accountability Office: Education, Workforce, and Income Security Phone: (202) 512-8403GAO-10-644, Department of Education: Improved Dissemination and Timely Product Release Would Enhance the Usefulness of the What Works Clearinghouse
This is the accessible text file for GAO report number GAO-10-644
entitled 'Department Of Education: Improved Dissemination and Timely
Product Release Would Enhance the Usefulness of the What Works
Clearinghouse' which was released on July 23, 2010.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
United States Government Accountability Office:
GAO:
July 2010:
Department Of Education:
Improved Dissemination and Timely Product Release Would Enhance the
Usefulness of the What Works Clearinghouse:
GAO-10-644:
GAO Highlights:
Highlights of GAO-10-644, a report to congressional committees.
Why GAO Did This Study:
In connection with the Omnibus Appropriations Act, 2009, GAO was
required to study the What Works Clearinghouse (WWC), a federal source
of evidence about effective education practices. Operating through a 5-
year contract awarded by the U.S. Department of Education‘s Institute
of Education Sciences (IES), the WWC reviews education research and
disseminates its findings. GAO examined: (1) the extent to which the
WWC review process meets accepted standards for research evaluation
and how the WWC has responded to recommendations and criticism, (2)
how WWC output and costs have changed over time and how its
performance is measured, and (3) how WWC products are disseminated and
how useful educators find them to be. To conduct its work, GAO
reviewed WWC-related documents, analyzed cost and performance data,
surveyed all states and a nationally representative sample of school
districts, and interviewed IES officials, WWC contractors,
researchers, and others.
What GAO Found:
GAO as well as a congressionally mandated panel of experts, found that
the WWC‘s review process, which includes screening studies to
determine if they meet WWC criteria, follows accepted standards for
evaluating research on the effectiveness of education interventions.
WWC is responding to recommendations made by the expert panel to
further improve its review and reporting processes. For example, the
panel recommended improvements in the way the WWC presents information
to readers on the reasons why studies do not qualify for review. The
WWC is revising a report template to include a table summarizing which
studies met or did not meet WWC criteria for evaluating research. The
WWC has also responded to researchers who have criticized the WWC for
presenting limited information because its screening criteria exclude
some rigorous research designs that may be appropriate for evaluating
certain education programs, such as special education. The WWC
responded to this criticism by creating new standards that include two
additional study designs and by creating a new product, called a
practice guide, which includes a wider range of research.
WWC‘s report output and scope increased under the current contract.
For example, the WWC increased its production of various reports,
introduced new products, and developed new processes for evaluating
research. However, IES had a substantial backlog in its product review
process from January 2009 to May 2010. The backlog generally decreased
the timeliness of WWC reports, with 20 reports being delayed by up to
6 months. To support the increases in output and scope, WWC‘s costs
doubled from the previous contract to the current one. Both contracts
designated about 60 percent of costs to production, while the other 40
percent of costs support other tasks, such as communications,
dissemination, and process development. IES‘ performance goals for the
WWC primarily relate to the number of reports produced. However, IES
has not developed performance measures related to the cost or
usefulness of WWC products.
Education uses WWC contractors, Regional Educational Laboratories
(RELS) and the Doing What Works (DWW) Web site to disseminate
information about WWC products; however, awareness and use of the WWC
varies among states, districts, teachers, and principals. WWC
contractors disseminate product information in various ways including
email alerts and presentations. The RELs host events featuring WWC
products for state, district, and school officials and DWW provides
resources to educators based on WWC products. Based on our survey,
officials from 33 of 38 state education agencies that responded to our
survey and an estimated 42 percent of school districts have heard of
the WWC. Those states and school districts generally used the WWC to a
small or moderate extent to inform decisions on effective practices.
Based on our survey, states and school districts reported that they
would likely increase their use of the WWC if it included a broader
array of information or more timely information.
What GAO Recommends:
GAO recommends that IES: develop and implement strategies to avoid
backlogs in WWC product reviews; establish performance measures
related to costs and usefulness; and improve dissemination efforts to
promote awareness and use of the WWC. Education generally agreed with
GAO‘s recommendations.
View [hyperlink, http://www.gao.gov/products/GAO-10-644] or key
components. For more information, contact Cornelia Ashby at (202) 512-
7215 or ashbyc@gao.gov.
[End of section]
Contents:
Letter:
Background:
WWC Reviews Research in Accordance with Accepted Standards and Has
Responded to Recommendations and Criticisms:
WWC's Output and Costs Increased; However, IES Has Not Developed
Adequate Performance Measures Related to Cost or Product Usefulness:
Education Has Three Primary Ways to Disseminate Information about WWC
Products, but Awareness and Use Vary among Target Audiences:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Other Sources of Information Districts Use To Identify
Effective Education Practices:
Appendix III: IES and WWC Response to Expert Panel Recommendations:
Appendix IV: Comments from the Department of Education:
Appendix V: GAO Contact and Staff Acknowledgments:
Related GAO Products:
Tables:
Table 1: Products and Registries Available on the WWC Web Site:
Table 2: Two Study Designs That Meet the WWC Standards with or without
Reservations:
Table 3: New WWC Publications and Reports:
Table 4: Task Category Definitions and Changes between Contracts:
Table 5: Estimates and Confidence Intervals for Figure 8:
Table 6: Estimates and Confidence Intervals for Figure 10:
Table 7: Estimates and Confidence Intervals for Figure 12:
Table 8: Estimates and Confidence Intervals for Figure 14:
Table 9: Conferences Attended to Administer Questionnaires to Teachers
and Principals:
Figures:
Figure 1: WWC Research Review Process for Interventions:
Figure 2: Percentage of Interventions with Positive or Potentially
Positive Ratings Categorized by the Amount of the Evidence Supporting
Those Ratings:
Figure 3: Studies Reviewed That Meet WWC Evidence Standards:
Figure 4: Publication Quantities, by Contract Year (CY) for Current
WWC Contract:
Figure 5: Average Time for IES Peer Reviews of Released Intervention
Reports and Quick Reviews, by Contract Year (CY) and Quarter (Q) for
Current WWC Contract:
Figure 6: IES Peer Review Backlog for Intervention Reports and Quick
Reviews, by Contract Year (CY) and Quarter (Q) for Current WWC
Contract:
Figure 7: WWC Costs, by Task Categories and Contracts:
Figure 8: Sources from Which District Officials Heard of the WWC:
Figure 9: Extent to Which States Use WWC for Various Purposes:
Figure 10: Extent to Which School Districts That Have Used the
Clearinghouse Used It for Various Purposes:
Figure 11: Extent to Which States Use Specific WWC Products:
Figure 12: Extent of Specific Product Use among Districts That Use the
Clearinghouse:
Figure 13: Number of States That Reported They Would Likely Increase
Their Use of WWC Given Certain Changes:
Figure 14: Estimated Percent of School Districts That Have Accessed
the WWC That Would Likely Increase Their Use of the WWC Given Various
Changes:
Figure 15: GAO's Web-based Survey of State Departments of Education
and Local Educational Agencies in the 50 States and the District of
Columbia:
Abbreviations:
AYP: adequate yearly progress:
DWW: Doing What Works:
Education: U.S. Department of Education:
ESEA: Elementary and Secondary Education Act of 1965:
IES: Institute of Education Sciences:
LEA: local educational agency:
Recovery Act: American Recovery and Reinvestment Act of 2009:
REL: Regional Educational Laboratories:
WWC or Clearinghouse: What Works Clearinghouse:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
July 23, 2010:
Congressional Committees:
The U.S. Department of Education's What Works Clearinghouse (WWC or
Clearinghouse) was established as a federal source of scientific
evidence about "what works" in education. The Institute of Education
Sciences (IES), a division of the U.S. Department of Education
(Education), created the WWC in 2002, in part to help educators
identify and use scientifically-based practices as specified in the
Elementary and Secondary Education Act of 1965 (ESEA).[Footnote 1] The
WWC, which is operated by an independent contractor, conducts
systematic reviews of education research and disseminates information
on its Web site about the effectiveness of the practices reported in
these research studies. Currently operating under a $50 million 5-year
contract, the Clearinghouse has generated criticism in the education
research evaluation field on the timeliness of its reviews, its
standards for study inclusion, and the methodological soundness of its
research review process.[Footnote 2]
An explanatory statement submitted in lieu of a conference report for
the Omnibus Appropriations Act, 2009, directed GAO to examine how the
WWC reviews education research and to address concerns about the
operation, cost, and usefulness of the WWC.[Footnote 3] Specifically,
GAO was required to determine whether the WWC review process met
current standards for evaluating research and to examine the output
and cost for completing reviews, the degree of consistency of review
procedures across the various topics addressed, and the usefulness of
the Clearinghouse for practicing educators. To conduct this work, we
examined (1) the extent to which the WWC review process meets accepted
standards for research evaluation and how the WWC has responded to
recommendations and criticisms of its processes, (2) how the WWC's
output and costs have changed over time and how IES measures WWC
performance, and (3) how WWC products are disseminated and how useful
education professionals find them to be.
To address all of our objectives, we interviewed and obtained
information from IES officials and the current and former WWC
contractors, as well as representatives from various educational
organizations. In addition, to address objective 1, we reviewed a
prior GAO report that examined WWC procedures and standards, an expert
panel report that previously assessed the validity of the WWC review
process, literature, and procedures used by other organizations that
conduct systematic reviews of research. We also reviewed the
Clearinghouse's response to the expert panel and to specific
criticisms in education research literature. To determine how
performance and costs changed over time (objective 2), we analyzed the
costs and productivity of the WWC contractors by reviewing budget,
expenditure, and performance data. For objective 3, we administered a
Web-based survey to state education agencies in all 50 states and the
District of Columbia and a nationally representative sample of school
districts;[Footnote 4] interviewed IES's 10 Regional Educational
Laboratories; and gathered nongeneralizeable information from teachers
and principals at four conferences. Appendix I explains our scope and
methodology in more detail. We performed our work from September 2009
to July 2010 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objectives.
We believe that the evidence obtained provides a reasonable basis for
our findings and conclusions based on our audit objectives.
Background:
The mission of the WWC is to be a central source of scientific
evidence for what works in education.[Footnote 5] To accomplish this,
the WWC reviews existing education research and posts information
based on its reviews on the WWC Web site, http://ies.ed.gov/ncee/wwc/.
The types of products currently available on the WWC Web site are
described in table 1.
Table 1: Products and Registries Available on the WWC Web Site:
Product: Intervention reports;
Description: Summarize all of the research reviewed for a particular
intervention within a topic area. Each report offers an overview of
the intervention, summarizes all relevant research, and provides a
rating of effectiveness. Studies featured in intervention reports must
meet WWC evidence standards with or without reservations;
Number: 130;
Example: Accelerated Reader: WWC reviewed the evidence pertaining to
the effectiveness of this specific curriculum with respect to certain
reading outcomes.
Product: Practice guides;
Description: Contain recommendations for educators to address
challenges in their classrooms. Assign strength of evidence ratings to
each recommendation (strong, moderate, low). Rely to some extent on
expert opinion;
Number: 12;
Example: Structuring Out-of-School Time to Improve Academic
Achievement: WWC published general recommendations on how to design
out-of-school time programs that will increase student learning.
Product: Quick reviews;
Description: Assess the quality of research evidence from single
studies recently featured in the media to determine if they meet WWC
evidence standards;
Number: 40;
Example: Recess and Classroom Behavior: WWC reviewed a study profiled
in the news that examined whether providing daily recess to third
graders improves their classroom behavior.
Product: Multimedia;
Description: Audio files, video files, presentations, and transcripts
from WWC events;
Number: N/A;
Example: Reducing Behavior Problems in the Elementary School
Classroom: WWC held a webinar featuring a practice guide on this topic.
Product: Registry of evaluation researchers;
Description: An online database of researchers who conduct evaluations
of the effectiveness of educational interventions to help schools,
school districts, and educational program developers identify
potential researchers;
Number: N/A;
Example: Individual researchers and various organizations.
Product: Registry of randomized controlled trials;
Description: An online database of completed and in-progress
randomized controlled trials in education. This resource is designed
to help schools, school districts, and educational program developers
identify research regarding the effectiveness of educational
interventions;
Number: N/A;
Example: A Randomized Trial of Two Promising Interventions for
Students with Attention Problems: WWC included this randomized
controlled trial in its registry.
Source: GAO analysis of WWC information.
Note: This table summarizes the WWC products and registries available
as of May 18, 2010. The Clearinghouse previously published topic
reports summarizing findings from all studies on all relevant
interventions for a particular topic, such as beginning reading. These
reports were replaced by dynamically generated summaries of evidence.
[End of table]
In addition to the Clearinghouse, Education provides other technical
assistance and research-related resources to assist states, districts,
and schools. Examples of research-related resources include the
Regional Educational Laboratories (REL) and the Doing What Works (DWW)
Web site (http://dww.ed.gov):
Regional Educational Laboratories. IES's Regional Educational
Laboratory Program is a network of 10 laboratories that conduct
research and provide policymakers and practitioners with expert
advice, training, and technical assistance on how to interpret
findings from scientifically valid research.[Footnote 6]
Doing What Works. Led by Education's Office of Planning, Evaluation
and Policy Development, DWW is a Web-based resource intended to help
teachers, schools, districts, states and technical assistance
providers implement research-based instructional practice.
Initial Years of the What Works Clearinghouse:
In 2002, IES awarded a $27 million 5-year contract to the initial
contractors to operate the Clearinghouse.[Footnote 7] The WWC
contractors developed the Clearinghouse's research review standards
with IES and reviewed research related to topic areas considered to be
pressing issues in education.[Footnote 8] One of the goals of the
Clearinghouse was to promote informed education decision making
through a Web-based dissemination system that featured rigorous
reviews of studies on the effectiveness of educational interventions.
The WWC experienced a slow start due in part to the amount of work
involved in developing a research review and reporting process that
was valid, transparent, and replicable, according to the initial
contractors. In developing the research review process, the
contractors and IES addressed over 60 technical issues, such as
determining what constitutes an acceptable level of participant loss
(attrition) from a study and what methods should be in place to
accommodate common education research techniques. In addition, initial
plans for topic areas and reporting formats were modified. For
example, IES decided to drop one planned topic area because IES
officials determined it to be too broad.[Footnote 9] The WWC and IES
also spent a substantial amount of time developing and refining a
reporting format to communicate research results to a lay audience. As
a result, the WWC began releasing reports in 2006. By September 2007,
the WWC had released 89 intervention reports, six topic reports, and
three practice guides.
WWC Research Review Process for Intervention Reports:
The WWC uses a three-step review process to assess the quality of
studies and report on what the research indicates about the
effectiveness of interventions. The WWC definition of interventions
includes programs (such as whole school reform), products (such as a
textbook or curriculum), practices (such as mixed-age grouping), or
policies (such as class size reduction).[Footnote 10] The process
begins with an initial screening of published and unpublished studies
relevant to the intervention being reviewed. Studies are collected
from electronic databases, journals, conference proceedings, and
nominations solicited from the general public. The studies that pass
initial screens are reviewed to determine whether they provide valid
evidence of an intervention's effectiveness. Using these studies, the
WWC then synthesizes the evidence about the effectiveness of an
intervention and publishes a report describing its findings. The
Clearinghouse categorizes interventions as either having positive
effects, potentially positive effects, mixed effects, no discernible
effects, potentially negative effects, or negative effects (see figure
1).
Figure 1: WWC Research Review Process for Interventions:
[Refer to PDF for image: illustration]
Step 1: Initial screening:
Is the study:
* Randomized or quasi-experimental?
* Published within 20 years of the beginning of the topic area review?
* Focused on a relevant intervention to the topic under review?
And does the study:
* Target students in the topic area‘s age or grade range and specified
location?
* Focus on populations relevant to the topic area (e.g., students with
learning disabilities, English language learners)?
* Report on at least one outcome relevant to the review?
Screened out:
Studies not meeting initial screening criteria.
Step 2: Quality review:
Studies meeting initial screening criteria:
Two Ph.D.-level research analysts independently rate studies using a
codebook that considers study design and execution, validity and
reliability of outcome measures, and data analysis and reporting to
evaluate the strength of the evidence in the study.
Screened out:
Studies not meeting evidence standards.
Step 3: Synthesize evidence:
Studies meeting evidence standards (with or without reservations):
Evidence from all studies meeting standards is synthesized and
summarized for use in reports;
Intervention reports summarize evidence on the effects of a specific
intervention.
Source: GAO analysis of WWC guidelines.
[End of figure]
The WWC uses evidence standards to assess the strengths and weaknesses
of a study's methodology, such as the type of design it uses, the
quality of the study's data, and the appropriateness of the study's
statistical procedures. Until recently, the WWC accepted two types of
study designs--randomized experiments and quasi-experimental studies.
[Footnote 11] Only randomized controlled trials (or randomized
experiments) that WWC has determined to be well-designed and well-
implemented are considered strong evidence and can receive the highest
rating of "meets evidence standards without reservations." The WWC
also considers evidence from quasi-experiments it has determined to be
well-designed and well-implemented. The highest rating a study using
quasi-experimental design can receive is "meets evidence standards
with reservations." This rating category is intended to inform
educators to interpret the study results with caution, as the results
may reflect other factors, in addition to the impact of the
intervention (see table 2).
Table 2: Two Study Designs That Meet the WWC Standards with or without
Reservations:
Study design: Randomized control-group experiments;
Description: Compare the outcomes of groups that were randomly
assigned either to the intervention group or to a nonparticipating
control group before the intervention. Such an assignment helps ensure
that any differences in outcomes can be attributed to the intervention;
Highest rating category WWC will assign if well-conducted, and why:
Meets evidence standards: Considers randomized experiments as the
design that is most likely to yield unbiased estimates of a program's
impact on student outcomes.
Study design: Comparison-group quasi-experiments;
Description: Compare the outcomes of groups in which individuals are
assigned to an intervention or control group in a way that minimizes
observable differences between the groups that could affect outcomes.
The researcher must demonstrate that the groups are equivalent on
observable participant characteristics, such as age, grade level,
prior academic achievement, or pretest results;
Highest rating category WWC will assign if well-conducted, and why:
Meets evidence standards with reservations: Even with equivalent
observable characteristics, there may be differences in other
participant characteristics related to the desired outcomes--for
example, certain family or social structures that are unknown to the
researcher.
Source: GAO analysis of WWC information.
[End of table]
IES Oversight and Support of the WWC:
The WWC is administered by IES through a contract with a private
research organization. IES monitors implementation of the specific
tasks detailed in the WWC contract by reviewing an annual work plan
and monthly performance and expenditure reports submitted by the
contractor. IES tracks implementation of the tasks, completion of
performance goals, and adherence to the budget outlined in the
contractor work plan.[Footnote 12] The contractor monitors the work of
any subcontractors that it uses to perform services such as research
reviews, technological support, and communications support.
IES is also involved in the development and dissemination of WWC
products. IES reviews and approves proposed topics for WWC products,
product formats, and the research review procedures. It also
coordinates a group of independent researchers to peer review WWC
products and reviews and approves all WWC products prior to public
release. IES required the contractor to develop a communications plan
to inform WWC customers about features of the Web site.
WWC Reviews Research in Accordance with Accepted Standards and Has
Responded to Recommendations and Criticisms:
WWC Follows Accepted Review Standards and Is Improving Its Review
Process in Response to a Congressionally Mandated Expert Panel Report:
We found that the WWC review process follows generally accepted
practices for reviewing research. Specifically, GAO's November 2009
report reviewing federally supported efforts to identify effective
interventions found that the WWC determines whether a study provides
credible evidence on effectiveness based on several dimensions,
including the quality of the research design, how the study was
implemented, and other technical considerations.[Footnote 13] Our 2009
report also noted that WWC follows a common approach to conducting its
reviews,[Footnote 14] and provides information to help educators
understand the body of existing research on specific interventions.
[Footnote 15]
Additionally, a congressionally mandated panel of experts found in
October 2008 that WWC's research review process was based on
appropriate methods for judging the strength of the evidence regarding
the effectiveness of interventions.[Footnote 16] For example, the
panel agreed that the minimum qualifications a study must meet in
order to be reviewed by the WWC are appropriate. The panel also found
that WWC's reporting process is reasonable and that the WWC provides
succinct and relevant evidence on the effectiveness of education
interventions. While the panel concluded that the WWC's processes are
generally appropriate, the panel made several recommendations to the
WWC for continued improvement. The recommendations primarily related
to establishing or clarifying procedures, reviewing statistical
methods, and documenting the screening process.
The WWC implemented or is considering implementing 14 of the panel's
17 recommendations.[Footnote 17] The WWC implemented nine
recommendations, in part by modifying some procedures and creating a
procedures and standards handbook.[Footnote 18] For example, in
response to the panel's recommendation that the WWC include a table of
study dispositions (e.g., whether studies meet WWC evidence standards)
at the front of intervention reports, the WWC is modifying the report
template to include a summary table along with the existing listing of
dispositions in the reference section. The WWC also addressed panel
concerns about technical issues in its review process by making its
treatment of study attrition--the rate at which subjects drop out of a
study--more consistent across topic areas. The panel noted that the
WWC's practice of determining acceptable attrition levels by topic
area led to arbitrary inconsistencies across the topic areas. In
response to the panel's recommendation that the WWC reconsider this
practice, the WWC took steps to increase its consistency by developing
attrition guidance that applies to all topic areas.[Footnote 19] (See
appendix III for a table detailing the recommendations, WWC and IES's
response, and the status of any changes made in response to
recommendations.)
In addition, the WWC is considering implementing five other panel
recommendations. For example, the panel raised concerns that the WWC
does not document some potential conflicts of interest for the studies
it reviews. In response to this concern, the WWC is considering
tracking and publishing whether studies of a program are funded or
conducted by the program's developers.[Footnote 20] Further, in
response to the panel's concern that the WWC's screening process may
exclude some eligible studies, the WWC is undertaking an evaluation of
the reliability of its screening process. According to IES officials,
they will postpone decisions about the recommendations until the newly
appointed Commissioner for the WWC is on board and actively involved
in the decision making.
WWC Also Responded to Criticism That It Produces Limited and
Potentially Misleading Information:
Some researchers claim that the WWC presents potentially misleading
information by including brief experiments involving small numbers of
students when evaluating interventions.[Footnote 21] As a result,
according to critics, educators may accept the WWC's rating of the
intervention's effectiveness, even though the evidence behind the
rating is limited. One researcher suggested the WWC emphasize larger
studies that span significant periods of time and set a minimum sample
size requirement. According to WWC staff, such changes would exclude
valuable research and prevent the WWC from providing educators with
research-based information about some interventions.[Footnote 22]
Instead of changing its treatment of sample size and study duration,
the WWC began publishing information on the extent of the evidence
supporting its findings in 2007. The WWC's "extent of evidence" rating
alerts educators when the WWC effectiveness ratings are based on a
small amount of evidence. As figure 2 shows, 76 percent of
interventions with positive or potentially positive ratings of
effectiveness are based on a small amount of evidence (see figure 2).
Figure 3: Percentage of Interventions with Positive or Potentially
Positive Ratings Categorized by the Amount of the Evidence Supporting
Those Ratings:
[Refer to PDF for image: pie-chart]
Small (51 interventions): 76%;
Medium to large (13): 19%;
Not rated (3): 4%.
Source: GAO analysis of WWC data.
Note: The figure excludes seven interventions that were rated with
different amounts of evidence as of April 27, 2010. Currently, the
extent of evidence rating has two categories: small and medium to
large. A rating of "medium to large" requires at least two studies and
two schools across studies and a total sample size across studies of
at least 350 students or 14 classrooms. Otherwise, the rating is
"small."
[End of figure]
Further, researchers suggested that the WWC presents misleading
information by rating interventions based on studies in which measures
of student performance closely match the content taught to the
intervention group, but not the control group.[Footnote 23] In such
studies, higher test scores among the intervention group may not
accurately represent the effectiveness of the intervention more
generally. The researchers suggested that the WWC exclude such
measures, or at least report on them separately. However, the WWC
includes these measures because, according to IES officials, they
answer questions about whether different interventions lead to
different content knowledge. The WWC agrees that there is a concern
regarding the reliability of outcome measures that are overly similar
to the intervention, but maintains that WWC procedures attempt to
exclude such measures. In addition, in response to researcher concerns
that tests created by intervention developers may be biased,[Footnote
24] the WWC added information to the intervention reports noting
whether outcome measures are based on tests created by the developer.
Some researchers and education professionals we interviewed suggested
that the WWC produces limited information because its screening
criteria are too restrictive--currently screening out about 90 percent
of studies initially identified as potentially relevant (see figure
3). Until recently, the WWC reviewed only two types of study designs--
randomized experiments and quasi-experimental studies--and according
to critics, this limited the amount and type of information available
to educators.[Footnote 25] For example, staff from one REL noted that
educators may not be able to find reviews of the interventions they
are using or considering because so few studies meet WWC standards.
[Footnote 26] Staff from another REL told us that if educators cannot
find relevant and useful information, they may be discouraged from
using evidence-based practices. Staff from a third lab noted that the
narrow focus prevents educators from learning from less rigorous but
nonetheless useful research, such as case studies describing an
intervention's costs and implementation requirements.
Figure 3: Studies Reviewed That Meet WWC Evidence Standards:
[Refer to PDF for image: illustration]
Initially reviewed: 2,669 studies;
Met evidence standards with or without reservations: 226 studies;
Out of scope or did not meet evidence standards: 2,443 studies.
Source: GAO analysis of WWC data.
[End of figure]
The WWC maintains that its screening criteria and study inclusion
standards focus on studies that provide strong evidence of an
intervention's effectiveness, and lowering these standards could
undermine the validity of the findings reported by the WWC. Although
the Clearinghouse screens out most studies, many of its reports have
identified interventions with positive effects. Data from the
contractor indicate that 58 percent of WWC's intervention reports
identify positive or potentially positive effects of interventions.
While the WWC plans to continue using its methodological standards for
reviewing randomized and quasi-experimental studies, the Clearinghouse
acknowledges that the emphasis on randomized experiments and quasi-
experiments can exclude useful information on interventions in certain
topic areas, such as special education, that do not lend themselves to
these study designs. The WWC created new standards to include
additional study designs.[Footnote 27]
The WWC also introduced practice guides in 2007 in response to
criticisms that its intervention reviews exclude too much research and
consequently provide limited information to educators. Written by a
panel of experts, practice guides include recommendations for
educators on various topics, such as reducing high school drop-out
rates and reducing behavioral problems in the classroom.[Footnote 28]
Whereas WWC's intervention reviews are based entirely on studies that
meet WWC evidence standards, practice guides also incorporate studies
that do not have designs that are eligible for WWC review, or in some
cases, are reviewed and do not meet WWC evidence standards, and
include the views of experts. To develop recommendations, the practice
guide panel reviews available literature about the particular topic
and then meets several times to discuss the topic. Through consensus,
the panel identifies effective practices based on the evidence. Once
the practice guide is developed, it undergoes a quality assurance
review by WWC and IES staff and external peer review. The following
text box provides an example of practice guide recommendations and the
level of evidence supporting them.
Table: Example of Practice Guide Recommendations and Evidence
Levels[A]:
In 2009, the WWC published a practice guide to help educators assist
students struggling with reading in the primary grades. The practice
guide authors used an early detection and prevention framework known
as Response to Intervention. The panel that authored the practice
guide consisted of six researchers and one expert in implementation of
the Response to Intervention model. Two WWC staff also assisted in the
practice guide development. The panel's recommendations follow.
Recommendation:
Screen all students for potential reading problems twice per year and
monitor those with higher risk;
Basis for recommendation: Numerous studies with designs that did not
meet WWC evidence standards or that did not use samples that
adequately resembled the population of interest;
Level of evidence[B]: Moderate.
Recommendation:
Provide time for differentiated reading instruction for all students
based on assessments of students' current reading level;
Basis for recommendation: One descriptive study and expert opinion;
Level of evidence[B]: Low.
Recommendation:
Provide intensive, systematic instruction on foundational reading
skills in small groups to students who score below the benchmark score;
Basis for recommendation: 11 studies that met WWC evidence standards;
Level of evidence[B]: Strong.
Recommendation:
Monitor the progress of these students at least once a month;
Basis for recommendation: 3 studies that met WWC evidence standards,
but did not evaluate the effectiveness of monitoring so no conclusive
inferences could be made, and expert opinion;
Level of evidence[B]: Low.
Recommendation:
Provide intensive interaction on a daily basis to students who show
minimal progress after reasonable time in small group instruction;
Basis for recommendation: 5 studies that met WWC evidence standards
but did not report statistically significant impacts on reading
outcomes; Level of evidence[B]: Low.
Source: GAO review of a WWC practice guide.
[A] A strong rating indicates that studies supporting the
recommendation generally meet WWC standards. A moderate rating
indicates that studies supporting the recommendation generally meet
WWC standards with reservations. A low rating indicates the
recommendation is based on expert opinion, derived from theory or
experience, and supported with evidence that does not rise to the
moderate or strong levels.
[B] Our analysis of practice guide recommendations found that almost
half of the 67 recommendations made in the 12 practice guides released
as of May 2010 were based on a low level of evidence.
[End of table]
WWC's Output and Costs Increased; However, IES Has Not Developed
Adequate Performance Measures Related to Cost or Product Usefulness:
WWC Increased Output and Introduced New Products:
WWC's report output increased under the current contract, and its
scope expanded to include new products and processes to support
production. Under the current contract, the WWC increased its total
number of publications from the first contract year to the second
contract year and generally kept pace with its increased scope, as
specified in the Clearinghouse's annual plans.[Footnote 29] For
example, the current contract calls for the WWC to increase the number
of topic areas and intervention reports. Under the current contract,
the WWC added three new topic areas and released 60 intervention
reports, including 5 in the new topic areas as of June 2010.[Footnote
30] In addition, the WWC produces practice guides and quick reviews
and increased its production of both of these products between the
first and second year of the current contract. Figure 4 shows the
production of all three WWC products as of June 30, 2010, the end of
the third contract year.
Figure 4: Publication Quantities, by Contract Year (CY) for Current
WWC Contract:
[Refer to PDF for image: 3 horizontal bar charts]
Intervention reports:
Contract year 1: 4 publications;
Contract year 2: 37 publications;
Contract year 3: 19 publications.
Practice guides:
Contract year 1: 0 publications;
Contract year 2: 3 publications;
Contract year 3: 3 publications.
Quick reviews:
Contract year 1: 5 publications;
Contract year 2: 21 publications;
Contract year 3: 16 publications.
Source: GAO analysis of WWC data.
Notes: CY is July 1 to June 30. CY1 covers this period for 2007 to
2008, CY2 for 2008 to 2009, and CY3 for 2009 to 2010.
[End of figure]
The WWC's scope of work increased under the current contract with the
addition of new products and work processes, as well as
responsibilities related to the American Recovery and Reinvestment Act
of 2009 (Recovery Act), which provided additional innovation and
improvement funding.[Footnote 31] The WWC is developing three new
types of publications and conducts an annual review of Education-
sponsored studies for IES's internal use. Specifically, the WWC is
developing research briefs, research perspectives, and practice
briefs, which will focus on Education policy priorities. Like practice
guides, the new publications will incorporate expert opinion and a
broad range of research. Table 3 provides more information on these
new initiatives.[Footnote 32]
Table 3: New WWC Publications and Reports:
Product: Research briefs;
Description: Short summaries of what research indicates about the
effectiveness and implementation challenges of policies, practices, or
issues in education;
Status: In process and template has been approved. First publications
projected for release in 2010.
Product: Research perspectives;
Description: Researchers' perspectives on what research has found will
work in addressing pressing educational issues. Topics will initially
focus on issues relevant to the Recovery Act;
Status: In process and template has been approved. First publications
projected for release in 2010.
Product: Practice briefs;
Description: Provide explicit information on how to implement one
practice from a WWC practice guide, and provides educators with
research-based, how-to steps and strategies for overcoming roadblocks,
and tools for educators;
Status: Template has been drafted but further work on this product is
on hold pending direction from IES.
Product: Reviews of IES-sponsored studies (annual);
Description: IES uses this report to evaluate the research it funds.
WWC reviews this research using WWC standards and reports on whether
the research studies identify effective or promising practices;
Status: First produced in 2008, with plans for annual reporting to IES.
Source: GAO review of WWC contracts and annual plans.
[End of table]
IES's Reviews Have Delayed the Release of Some Reports:
While the WWC contractor increased its report production, IES's review
process did not keep pace with output. IES is responsible for
administering independent peer reviews of all products and conducting
final reviews and approvals before products are released, and has
internal time frame estimates used in scheduling and completing such
reviews. For example, according to IES planning documents, IES
estimates 15 business days for the completion of peer reviews for
intervention reports and 6 business days for WWC quick reviews.
However, throughout 2009, IES took increasingly more time to schedule
and coordinate the completion of peer reviews for some intervention
reports and quick reviews. As a result, the release of 20 reports--11
intervention reports and nine quick reviews--was delayed by more than
6 months. For example, in the first quarter of the current contract
year (third contract year, 2009 to 2010), IES took an average of over
50 business days to have intervention reports and quick reviews peer
reviewed, compared to an average of 7 business days during the first
quarter of second contract year (see figure 5).
Figure 5: Average Time for IES Peer Reviews of Released Intervention
Reports and Quick Reviews, by Contract Year (CY) and Quarter (Q) for
Current WWC Contract:
[Refer to PDF for image: 2 vertical bar charts]
IES peer review goal:
Intervention reports: 15;
Quick reviews: 6.
Quarter released: CY2-Q1;
Intervention reports: 7;
Quick reviews: 7.
Quarter released: CY2-Q2;
Intervention reports: 10;
Quick reviews: 6.
Quarter released: CY2-Q3;
Intervention reports: 13;
Quick reviews: 33.
Quarter released: CY2-Q4;
Intervention reports: 16;
Quick reviews: 45.
Quarter released: CY3-Q1;
Intervention reports: 54;
Quick reviews: 56.
Quarter released: CY3-Q2;
Intervention reports: 55;
Quick reviews: NA.
Quarter released: CY3-Q3;
Intervention reports: 54;
Quick reviews: 134.
Quarter released: CY3-Q4;
Intervention reports: 111;
Quick reviews: 99.
Source: GAO analysis of WWC data.
Notes: CY is July 1 to June 30. CY1 covers this period for 2007 to
2008, CY2 for 2008 to 2009, and CY3 for 2009 to 2010. In CY3-Q2, no
quick reviews were released.
[End of figure]
These delays in the IES-administered peer review process resulted in
significant backlogs of intervention reports and quick reviews
awaiting release. For example, as shown in figure 6, reports that
entered the peer review process in the first quarter of the second
contract year (CY2-Q1) were completed within that quarter. However,
the majority of reports entering review the first quarter of the third
contract year (CY3-Q1) remained in process for subsequent quarters.
While the backlog persisted through the third quarter of the third
contract year (CY3-Q3), the number of reports that completed peer
review in the third and fourth quarters increased from prior quarters.
Figure 6 shows that 11 intervention reports completed peer review in
CY3-Q3 and an additional 27 completed peer review in CY3-Q4, compared
with 4, 5, and 8 intervention reports in the prior three quarters.
Figure 6: IES Peer Review Backlog for Intervention Reports and Quick
Reviews, by Contract Year (CY) and Quarter (Q) for Current WWC
Contract:
[Refer to PDF for image: horizontal bar graph]
CY2-Q1:
Intervention reports entering peer review: 11;
Intervention reports completing peer review: 10;
Intervention reports remaining in peer review: 1;
Quick reviews entering peer review: 3;
Quick reviews completing peer review: 3;
Quick reviews remaining in peer review: 0.
CY2-Q2:
Intervention reports entering peer review: 10;
Intervention reports completing peer review: 10;
Intervention reports remaining in peer review: 1;
Quick reviews entering peer review: 7;
Quick reviews completing peer review: 5;
Quick reviews remaining in peer review: 2.
CY2-Q3:
Intervention reports entering peer review: 19;
Intervention reports completing peer review: 14;
Intervention reports remaining in peer review: 6;
Quick reviews entering peer review: 7;
Quick reviews completing peer review: 4;
Quick reviews remaining in peer review: 5.
CY2-Q4:
Intervention reports entering peer review: 11;
Intervention reports completing peer review: 8;
Intervention reports remaining in peer review: 9;
Quick reviews entering peer review: 11;
Quick reviews completing peer review: 7;
Quick reviews remaining in peer review: 9.
CY3-Q1:
Intervention reports entering peer review: 8;
Intervention reports completing peer review: 5;
Intervention reports remaining in peer review: 12;
Quick reviews entering peer review: 3;
Quick reviews completing peer review: 1;
Quick reviews remaining in peer review: 11.
CY3-Q2:
Intervention reports entering peer review: 6;
Intervention reports completing peer review: 4;
Intervention reports remaining in peer review: 14;
Quick reviews entering peer review: 4;
Quick reviews completing peer review: 3;
Quick reviews remaining in peer review: 12.
CY3-Q3:
Intervention reports entering peer review: 16;
Intervention reports completing peer review: 11;
Intervention reports remaining in peer review: 19;
Quick reviews entering peer review: 3;
Quick reviews completing peer review: 10;
Quick reviews remaining in peer review: 5.
CY3-Q4:
Intervention reports entering peer review: 10;
Intervention reports completing peer review: 27;
Intervention reports remaining in peer review: 2;
Quick reviews entering peer review: 7;
Quick reviews completing peer review: 11;
Quick reviews remaining in peer review: 1.
Source: GAO analysis of WWC data.
Notes: CY is July 1 to June 30. CY1 covers this period for 2007 to
2008, CY2 for 2008 to 2009, and CY3 for 2009 to 2010. At the end of
each quarter, any report remaining in peer review would carry over to
the next quarter. For example, for intervention reports in CY2-Q3, 19
new reports entered peer review, joining the 1 report that remained
from the previous quarter. Fourteen of these 20 reports completed peer
review, and 6 remained.
[End of figure]
IES attributed these delays to several factors and recently took steps
to eliminate the backlog. IES officials told us that delays were, in
part, attributable to difficulty in identifying and scheduling
independent peer reviewers, vacancies in WWC-related positions at IES,
and an increasing amount of research that met WWC standards.[Footnote
33] For example, IES officials told us identifying and scheduling a
sufficient number of qualified, independent peer reviewers had become
increasingly difficult because several former peer reviewers were now
associated in some way with the WWC and therefore were no longer
independent. To reduce the delays and eliminate the backlog, IES
recently implemented a new database to help staff track and manage the
work of peer reviewers and other WWC-related tasks. IES officials also
told us that they began identifying additional potential peer
reviewers using the WWC online registry of researchers. In addition,
IES increased a staff member's responsibilities related to scheduling
and coordinating peer reviews. These efforts reduced the amount of
time reports remain in the IES peer review process and eliminated the
backlog as of June 2010.[Footnote 34]
In addition to delays in the peer review process, WWC contractors told
us that many of their daily decisions need IES approval, and slow
responses delayed contractor processes. For example, the contractor
needs IES approval on the format and content of the products in
development, hindering further work when responses are delayed. IES
officials acknowledged that some delays in the approval process
occurred during contract year three and told us that this was largely
due to staff vacancies that they anticipate filling.
The Cost of the Current WWC Contract Has Increased from the Previous
One:
WWC's contracted costs have doubled from about $5.3 million per year
under the previous 5-year contract to the current level of about $10.7
million per year.[Footnote 35] The increase in contracted costs
reflects the expanded scope--more publications and new products and
processes--of the second contract compared to the first. IES's
contract for the WWC includes a variety of tasks that the contractor
is responsible for, including tasks related to report production and
product development. Table 4 provides a description of six broad task
categories and how they changed between contracts.
Table 4: Task Category Definitions and Changes between Contracts:
Task category: WWC products;
Task category includes expenditures related to: Conducting research
reviews and developing and publishing WWC products;
Changes from first contract to current contract: New product types,
expanded practice guide review process;
Stopped producing topic reports (2008).
Task category: Strategic planning and coordination with IES;
Task category includes expenditures related to: Preparing annual
plans, managing reporting requirements, and communications and
workflow with IES;
Changes from first contract to current contract: New contractor
database increased process documentation and reporting capabilities to
IES.
Task category: Communications, collaboration, and dissemination of WWC
products;
Task category includes expenditures related to: Maintaining WWC Help
Desk; Promoting WWC through various means; Developing/implementing a
communications/dissemination plan;
Changes from first contract to current contract: WWC staff attend
conferences and coordinate some dissemination efforts with other IES
departments in the current contract.
Task category: WWC development, process revisions, and maintenance;
Task category includes expenditures related to: Revising and
developing review processes and policies; Administrating and
supporting technical staff training, technical advisory group, online
registries, and conflict of interest procedures;
Changes from first contract to current contract: Enhanced review
processes and standards, added new research designs; Developed policy
and procedures handbooks, new products, and staff training.
Task category: Web site and technical maintenance;
Task category includes expenditures related to: Coordinating content,
maintaining databases/search functions, and processing federal data
collection forms;
Changes from first contract to current contract: New online searchable
system and database.
Task category: Award fees;
Task category includes expenditures related to: Fixed and performance-
based contractor award fees based on a percentage of the overall
contract total;
Changes from first contract to current contract: No change.
Source: GAO review of WWC contracts, annual plans, and budget
documents.
[End of table]
Our analyses of costs associated with these six broad task categories
shows that the proportion of funds dedicated to producing WWC reports
was about 60 percent under both contracts (see figure 7).[Footnote 36]
Figure 7: WWC Costs, by Task Categories and Contracts:
[Refer to PDF for image: 2 pie-charts]
First 5-year contract ($26.5 million):
WWC products: 60%:
- Intervention and topic reports: 57%;
- Practice guides: 3%;
Other costs: 41%:
- A: 6%;
- B: 12%;
- C: 5%;
- D: 9%;
- E: 9%.
Second 5-year contract ($53.3 million):
WWC products: 60%:
- Intervention and topic reports: 29%;
- Practice guides: 21%;
Other costs: 41%:
- A: 7%;
- B: 6%;
- C: 11%;
- D: 7%;
- E: 9%.
Breakdown of other costs:
A: Strategic planning and coordination with Education.
B: Communications, collaboration, and dissemination.
C: WWC development, process revisions, and staff training.
D: Web site and technical maintenance.
E: Award fees.
Source: GAO analysis of WWC budget data.
Notes: Figures reflect actual expenditures for the first 5-year
contract and updated budgeted expenditures for the current 5-year
contract, which began on July 1, 2007. Percentages do not add to 100%
due to rounding. Total WWC expenditures for the first 5-year contract
were about $26,527,760, but cost category percentages do not reflect
$1,222,714 billed by a co-contractor but not itemized by task.
Category proportions for the first contract are estimates because IES
could not provide documentation that included final adjusted
expenditures by tasks. The WWC budget is $53,315,166 for the current 5-
year contract, of which $23,643,891 had been spent as of October 31,
2009. Cost category proportions for the current contract do not
include $104,559 related to transition from the first contract to the
second contract.
[End of figure]
The proportion of funds dedicated to some tasks changed from the first
contract to the second. For example, costs for tasks related to
process development and revisions doubled from 5 percent to 11
percent, supporting various activities such as expanding the practice
guide review process and revising the Clearinghouse's procedures and
standards handbook. According to IES officials, the current WWC
contractor developed and implemented new or enhanced processes that
affect all publications and deliverables. For example, the current
contractor developed a standardized system for conducting and
recording the WWC's searches of research studies.[Footnote 37]
Most WWC cost increases supported additional output and expansions in
product scope. While under both contracts more resources were devoted
to intervention reports than any other product, the proportion devoted
to practice guides increased significantly, currently comprising about
21 percent of total budgeted costs. IES noted that practice guides
were only added during the last year of the prior contract, but are
now a primary product. Other new WWC products make up a relatively
small proportion of budgeted costs in the current contract,
representing about 9 percent of the total contract budget combined.
IES Has Not Developed Performance Measures Related to Production Costs
or Product Usefulness:
IES established performance goals, which the WWC met or exceeded;
however, these goals do not address production costs or the usefulness
of WWC products. IES established WWC-related performance goals in its
annual organizational assessment, but Education discontinued the use
of these performance measures for fiscal year 2010.[Footnote 38] In
addition, IES established performance goals for its WWC contractor in
the contractor award fee plan, which IES uses to determine the amount
of performance-based funds awarded to the contractor.[Footnote 39]
IES measured WWC program performance from fiscal year 2003 to fiscal
year 2009, as part of Education's Organizational Assessment--its
departmentwide performance management system. The WWC-related
performance goals included in Education's Organizational Assessment
focused on WWC Web site visits and the quantity of publications, both
of which were areas of concern as the WWC was getting established.
[Footnote 40] Specifically, these performance goals included increased
WWC Web site visits, sustained productivity in the release of
intervention reports and quick reviews, and increased practice guide
production. The WWC met or exceeded these performance goals annually;
however, according to IES officials, these performance goals will not
be included in Education's fiscal year 2010 Organizational Assessment,
in part because IES is now satisfied with WWC activity in these areas.
[Footnote 41]
IES has not developed performance measures related to the cost of
specific WWC products.[Footnote 42] IES officials noted that the costs
per WWC publication vary greatly depending on the amount of available
research relevant to the specifications of a product. For example,
intervention reports based on a large number of studies meeting WWC
standards take longer and cost more to produce than do reports for
which few studies qualify for review. IES has tasked the current WWC
contractor to develop ways to streamline production processes and to
conduct a cost study, the results of which would improve budget
estimates and strengthen IES's monitoring of the contract. While the
contractor has begun this work, IES officials told us that they do not
know when cost-related performance measures, such as acceptable cost
ranges for each type of product, will be established.
WWC does not currently have a performance measure related to product
usefulness. While Web site visits were tracked as a measure of WWC
utilization in IES's Organizational Assessment through fiscal year
2009, this metric did not assess the degree to which WWC products were
reaching their target audience and did not provided any information on
the extent to which educators find WWC products to be useful. IES's
2010 budget justification calls for a representative survey of WWC use
among education decision makers to be conducted by 2013. However, IES
officials told us that they were unsure whether the survey would take
place, and IES does not currently have a plan in place to implement
this survey.[Footnote 43]
Education Has Three Primary Ways to Disseminate Information about WWC
Products, but Awareness and Use Vary among Target Audiences:
Education Has Various Ways to Disseminate Information about WWC
Products, but Awareness of the Clearinghouse Is Generally Limited:
Education uses the WWC contractor, RELs, and DWW to disseminate
information about the Clearinghouse to its target audience, which
includes state and school district officials, as well as teachers and
principals. In accordance with its contract, the WWC contractor
disseminates information about its products electronically and through
various events, such as formal presentations at conferences. The
Clearinghouse's electronic dissemination methods include an e-mail
listserv, Web-based seminars (webinars), and newsletters. For example,
the WWC sends out notices to its e-mail listserv, alerting subscribers
of the availability of new products, including intervention reports,
practice guides, and quick reviews.[Footnote 44] WWC staff told us
that the webinars cover the same topics as their reports and are a
relatively cost-effective way to disseminate information about
products and methodology. In addition, WWC staff disseminate
information about WWC products at education conferences, such as
teacher, principal, and researcher conferences. At these conferences,
WWC staff may conduct formal presentations, have an exhibit featuring
their products, or both. At conference exhibits, Clearinghouse staff
answer questions about their products and provide literature to
conference attendees. From July 2009 through June 2010, WWC staff were
scheduled to present or have an exhibit at 14 conferences. WWC staff
also told us that they work with other groups, such as education,
research, and legislative organizations, in order to further
disseminate information about WWC products to their members.
In addition, Education disseminates information about WWC products
through IES's 10 RELs, which hold events that may feature information
based on practice guides and refer educators to Clearinghouse
products. Officials at all 10 RELs told us that they spent time
disseminating information about WWC, in part by holding events that
bridge research and practice. According to REL officials, these bridge
events are attended primarily by school-, district-, and state-level
education professionals and provide an opportunity for educators to
discuss ways to implement research-based practices. Officials at all
10 RELs told us that bridge events focused on practice guides to some
extent, and 7 indicated that WWC practice guides were the primary
focus of these events. According to REL officials and WWC staff, these
events sometimes included a WWC staff member to discuss methodology
and panelists who helped develop the practice guides. RELs also
disseminate research from WWC when responding to educator questions or
concerns.[Footnote 45] Officials from 7 of the 10 RELs told us their
respective RELs generally use relevant WWC products (practice guides
and others) when searching for research-based information to address
educator questions.
In addition, Education's Office of Planning, Evaluation, and Policy
Development disseminates information about WWC practice guides on its
DWW Web site, which provides an online library of resources designed
to help educators implement research-based instructional practice.
This Web site uses different formats to present content based
primarily on WWC Practice Guides and provides examples of possible
ways educators might apply WWC research findings. For instance, to
help educators implement the recommendations from the practice guide
on dropout prevention, the DWW Web site features slideshows with
examples of supportive academic environments and interviews with
educators and experts on dropout prevention. In addition, the Web site
includes sample materials, such as lesson plan templates, that provide
an example of how to implement recommendations. The DWW also includes
information on the research behind the recommendations and a link to
the WWC Web site and the individual practice guides. According to IES
officials, a recent analysis of the DWW Web site traffic showed that
49 state Web sites have links to the DWW Web site, which helps
disseminate WWC products further to the education community.
We found that 33 of the 38 states[Footnote 46] that responded to our
survey reported that they had heard of the WWC. Based on our survey
results, we estimate that 42 percent of school districts have heard of
the WWC and that the percentage is greater for school districts that
rely to a very large extent on external sources for information on
research-based practices.[Footnote 47]
School districts identified several sources of information about the
Clearinghouse, including conferences and Education (see figure 8).
Figure 8: Sources from Which District Officials Heard of the WWC:
[Refer to PDF for image: vertical bar graph]
Potential sources of WWC information: Education;
Not a source of WWC information: 23;
Don't know: 5;
Source of WWC information: 72.
Potential sources of WWC information: Conferences;
Not a source of WWC information: 22;
Don't know: 5;
Source of WWC information: 73.
Potential sources of WWC information: Peers;
Not a source of WWC information: 28;
Don't know: 6;
Source of WWC information: 66.
Potential sources of WWC information: RELs;
Not a source of WWC information: 45;
Don't know: 8;
Source of WWC information: 47.
Potential sources of WWC information: Other;
Not a source of WWC information: 24;
Don't know: 40;
Source of WWC information: 37.
Source: GAO analysis of school district survey responses.
Notes: Estimates shown are based upon a probability survey. See
appendix I for associated confidence intervals. The responses for
other sources included state departments of education, Internet
searches, and journals.
[End of figure]
While the majority of states have accessed the WWC Web site, we
estimate that only 34 percent of school districts have done so.
Specifically, among the states that responded to our survey, 33 of 38
states[Footnote 48] reported that they had accessed the WWC Web site
at least once. In addition, 19 of states reported visiting the Web
site at least seven times per year.[Footnote 49] In contrast, an
estimated 34 percent of school districts accessed the WWC Web site at
least once.[Footnote 50] Further, we estimate that only 11 percent of
school districts visited the Web site at least seven times per year.
[Footnote 51] States and school districts that visited the WWC Web
site less than seven times per year most often cited time constraints
as the primary reason for their infrequent use.[Footnote 52]
In addition to the WWC, states and school districts use a variety of
other sources of information to identify effective education
practices. Most states and school districts use several broad sources
of information, such as academic journals, education periodicals, and
associations of educators. For example, 37 states reported using
academic journals to identify such practices, and we estimate that
about 97 percent of school districts used academic journals.[Footnote
53] Overall, more school districts and states that responded to our
survey used the WWC than used other research synthesis organizations.
[Footnote 54]
While the WWC also includes teachers and principals in its target
audience, we found that relatively few of the teachers and principals
we contacted at education conferences had heard of the WWC. While not
a generalizeable sample, we found that out of a total of 391 teachers
who completed our questionnaire at four education conferences, only 18
had accessed the WWC Web site.[Footnote 55] In addition, 341 teachers
who had not accessed the WWC Web site told us they had not heard of
the Web site. Similarly, among the 208 principals and other school
administrators who completed the questionnaire, only 32 had accessed
the WWC Web site. Further, 135 principals and other school
administrators told us they had not heard of the WWC.
States and School Districts Generally Used the Clearinghouse to a
Small or Moderate Extent to Inform Decisions and Used Specific WWC
Products to Varying Extents:
Based on our survey, most states and school districts that reported
accessing the WWC Web site used it to inform decisions on effective
education practices--a stated purpose of the WWC--to a small or
moderate extent. Specifically, 25 of the 33 states that use the
Clearinghouse indicated that they use it to a small or moderate extent
to inform their decisions, while 6 reported using it to a large or
very large extent.[Footnote 56] We estimate that 72 percent of school
districts that have accessed the Clearinghouse use the WWC to inform
education decisions to a small or moderate extent, while only 18
percent use it to a large or very large extent.[Footnote 57]
States that used the WWC to inform decisions reported that they used
the Clearinghouse for various purposes, including informing
professional development and curriculum decisions. For example, 25
states reported using the Clearinghouse to inform professional
development programs for teachers, and 22 reported using it to inform
curriculum decisions. Fewer states used the Clearinghouse to advise
districts that were not making adequate yearly progress (AYP) in
meeting academic targets or to develop improvement plans for such
districts. (Figure 9 provides a breakdown of the extent to which these
states use the Clearinghouse for various purposes.)
Figure 9: Extent to Which States Use WWC for Various Purposes:
[Refer to PDF for image: vertical bar graph]
Number of states:
Inform curriculum decisions:
Not at all: 4;
Small extent: 2;
Moderate extent: 12;
Large or very large extent: 8.
Inform professional development of teachers:
Not at all: 2;
Small extent: 4;
Moderate extent: 12;
Large or very large extent: 9.
Develop school improvement plan:
Not at all: 4;
Small extent: 6;
Moderate extent: 9;
Large or very large extent: 4.
Advise schools not making AVP on potential interventions:
Not at all: 4;
Small extent: 4;
Moderate extent: 11;
Large or very large extent: 5.
Source: GAO analysis of state responses.
Note: "I Don't Know" was also a response option, and is not displayed
in the figure.
[End of figure]
In addition, we estimate that among school districts that use the WWC
to inform decisions on effective education practices, about 90 percent
used it to inform curriculum decisions at least to a small extent,
similar to the percentage that used the WWC to inform professional
development decisions. However, fewer school districts used it to
advise schools that did not meet academic goals or to develop school-
level plans to help such schools improve.[Footnote 58] Figure 10
provides a breakdown of the extent to which these school districts use
the Clearinghouse for various purposes.
Figure 10: Extent to Which School Districts That Have Used the
Clearinghouse Used It for Various Purposes:
[Refer to PDF for image: vertical bar graph]
Estimated percentage of school districts:
Inform curriculum decisions:
Not at all: 3%;
Small extent: 20%;
Moderate extent: 37%;
Large or very large extent: 37%.
Inform professional development of teachers:
Not at all: 6%;
Small extent: 19%;
Moderate extent: 38%;
Large or very large extent: 32%.
Develop school improvement plan:
Not at all: 13%;
Small extent: 19%;
Moderate extent: 35%;
Large or very large extent: 27%.
Advise schools not making AVP on potential interventions:
Not at all: 29%;
Small extent: 11%;
Moderate extent: 28%;
Large or very large extent: 23%.
Source: GAO analysis of school district survey responses.
Note: Estimates shown are based upon a probability survey. See
appendix I for associated confidence intervals. "I Don't Know" was
also a response option, and is not displayed in the figure.
[End of figure]
States reported using specific WWC products--intervention reports and
practice guides--more than quick reviews. Specifically, of the states
that had used the Clearinghouse, 21 reported that they used
intervention reports and 20 reported using practice guides, while only
12 reported using quick reviews. States used intervention reports and
practice guides to a similar extent to inform education decisions. For
example, for each product, six states reported using them to large or
very large extent to inform such decisions (see figure 11).
Figure 11: Extent to Which States Use Specific WWC Products:
[Refer to PDF for image: vertical bar graph]
Number of states:
Intervention reports:
Not at all: 4;
Small extent: 3;
Moderate extent: 12;
Large or very large extent: 6.
Practice guides:
Not at all: 3;
Small extent: 3;
Moderate extent: 11;
Large or very large extent: 6.
Quick reviews:
Not at all: 10;
Small extent: 3;
Moderate extent: 7;
Large or very large extent: 2.
Source: GAO analysis of state survey responses.
[End of figure]
However, the relative use of specific WWC products was different among
school districts. We estimate that among school districts that use the
Clearinghouse to inform decisions on effective education practices,
more school districts use intervention reports relative to practice
guides or quick reviews. Specifically, we estimate that 74 percent of
those school districts that use the WWC have used its intervention
reports to inform education decisions,[Footnote 59] while practice
guides and quick reviews were each used by about half of such
districts.[Footnote 60] Based on our survey, an estimated 21 percent
of school districts that use the WWC have used intervention reports to
a large or very large extent,[Footnote 61] while about 10 percent use
practice guides to a large or very large extent (see figure 12).
[Footnote 62]
[Refer to PDF for image]
[End of figure]
Figure 12: Extent of Specific Product Use among Districts That Use the
Clearinghouse:
[Refer to PDF for image: vertical bar graph]
Intervention reports:
Not at all: 21;
Small extent: 24;
Moderate extent: 29;
Large or very large extent: 21.
Practice guides:
Not at all: 42;
Small extent: 17;
Moderate extent: 28;
Large or very large extent: 10.
Quick reviews:
Not at all: 40;
Small extent: 25;
Moderate extent: 17;
Large or very large extent: 13.
Source: GAO analysis of school district survey responses.
Note: Estimates shown are based upon a probability survey. See
appendix I for associated confidence intervals.
[End of figure]
States and School Districts Would Likely Increase Their Use of the
Clearinghouse If the WWC Made Certain Changes:
Many states and school districts that had accessed the Clearinghouse
reported that they would likely increase their use of the WWC if the
Clearinghouse provided a broader array of information. For example,
many states and school districts would be likely to increase their use
of the Clearinghouse if it reviewed more studies, covered additional
topics, or provided more relevant or timely reports. For example, 21
of the 33 states that had used the Clearinghouse reported that they
would be somewhat or very likely to use the Clearinghouse more often
if it had reviews that were more timely (see figure 13).[Footnote 63]
Figure 13: Number of States That Reported They Would Likely Increase
Their Use of WWC Given Certain Changes:
[Refer to PDF for image: horizontal bar graph]
Potential changes:
A greater number of intervention reports showing positive effects;
Likely or very likely: 26.
Reviews of efficacy of programs being used or considered in my state;
Likely or very likely: 23.
Additional practices with positives reviews;
Likely or very likely: 22.
Reviews that are more timely;
Likely or very likely: 21.
Additional topic areas;
Likely or very likely: 19.
Additional studies reviewed;
Likely or very likely: 19.
Additional practice guides;
Likely or very likely: 18.
Additional information on interventions based on studies that may not
meet WWC standards;
Likely or very likely: 15.
A broader definition of what studies meet WWC standards;
Likely or very likely: 15.
An easier web site to navigate;
Likely or very likely: 14.
Source: GAO analysis of state survey responses.
In addition, based on our survey, we estimate that about two thirds of
school districts that had accessed the Clearinghouse would likely
increase their use if it included reviews of programs or interventions
being used or considered in their school district.[Footnote 64] An
estimated 50 percent of school districts would likely increase their
use of the Clearinghouse if it had reviews that were more timely (see
figure 14).[Footnote 65]
Figure 14: Estimated Percent of School Districts That Have Accessed
the WWC That Would Likely Increase Their Use of the WWC Given Various
Changes:
[Refer to PDF for image: horizontal bar graph]
Potential changes:
Reviews of efficacy of programs being used or considered in my
district: 68%.
A greater number of intervention reports showing positive effects:
Likely or very likely: 67%.
Additional practices with positives reviews:
Likely or very likely: 60%.
Additional topic areas:
Likely or very likely: 57%.
Additional information on interventions based on studies that may not
meet WWC standards:
Likely or very likely: 56%.
Additional practice guides:
Likely or very likely: 56%.
Additional studies reviewed:
Likely or very likely: 55%.
Reviews that are more timely:
Likely or very likely: 50%.
A broader definition of what studies meet WWC standards:
Likely or very likely: 47%.
An easier web site to navigate:
Likely or very likely: 47%.
Source: GAO analysis of school district survey responses.
Note: Estimates shown are based upon a probability survey. See
appendix I for associated confidence intervals.
[End of figure]
Conclusions:
In 2007, Education substantially increased its financial investment in
the WWC, and the Clearinghouse is significantly expanding its scope in
an effort to better serve its target audience. Some of the new
products aim to be more responsive to educators and education decision
makers by providing timely information about evidence-based practices
relevant to pressing needs. Such information could help states and
districts identify strategies as they implement educational reform
efforts--such as reforming low-performing schools or improving
professional development--under ESEA and the Recovery Act. For
example, WWC research perspectives, still in development, are intended
to help education decision makers as they address challenges related
to spending Recovery Act funds. However, the development of these
products and the release of other products were delayed, in part, by a
substantial backlog in IES's review and approval processes. These
delays hindered the timely release of several publications, and some
products were released months after they were completed by the
contractor. While IES recently eliminated the backlog, educators need
to be able to rely on the Clearinghouse for timely and relevant
information. According to our survey, many states and school districts
reported that they would likely increase their use of the
Clearinghouse if it released information more quickly.
While IES has increased annual report production, IES has not
established reasonable production cost ranges or specific cost-related
performance measures related to each product type. Without acceptable
per product cost ranges, it is difficult for IES to assess the
reasonableness of costs associated with certain products, even as IES
takes steps to streamline production. IES's current study on costs may
help IES establish acceptable cost ranges that could inform IES's
performance measurements related to the WWC. In addition, such
information could inform cost comparisons between the WWC and other
research evaluation organizations or provide baselines for future
contractor work.
In addition, IES has not established meaningful performance measures
related to product usefulness. Until fiscal year 2010, IES tracked
visits to its Web site and annual report production as a way to
measure the productivity of the Clearinghouse. While these measures
were important to accurately track the WWC's initial growth, they did
not evaluate the degree to which the products were meeting the needs
of educators. Specifically, IES currently does not have a way to gauge
user satisfaction with WWC products, which is a common practice when
developing and providing new products. Further, while IES currently
incorporates some feedback from the WWC Web site users, to inform
future topic areas, it does not systematically gauge its target
audience's major areas of interest or concern--such as gathering
information on interventions currently being used or considered in
specific school districts or states. IES decides how to spread its
limited resources across the various product types without directly
measuring the extent to which educators use the WWC or how useful they
find the various products to be. Measuring the use and usefulness of
its products could help IES continue to improve content, develop
products, and respond to the needs of educators and policymakers.
While some educators and policymakers find WWC products useful, many
other educators are not familiar with the Clearinghouse. IES has spent
a substantial amount of money, time, and effort producing various
summaries of evidence-based practices, which cover both specific
education interventions and general practices. This investment in the
WWC was made in order to inform education professionals at all levels--
from classroom teachers to policymakers--as they make decisions on how
best to educate the nation's children. Improved dissemination of WWC
products could increase awareness and use of the WWC. Increased use of
the Clearinghouse could help education professionals identify and
implement effective educational interventions and practices, and
potentially lead to increased student achievement.
Recommendations for Executive Action:
We are making the following four recommendations based on our review.
To consistently release WWC products in a timely manner, we recommend
the Secretary of Education direct IES to develop and implement
strategies that help avoid future backlogs and ensure that IES's
review and approval processes keep pace with increased contractor
production. Strategies could include shifting IES resources to ensure
sufficient staff time for managing the peer review process and
streamlining its approval processes.
To better track the costs and usefulness of the WWC, we recommend that
the Secretary of Education direct IES to:
* incorporate findings from its cost studies to develop performance
measures related to costs, such as identifying a range of acceptable
costs per product and using that information to monitor contractor
spending; and:
* develop performance measures related to product usefulness and
periodically assess whether WWC products are meeting the needs of
target audiences by gathering information on product usefulness in the
proposed survey or through other means.
To reach more members of the target audience, we recommend the
Secretary of Education direct IES to assess and improve its
dissemination efforts to promote greater awareness and use of the WWC,
for example, by developing a way to inform school districts of new
products or encouraging educator professional development programs to
focus on research-based practices such as those discussed in practice
guides.
Agency Comments and Our Evaluation:
We provided a draft of this report to the U.S. Department of Education
for review and comment. Education officials provided written comments
on a draft of this report, which are reproduced in appendix IV.
Education also provided technical comments, which we incorporated into
the report as appropriate.
Education generally agreed with our recommendations. Specifically,
Education agreed to our recommendations on consistently releasing WWC
products in a timely manner and assessing and improving its
dissemination efforts. In its response to our recommendation on
tracking the cost and usefulness of the WWC and its products,
Education noted that IES has taken some steps that address the
recommendation. With regard to costs, Education stated that it intends
to incorporate the results of current cost studies into future work
plans and monitoring efforts. We continue to recommend that these
results be used to inform performance measures related to costs for
future operations. With regard to tracking the usefulness of the WWC,
Education noted that it uses a variety of tools to gather consumer
input, such as a Help Desk and online voting for future report topics.
While such feedback provides some information to the WWC, it relies on
existing users and reflects the views of those users who provide
feedback, rather than those of the broader population. However, as
shown in our survey, only an estimated 34 percent of school districts
have accessed the WWC Web site at least once--and fewer have used the
Web site frequently. Education also noted that it would include a
customer satisfaction survey in IES's review of its own performance,
but whether the survey would be directed at current Clearinghouse
customers or a broader audience, or whether the survey would identify
how useful various WWC products are and how the WWC can be improved is
unclear. More nationally representative information could help IES
prioritize topics for intervention reports and practice guides and
inform budget priorities.
We are sending copies of this report to the appropriate congressional
committees, the Secretary of the U.S. Department of Education, and
other interested parties. In addition, the report will be available at
no charge on the GAO Web site at [hyperlink, http://www.gao.gov].
If you or your staffs have any questions regarding this report, please
contact me at (202) 512-7215 or ashbyc@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. GAO staff who made major contributors to
this report are listed in appendix V.
Signed by:
Cornelia M. Ashby:
Director, Education, Workforce, and Income Security Issues:
List of committees:
The Honorable Tom Harkin:
Chairman:
The Honorable Thad Cochran:
Ranking Member:
Subcommittee on Labor, Health and:
Human Services, Education and Related Agencies:
Committee on Appropriations:
United States Senate:
The Honorable David Obey:
Chairman:
The Honorable Todd Tiahrt:
Ranking Member:
Subcommittee on Labor, Health and:
Human Services, Education and Related Agencies:
Committee on Appropriations:
House of Representatives:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
To address all three objectives, we interviewed officials from the
Institute of Education Sciences (IES), What Works Clearinghouse (WWC
or Clearinghouse) contractors, and representatives from various
educational organizations. To assess the research review process used
by the IES's WWC, we reviewed WWC standards and procedures, reviewed
an expert panel report that assessed the validity of the WWC review
process, and collected information about the extent to which the WWC
has implemented the panel's recommendations. To determine how
performance and costs changed over time, we analyzed the costs and
productivity of the two WWC contractors. To obtain information about
the usefulness of WWC products, we conducted a Web-based survey of all
state education agencies and a nationally representative sample of
school districts. We also collected information about the usefulness
of the WWC from teachers and principals at four education conferences.
We conducted our work from September 2009 through July 2010 in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.
Assessment of WWC Research Review Process:
GAO previously assessed the procedures and criteria used by the WWC by
reviewing documents and interviewing IES officials and WWC
contractors.[Footnote 66] We reviewed WWC standards and procedures and
examined the degree of consistency of these standards and procedures
across education topic areas. We also reviewed the findings and
recommendations from an expert panel report that assessed the validity
of the WWC review process.[Footnote 67] We obtained information from
IES officials and WWC contractors on the extent to which the WWC has
implemented the panel's recommendations. Further, we identified other
concerns about the WWC review process through a literature review and
interviews with researchers, and we interviewed IES officials and WWC
contractors to assess the extent to which the Clearinghouse has
addressed these concerns. We also examined the degree to which the
WWC's review process is similar to that used by other entities engaged
in systematic research review efforts.
Performance and Cost Data Analyses:
To determine how performance and costs changed over time, we analyzed
the costs and productivity of the two WWC contractors. We reviewed
budget data and product release dates to analyze cost and productivity
trends of the WWC. To examine performance, we interviewed the two
contractors, as well as IES officials, and compared IES's performance
measures and goals to actual outcomes.
We assessed the reliability of the WWC performance and cost data by
(1) reviewing existing information about the data and the system that
produced them and (2) interviewing agency officials knowledgeable
about the data. We determined that the data were sufficiently reliable
for the purposes of this report.
Survey of States and School Districts:
To determine how WWC products are disseminated, we interviewed
officials from IES and all 10 RELs, as well as WWC contractors. To
determine how useful education professionals find WWC products to be,
we designed and administered a Web-based survey of state education
agencies in the 50 states and the District of Columbia and a
nationally representative sample of local educational agencies (LEA).
Specifically, the survey asked officials about (1) their general
sources of information on effective educational practices, (2) the
extent to which they use WWC products to inform curriculum decisions
(including questions on specific intervention reports and practice
guides), (3) how useful the officials find the information in the WWC,
(4) the likelihood they would increase their usage if certain changes
were made to the WWC Web site, and (5) the extent to which the
officials use the Doing What Works and Best Evidence Encyclopedia Web
sites to inform curriculum decisions and how useful the officials find
these other information sources to be. We reproduce the questions we
used in our analysis in figure 15. The survey was administered from
February 18, 2010 to April 14, 2010.
To determine how the WWC was being used at the state level, we
surveyed the state Secretary, Commissioner, or Superintendent of
Education in the 50 states and the District of Columbia. Out of the 51
state officials surveyed, 38 responded to the survey.
To determine how the WWC was being used at the school district level,
we surveyed a nationally representative sample of school districts
across the country. We selected a stratified random sample of 625 LEAs
from the population of 17,620 LEAs included in our sample frame of
data obtained from the Common Core of Data for the 2007-08 school
year. A total of 454 LEAs responded, resulting in a final response
rate of 74 percent. Because we surveyed a sample of LEAs, survey
results for the district are estimates of a population of LEAs and
thus are subject to sampling errors that are associated with samples
of this size and type. Our sample is only one of a large number of
samples that we might have drawn. As each sample could have provided
different estimates, we express our confidence in the precision of our
particular sample's results as a 95 percent confidence interval (e.g.,
plus or minus 10 percentage points). We excluded 12 of the sampled
LEAs for various reasons--6 were closed, 3 did not administer any
schools, 2 managed schools in a correctional facility, and 1 was a
private school--and therefore were considered out of scope. All
estimates produced from the sample and presented in this report are
representative of the in-scope population.
The practical difficulties of conducting any survey may introduce
nonsampling errors, such as difficulties interpreting a particular
question, which can introduce unwanted variability into the survey
results. We took steps to minimize nonsampling errors by pretesting
the questionnaire over the phone with officials from two school
districts and one state department of education in November and
December 2009. We conducted pretests to verify that (1) the questions
were clear and unambiguous, (2) terminology was used correctly, (3)
the questionnaire did not place an undue burden on officials, and (4)
the questionnaire was comprehensive and unbiased. An independent
reviewer within GAO also reviewed a draft of the questionnaire prior
to its administration. We made revisions to the questionnaire based on
feedback from the pretests and independent review before administering
the survey.
The survey-related data used in this report is based on the state and
school district responses to the survey questions.
Figure 15: GAO's Web-based Survey of State Departments of Education
and Local Educational Agencies in the 50 States and the District of
Columbia:
[Refer to PDF for image: survey]
3. To what extent, if at all, does your rely on external evidence-
based research to inform curriculum decisions?
(Check only one answer)
1. Very large extent:
2. Large extent:
3. Moderate extent:
4. Small extent:
5. Does not use external evidence-based research:
6. Don't know:
4. How useful, if at all, are each of the following research evaluation
resources to you or your staff in identifying effective practices to
implement in your______? (Please choose one response for each
resource.)
4a. Best Evidence Encyclopedia (Johns Hopkins University):
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
4b. Child Trends:
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
4c. Coalition for Evidence Based Policy:
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
4d. Doing What Works:
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
4e. RAND's Promising Practices:
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
4f. What Works Clearinghouse:
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
4g. Other research synthesis clearinghouses:
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
5. Recognizing that research evaluation resources are not necessarily
the primary sources of information used to identify effective
education practices, GAO is also interested in the role of other
information sources.
How useful, if at all, are each of the following sources to you or your
staff in identifying effective practices to implement in your ______?
(Please choose one response for each resource.)
5a. Academic journals:
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
5b. Education-related periodicals:
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
5c. Online databases (ERIC or others):
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
5d. University-based research institutions:
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
5e. Non-profit organizations:
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
5f. Associations of educators or researchers:
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
5g. Peer conferences:
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
5h. Regional Education Laboratories (Department of Ed):
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
5i. Other federal outreach center:
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
5j. State government offices and/or outreach centers:
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
5k. Local data and/or internal research:
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
51. Community and parent input:
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
5m. Mentors/Colleagues:
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
5n. Personal experience:
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
5o. Other resource (Please specify below):
Very useful:
Useful:
Somewhat useful:
Slightly useful:
Not at all useful:
No opinion:
Have not used this source of information:
Other resource:
7. Have you or your staff heard of the What Works Clearinghouse (WWC)?
(Check only one answer)
1. Yes:
2. No (Go to Section 3: Use of Doing What Works):
3. Don't know (Go to Section 3: Use of Doing What Works):
8. From which of the following source(s) did you or your staff hear
about the WWC?
(Please choose one response for each source.)
8a. Conferences:
Yes:
No:
Don't know:
8b. Peers:
Yes:
No:
Don't know:
8c. Regional Education Labs:
Yes:
No:
Don't know:
8d. U.S. Department of Education:
Yes:
No:
Don't know:
8e. Other source (Please specify below):
Yes:
No:
Don't know:
Other source:
9. How frequently, if at all, do you or your staff access the WWC
website? (Check only one answer)
1. Never (Go to question 28):
2. Less than twice a year (Go to question 10):
3. Between 2 and 6 times per year (Go to question 10):
4. Between 7 and 11 times per year (Go to question 11):
5. Monthly (Go to question 11):
6. More than once a month (Go to question 11):
10. You indicated that you or your staff access the WWC website less
than 7 times per year. Which of the following reasons best describes
why you and your staff do not access the website more frequently?
(Check only one answer)
1. Time Constraints:
2. Content is not relevant to our decisions:
3. Disagree with recommendations on the site:
4. Site is difficult to navigate:
5. Other reason (Please specify below) Other reason:
11. To what extent, if at all, have you or your staff used the WWC
website to inform decisions on effective education practices?
(Check only one answer)
1. Very large extent (Go to question 13):
2. Large extent (Go to question 13):
3. Moderate extent (Go to question 13):
4. Small extent (Go to question 13):
5. Have not used the VVWC to inform any decisions (Go to question 12):
6. Don't know (Go to question 12):
13. To what extent, if at all, have you or your staff used information
in the WWC to do any of the following? (Please choose one response for
each action.)
13a. Inform professional development of teachers:
To a very large extent:
To a large extent:
To a moderate extent:
To a small extent:
Not at all:
Don't know:
13b. Advise ______s that are not making AYP on potential interventions:
To a very large extent:
To a large extent:
To a moderate extent:
To a small extent:
Not at all:
Don't know:
13c. Develop ______improvement plans:
To a very large extent:
To a large extent:
To a moderate extent:
To a small extent:
Not at all:
Don't know:
13d. Inform curriculum decisions:
To a very large extent:
To a large extent:
To a moderate extent:
To a small extent:
Not at all:
Don't know:
13e. Other use (Please specify below):
To a very large extent:
To a large extent:
To a moderate extent:
To a small extent:
Not at all:
Don't know:
Other use:
14. To what extent have you or your staff used the WWC's Intervention
Reports to inform decisions on effective education practices?
Intervention Reports provide an assessment of the efficacy of
interventions based on existing research that meets certain standards.
(Check only one answer)
1. Very large extent:
2. Large extent:
3. Moderate extent:
4. Small extent:
5. Don't know:
6. Have not used the VVWC Intervention Reports to inform any decisions
(Go to question 18):
18. To what extent have you or your staff used the WWC's Practice
Guides to inform decisions on effective education practices?
Practice Guides are developed by a panel of experts and provide
recommendations to help educators address common classroom or school-
wide challenges. (Check only one answer)
1. Very large extent:
2. Large extent:
3. Moderate extent:
4. Small extent:
5. Don't know:
6. Have not used the WWC Practice Guides to inform any decisions (Go
to question 22):
22. To what extent have you or your staff used the WWC's Quick Reviews
to inform decisions on effective education practices?
Quick Reviews are designed to help educators and policy makers assess
the quality of recently released research papers and reports.
(Check only one answer)
1. Very large extent:
2. Large extent:
3. Moderate extent:
4. Small extent:
5. Don't know:
6. Have not used the VVWC Quick Reviews to inform any decisions (Go to
question 25):
25. How likely or unlikely would you or your staff be to increase your
usage of the WWC if any of the following information were added to the
website? (Please choose one response for each type of information.)
25a. Additional topic areas:
Very Likely:
Somewhat Likely:
Neither likely nor unlikely:
Somewhat unlikely:
Very unlikely:
No opinion:
25b. Additional practice guides:
Very Likely:
Somewhat Likely:
Neither likely nor unlikely:
Somewhat unlikely:
Very unlikely:
No opinion:
25c. Additional studies reviewed:
Very Likely:
Somewhat Likely:
Neither likely nor unlikely:
Somewhat unlikely:
Very unlikely:
No opinion:
25d. Additional information on interventions based on studies that may
not meet WWC standards:
Very Likely:
Somewhat Likely:
Neither likely nor unlikely:
Somewhat unlikely:
Very unlikely:
No opinion:
25e. A broader definition of what studies meet WWC standards:
Very Likely:
Somewhat Likely:
Neither likely nor unlikely:
Somewhat unlikely:
Very unlikely:
No opinion:
25f. Additional practices with positive reviews:
Very Likely:
Somewhat Likely:
Neither likely nor unlikely:
Somewhat unlikely:
Very unlikely:
No opinion:
25g. Reviews of efficacy of programs being used or considered in
my______:
Very Likely:
Somewhat Likely:
Neither likely nor unlikely:
Somewhat unlikely:
Very unlikely:
No opinion:
25h. Reviews that are more timely:
Very Likely:
Somewhat Likely:
Neither likely nor unlikely:
Somewhat unlikely:
Very unlikely:
No opinion:
25i. A greater number of intervention reports showing positive effects:
Very Likely:
Somewhat Likely:
Neither likely nor unlikely:
Somewhat unlikely:
Very unlikely:
No opinion:
25j. An easier website to navigate:
Very Likely:
Somewhat Likely:
Neither likely nor unlikely:
Somewhat unlikely:
Very unlikely:
No opinion:
25k. Other information (Please specify below):
Very Likely:
Somewhat Likely:
Neither likely nor unlikely:
Somewhat unlikely:
Very unlikely:
No opinion:
Other information:
Source: GAO survey of states' and school districts' use of educational
clearinghouses.
[End of figure]
The following tables contain the estimates and associated confidence
intervals for the data displayed in figures 8, 10, 12, and 14.
Table 5: Estimates and Confidence Intervals for Figure 8:
Q8. From which of the following source(s) did you or your staff hear
about the WWC?
Label: Conferences;
Response: Yes;
Percentage: 72.72;
Lower bound: 65.26;
Upper bound: 80.19.
Label: Conferences;
Response: No;
Percentage: 21.83;
Lower bound: 15.17;
Upper bound: 29.77.
Label: Conferences;
Response: Don't know;
Percentage: 5.45;
Lower bound: 2.36;
Upper bound: 10.50.
Label: Peers;
Response: Yes;
Percentage: 66.43;
Lower bound: 58.39;
Upper bound: 74.47.
Label: Peers;
Response: No;
Percentage: 27.80;
Lower bound: 20.22;
Upper bound: 35.37.
Label: Peers;
Response: Don't know;
Percentage: 5.77;
Lower bound: 2.32;
Upper bound: 11.62.
Label: Regional Education Labs;
Response: Yes;
Percentage: 47.14;
Lower bound: 38.40;
Upper bound: 55.89.
Label: Regional Education Labs;
Response: No;
Percentage: 45.24;
Lower bound: 36.51;
Upper bound: 53.97.
Label: Regional Education Labs;
Response: Don't know;
Percentage: 7.62;
Lower bound: 3.81;
Upper bound: 13.34.
Label: U.S. Department of Education;
Response: Yes;
Percentage: 71.93;
Lower bound: 64.41;
Upper bound: 79.46.
Label: U.S. Department of Education;
Response: No;
Percentage: 22.85;
Lower bound: 16.17;
Upper bound: 30.72.
Label: U.S. Department of Education;
Response: Don't know;
Percentage: 5.22;
Lower bound: 2.00;
Upper bound: 10.80.
Source: GAO analysis.
[End of table]
Table 6: Estimates and Confidence Intervals for Figure 10:
Q13: To what extent, if at all, have you or your staff used
information in the WWC to do any of the following?
Label: Inform professional development of teachers:
Response: To a very large extent;
Percentage: 12.33;
Lower bound: 6.24;
Upper bound: 21.15.
Response: To a large extent;
Percentage: 19.49;
Lower bound: 11.74;
Upper bound: 29.45.
Response: To a moderate extent;
Percentage: 38.12;
Lower bound: 28.25;
Upper bound: 48.00.
Response: To a small extent;
Percentage: 19.48;
Lower bound: 11.77;
Upper bound: 29.35.
Response: Not at all;
Percentage: 5.90;
Lower bound: 2.40;
Upper bound: 11.80.
Response: Don't know;
Percentage: 4.67;
Lower bound: 1.09;
Upper bound: 12.44.
Label: Advise schools that are not making adequate yearly progress on
potential interventions:
Response: To a very large extent;
Percentage: 13.23;
Lower bound: 6.90;
Upper bound: 22.23.
Response: To a large extent;
Percentage: 10.12;
Lower bound: 5.12;
Upper bound: 17.48.
Response: To a moderate extent;
Percentage: 28.21;
Lower bound: 18.82;
Upper bound: 39.24.
Response: To a small extent;
Percentage: 11.42;
Lower bound: 5.69;
Upper bound: 19.82.
Response: Not at all;
Percentage: 28.65;
Lower bound: 19.55;
Upper bound: 39.23.
Response: Don't know;
Percentage: 8.36;
Lower bound: 3.54;
Upper bound: 16.16.
Label: Develop school improvement plans:
Response: To a very large extent;
Percentage: 12.07;
Lower bound: 6.19;
Upper bound: 20.56.
Response: To a large extent;
Percentage: 14.79;
Lower bound: 7.59;
Upper bound: 24.99.
Response: To a moderate extent;
Percentage: 34.83;
Lower bound: 24.68;
Upper bound: 44.98.
Response: To a small extent;
Percentage: 19.15;
Lower bound: 11.72;
Upper bound: 28.63.
Response: Not at all;
Percentage: 13.37;
Lower bound: 7.35;
Upper bound: 21.73.
Response: Don't know;
Percentage: 5.79;
Lower bound: 1.80;
Upper bound: 13.36.
Label: Inform curriculum decisions:
Response: To a very large extent;
Percentage: 14.95;
Lower bound: 8.53;
Upper bound: 23.61.
Response: To a large extent;
Percentage: 21.99;
Lower bound: 14.15;
Upper bound: 31.65.
Response: To a moderate extent;
Percentage: 36.50;
Lower bound: 26.71;
Upper bound: 46.28.
Response: To a small extent;
Percentage: 19.50;
Lower bound: 12.02;
Upper bound: 29.00.
Response: Not at all;
Percentage: 3.31;
Lower bound: 0.92;
Upper bound: 8.23.
Response: Don't know;
Percentage: 3.75;
Lower bound: 0.63;
Upper bound: 11.47.
Source: GAO analysis.
[End of table]
Table 7: Estimates and Confidence Intervals for Figure 12:
Q14: To what extent have you or your staff used the WWCs intervention
reports to inform decisions on effective education practices?
Q18: To what extent have you or your staff used the WWCs practice
guides to inform decisions on effective education practices?
Q22: To what extent have you or your staff used the WWCs quick reviews
to inform decisions on effective education practices?
Label: Intervention reports:
Response: Very large or large extent;
Percentage: 20.98;
Lower bound: 13.66;
Upper bound: 29.99.
Response: Moderate extent;
Percentage: 29.42;
Lower bound: 20.13;
Upper bound: 38.71.
Response: Small extent;
Percentage: 23.99;
Lower bound: 15.26;
Upper bound: 34.67.
Response: Don't know;
Percentage: 4.60;
Lower bound: 1.48;
Upper bound: 10.51.
Response: Have not used the WWC intervention report to inform
decisions;
Percentage: 21.01;
Lower bound: 13.23;
Upper bound: 30.73.
Label: Practice guides:
Response: Very large or large extent;
Percentage: 10.06;
Lower bound: 5.28;
Upper bound: 16.95.
Response: Moderate extent;
Percentage: 27.78;
Lower bound: 18.84;
Upper bound: 38.23.
Response: Small extent;
Percentage: 16.77;
Lower bound: 10.12;
Upper bound: 25.44.
Response: Don't know;
Percentage: 2.91;
Lower bound: 0.70;
Upper bound: 7.75.
Response: Have not used the WWC practice guides to inform decisions;
Percentage: 42.48;
Lower bound: 32.26;
Upper bound: 52.70.
Label: Quick reviews:
Response: Very large or large extent;
Percentage: 12.93;
Lower bound: 6.66;
Upper bound: 21.92.
Response: Moderate extent;
Percentage: 16.66;
Lower bound: 9.79;
Upper bound: 25.70.
Response: Small extent;
Percentage: 24.65;
Lower bound: 15.93;
Upper bound: 35.21.
Response: Don't know;
Percentage: 5.78;
Lower bound: 2.35;
Upper bound: 11.55.
Response: Have not used the WWC quick reviews to inform decisions;
Percentage: 39.98;
Lower bound: 30.16;
Upper bound: 49.81.
Source: GAO analysis.
[End of table]
Table 8: Estimates and Confidence Intervals for Figure 14:
Q25: How likely or unlikely would you or your staff be to increase
your usage of the WWC if any of the following information were added
to the Web site?
Label: Reviews of efficacy of programs being used or considered in my
district;
Response: Very likely or somewhat likely;
Percentage: 67.92;
Lower bound: 59.77;
Upper bound: 76.06.
Label: A greater number of intervention reports showing positive
effects;
Response: Very likely or somewhat likely;
Percentage: 66.54;
Lower bound: 58.30;
Upper bound: 74.77.
Label: Additional practices with positives reviews;
Response: Very likely or somewhat likely;
Percentage: 60.49;
Lower bound: 51.95;
Upper bound: 69.03.
Label: Additional Topic Areas;
Response: Very likely or somewhat likely;
Percentage: 57.27;
Lower bound: 48.66;
Upper bound: 65.88.
Label: Additional information on interventions based on studies that
may not meet WWC standards;
Response: Very likely or somewhat likely;
Percentage: 55.97;
Lower bound: 47.43;
Upper bound: 64.51.
Label: Additional Practice Guides;
Response: Very likely or somewhat likely;
Percentage: 55.80;
Lower bound: 47.12;
Upper bound: 64.47.
Label: Additional studies reviewed;
Response: Very likely or somewhat likely;
Percentage: 54.89;
Lower bound: 46.24;
Upper bound: 63.53.
Label: Reviews that are more timely;
Response: Very likely or somewhat likely;
Percentage: 49.95;
Lower bound: 41.32;
Upper bound: 58.57.
Label: A broader definition of what studies meet WWC standards;
Response: Very likely or somewhat likely;
Percentage: 47.38;
Lower bound: 38.70;
Upper bound: 56.05.
Label: An easier web site to navigate;
Response: Very likely or somewhat likely;
Percentage: 47.01;
Lower bound: 38.30;
Upper bound: 55.73.
Label: Other Information;
Response: Very likely or somewhat likely;
Percentage: 17.37;
Lower bound: 8.69;
Upper bound: 29.58.
Source: GAO analysis.
[End of table]
Information from Teachers, Principals, and Researchers:
In addition to interviews with teacher, principal, and research
organizations, we obtained information about the usefulness of the WWC
by administering a questionnaire at four conferences of teachers and
principals. Table 9 provides more information about the conferences we
attended.
Table 9: Conferences Attended to Administer Questionnaires to Teachers
and Principals:
Conference: National Council of Teachers of Mathematics;
Regional/national: Regional;
Location: Nashville, Tennessee;
Conference dates: November 18-20, 2009;
Attendance dates: November 19-20, 2009.
Conference: National Council of Teachers of English;
Regional/national: National;
Location: Philadelphia, Pennsylvania;
Conference dates: November 19-24, 2009;
Attendance dates: November 21-22, 2009.
Conference: ASCD (formerly the Association of Supervisors and
Curriculum Developers);
Regional/national: National;
Location: San Antonio, Texas;
Conference dates: March 6-8, 2010;
Attendance dates: March 7-8, 2010.
Conference: National Association of Secondary School Principals;
Regional/national: National;
Location: Phoenix, Arizona;
Conference dates: March 12-14, 2010;
Attendance dates: March 12-13, 2010.
Source: GAO.
[End of table]
We selected these conferences because they were relevant to segments
of the WWC's target population that we were not reaching through our
survey and they were held at times that coincided with our report time
frames. At each of these conferences, conference organizers agreed to
have GAO have a table either inside the exhibit hall or just outside
it. The questionnaires included questions on awareness and use of WWC-
-including use of specific products and use of other information
sources to identify effective educational practices. For those who had
not used the WWC, we also asked them to specify the reason they had
not used it. The information gathered through the questionnaires is
not generalizable and does not represent the views of teachers and
principals nationwide.
[End of section]
Appendix II: Other Sources of Information Districts Use To Identify
Effective Education Practices:
Source of information: Personal Experience;
Estimated percent of district officials that find this source Very
Useful or Useful: 80.6; (76.46, 84.79);
Estimated percent of district officials that find this source Somewhat
or Slightly Useful: 16.6; (12.73, 20.53);
Estimated percent of district officials find this source not at all
useful or no opinion: 2.3; (0.90, 4.67);
Not used: 0.5; (0.06, 1.67).
Source of information: Local data and/or internal research;
Estimated percent of district officials that find this source Very
Useful or Useful: 77.4; (72.85, 81.84);
Estimated percent of district officials that find this source Somewhat
or Slightly Useful: 14.9; (11.30, 19.13);
Estimated percent of district officials find this source not at all
useful or no opinion: 6.2; (3.79, 9.58);
Not used: 1.5; (0.48, 3.55).
Source of information: Peer Conferences;
Estimated percent of district officials that find this source Very
Useful or Useful: 77.3; (72.93, 81.74);
Estimated percent of district officials that find this source Somewhat
or Slightly Useful: 16.7; (12.88, 20.58);
Estimated percent of district officials find this source not at all
useful or no opinion: 4.2; (2.28, 7.14);
Not used: 1.7; (0.60, 3.73).
Source of information: Mentors/Colleagues;
Estimated percent of district officials that find this source Very
Useful or Useful: 74.5; (69.97, 79.08);
Estimated percent of district officials that find this source Somewhat
or Slightly Useful: 21; (16.74, 25.17);
Estimated percent of district officials find this source not at all
useful or no opinion: 3.2; (1.53, 5.84);
Not used: 1.3; (0.42, 3.05).
Source of information: Education-related periodicals;
Estimated percent of district officials that find this source Very
Useful or Useful: 70.9; (66.00, 75.72);
Estimated percent of district officials that find this source Somewhat
or Slightly Useful: 23; (18.50, 27.50);
Estimated percent of district officials find this source not at all
useful or no opinion: 3.2; (1.48, 6.02);
Not used: 2.9; (1.34, 5.49).
Source of information: Associations of educators or researchers;
Estimated percent of district officials that find this source Very
Useful or Useful: 65; (59.94, 70.04);
Estimated percent of district officials that find this source Somewhat
or Slightly Useful: 23.1; (18.59, 27.57);
Estimated percent of district officials find this source not at all
useful or no opinion: 6.3; (3.83, 9.66);
Not used: 5.6; (3.39, 8.72).
Source of information: Academic journals;
Estimated percent of district officials that find this source Very
Useful or Useful: 64.3; (59.22, 69.41);
Estimated percent of district officials that find this source Somewhat
or Slightly Useful: 27.7; (22.90, 32.45);
Estimated percent of district officials find this source not at all
useful or no opinion: 4.7; (2.59, 7.81);
Not used: 3.3; (1.64, 5.85).
Source of information: Online databases (ERIC or others);
Estimated percent of district officials that find this source Very
Useful or Useful: 54.4; (49.10, 59.63);
Estimated percent of district officials that find this source Somewhat
or Slightly Useful: 30.6; (25.62, 35.53);
Estimated percent of district officials find this source not at all
useful or no opinion: 7.9; (5.12, 11.59);
Not used: 7.1; (4.58, 10.52).
Source of information: Community and parent input;
Estimated percent of district officials that find this source Very
Useful or Useful: 51.7; (46.48, 56.97);
Estimated percent of district officials that find this source Somewhat
or Slightly Useful: 39.9; (34.67, 45.06);
Estimated percent of district officials find this source not at all
useful or no opinion: 6.2; (3.82, 9.53);
Not used: 2.2; (0.95, 4.19).
Source of information: University based research institutions;
Estimated percent of district officials that find this source Very
Useful or Useful: 50.3; (45.11, 55.49);
Estimated percent of district officials that find this source Somewhat
or Slightly Useful: 35.4; (30.25, 40.45);
Estimated percent of district officials find this source not at all
useful or no opinion: 9.3; (6.30, 13.16);
Not used: 5; (2.94, 7.98).
Source of information: Regional Educational Laboratories;
Estimated percent of district officials that find this source Very
Useful or Useful: 45.9; (40.66, 51.03);
Estimated percent of district officials that find this source Somewhat
or Slightly Useful: 31.3; (26.32, 36.18);
Estimated percent of district officials find this source not at all
useful or no opinion: 11.5; (8.18, 15.59);
Not used: 11.4; (8.16, 15.36).
Source of information: State government offices and/or outreach
centers;
Estimated percent of district officials that find this source Very
Useful or Useful: 39.8; (34.68, 45.01);
Estimated percent of district officials that find this source Somewhat
or Slightly Useful: 37.8; (32.64, 42.88);
Estimated percent of district officials find this source not at all
useful or no opinion: 15.79; (11.87, 20.10);
Not used: 6.7; (4.33, 9.91).
Source of information: Other federal outreach centers;
Estimated percent of district officials that find this source Very
Useful or Useful: 17.9; (13.99, 21.83);
Estimated percent of district officials that find this source Somewhat
or Slightly Useful: 34.7; (29.65, 39.77);
Estimated percent of district officials find this source not at all
useful or no opinion: 24; (19.41, 28.67);
Not used: 23.3; (18.82, 27.86).
Source of information: What Works Clearinghouse;
Estimated percent of district officials that find this source Very
Useful or Useful: 24.4; (20.07 28.76;
Estimated percent of district officials that find this source Somewhat
or Slightly Useful: 14.8; (11.20, 19.09);
Estimated percent of district officials find this source not at all
useful or no opinion: 16.1; (12.25 20.57);
Not used: 44.7; (39.37 49.96).
Source of information: Doing What Works;
Estimated percent of district officials that find this source Very
Useful or Useful: 22.8; (18.40, 27.10);
Estimated percent of district officials that find this source Somewhat
or Slightly Useful: 13.8; (10.42, 17.68);
Estimated percent of district officials find this source not at all
useful or no opinion: 14.2; (10.52, 18.50);
Not used: 49.3; (44.03, 54.62).
Source of information: Non-profit organization;
Estimated percent of district officials that find this source Very
Useful or Useful: 21.7; (17.63, 25.83);
Estimated percent of district officials that find this source Somewhat
or Slightly Useful: 43.9; (38.58, 49.11);
Estimated percent of district officials find this source not at all
useful or no opinion: 17; (13.06, 21.60);
Not used: 17.4; (13.22, 21.59).
Source of information: RANDs Promising Practices;
Estimated percent of district officials that find this source Very
Useful or Useful: 10.1; (7.34, 13.36);
Estimated percent of district officials that find this source Somewhat
or Slightly Useful: 14; (10.48, 18.09);
Estimated percent of district officials find this source not at all
useful or no opinion: 18.9; (14.54, 23.25);
Not used: 57.1; (51.77, 62.40).
Source of information: Child Trends;
Estimated percent of district officials that find this source Very
Useful or Useful: 9.7; (6.74, 13.29);
Estimated percent of district officials that find this source Somewhat
or Slightly Useful: 14.5; (10.93, 18.77);
Estimated percent of district officials find this source not at all
useful or no opinion: 16.73; (12.84, 21.25);
Not used: 59.1; (53.84, 64.33).
Source: GAO analysis of survey results from the following questions:
(4) How useful, if at all, are each of the following research
evaluation resources to you or your staff in identifying effective
practices to implement in your district?; and (5) Recognizing that
research evaluation resources are not necessarily the primary sources
of information used to identify effective education practices, GAO is
also interested in the role of other information sources. How useful,
if at all, are each of the following sources to you or your staff in
identifying effective practices to implement in your district?
[End of table]
[End of section]
Appendix III: IES and WWC Response to Expert Panel Recommendations:
Expert panel recommendation: 1. Full review. IES should commission a
full review of the WWC, including a review of the Clearinghouse's
mission and of the WWC practice guides, which the panel did not
attempt to evaluate. The panel also recommends that IES consider
instituting a regular review process to ensure that the WWC is using
the most appropriate standards in its work;
IES/WWC response: IES is considering an appropriate mechanism and time
for conducting a complete review of the WWC. IES believes that the
first 2 years of the current contract necessitated a tremendous
development effort to transfer the infrastructure of the Clearinghouse
in year one from one contractor to another, and in year two, to
complete reviews in a consistent manner that began under the original
contract. Now that the Clearinghouse is more clearly in the production
phase, this may be the appropriate time to plan for a complete review;
Implementation status: Under consideration.
Expert panel recommendation: 2(i). Protocol templates. WWC should
develop standards for crossover and assignment noncompliance, and for
adjusting intention to treat effects across studies;
IES/WWC response: IES is currently considering having the WWC develop
a standard for assessing crossover compliance, following the process
recently used to revise its attrition standard; Currently, the WWC
documents crossover reported in studies. Principal investigators have
discretion to use this information to determine whether a study
represents a reasonable test of the intervention. Evidence of
crossover and assignment noncompliance is documented in the
intervention report and its appendix table A.1. Readers can use that
information to assess the findings. IES agrees there is value in
adjusting intent-to-treat effects for compliance, but believes this
adjustment is inconsistent with its goal of having the WWC be
transparent in how it reports findings. Making its own estimates to
account for compliance will lead to differences between what the WWC
reports and what is found in publicly available literature; Currently
the WWC does adjust for clustering when authors report their findings
incorrectly. However, the purpose of the clustering adjustment is to
correct for an analytic problem in the methods authors use to estimate
variances, which generally causes them to overstate the precision of
their findings. In contrast, adjusting for compliance will yield an
alternate estimate of effects that may differ from the one reported by
the study;
Implementation status: Under consideration.
Expert panel recommendation: 2(ii). Protocol templates. Develop
standards for documenting the program received in the control arm of
randomized experiments (or by members of the comparison group in quasi-
experimental designs), and potentially incorporating this information
in making comparisons across studies and/or interventions;
IES/WWC response: Though not based on a standard, WWC practice is for
reviewers to document the counterfactual in study review guides and in
intervention reports (the information is reported in appendix table
A.1). Reviewers routinely send author queries for this information, if
it is not provided in the study; IES has asked the WWC to assess how
other review organizations report counterfactual information and the
utility of incorporating this information into its reports. IES
officials are also considering an alternative approach that would code
information about the counterfactual in a study into the study
database, which then would generate summary tables that would report
results for studies that have similar counterfactuals. This approach
has downsides as well, since the set of counterfactuals could be quite
varied and many assumptions would have to be made to group
counterfactuals together. We are therefore proceeding cautiously in
making any changes to current WWC practice;
Implementation status: Considered but not planning to implement.
Expert panel recommendation: 2 (iii). Protocol templates. Revise
standards for multiple comparisons in light of the recent research
report by Peter Schochet entitled Guidelines for Multiple Testing in
Experimental Evaluations of Educational Intervention;
IES/WWC response: WWC staff consulted with Dr. Schochet to investigate
the possibility of revising the multiple comparison standards;
Dr. Schochet indicated that his report focused on issues related to
multiple comparisons within single studies. It did not tackle issues
related to multiple comparisons issues that may arise when
synthesizing evidence for a set of studies. WWC procedures are
consistent with his report for handling multiple comparisons within a
study;
Implementation status: Considered but not planning to implement.
Expert panel recommendation: 2(iv). Protocol templates. Reconsider the
current process of setting different attrition standards in different
topic areas;
IES/WWC response: At the time of the National Board of Education
Sciences Expert Panel's data collection, the WWC was already reviewing
its attrition standards. The WWC released new attrition standards in
December 2008 in the Procedures and Standards Handbook. The new
standards requires a principal investigator in a topic area to choose
one of two well-specified attrition boundaries, and the standards
include guidance on how to choose between the boundaries based on the
nature of research in the topic area; The attrition discussion is in
the WWC Procedures and Standards Handbook (Version 2.0) posted on the
Clearinghouse's Web site;
Implementation status: Implemented.
Expert panel recommendation: 2(v). Protocol templates. Establish a
protocol to keep track of potential conflicts of interest, such as
cases where a study is funded or conducted by a program developer, and
consider making that information available in its reports;
IES/WWC response: IES is considering options for collecting and
documenting potential conflicts of interest; Sources of funding are
rarely included in published documents beyond government and
foundation support. An alternate source of information for tracking
potential conflicts of interest would be for the WWC to request that
study authors identify their source of funding, which would provide
the WWC with a basis for flagging a potential conflict of interest.
Any effort would depend on cooperation from authors because the WWC
has no leverage to formally require authors to declare potential
conflicts (which some academic journals require as a condition for
publication). WWC's experience to date is that study authors
frequently fail to respond to requests for additional information, and
IES officials expect that many study authors likewise will not respond
to requests for information about funding sources, or may judge that
it is not in their proprietary interest to provide the information.
Currently the WWC only queries authors in cases where the
Clearinghouse needs additional information. Querying all authors and
tracking their responses would increase costs for intervention reports;
Another potential option is to ask developers, when they are reviewing
the list of studies WWC found during the literature search for
comprehensiveness, to note any studies that they funded;
Implementation status: Under consideration.
Expert panel recommendation: 2(vi). Protocol templates. Define
precisely the standards for "randomization" in a multilevel setting;
IES/WWC response: The current version of the handbook gives guidance
on standards for random assignment in simple cases. The next version
of the handbook (forthcoming in 2010) will provide guidance and
examples for multilevel settings, with explicit guidance on acceptable
practice and potential issues with random assignment in a multilevel
setting;
Implementation status: Implemented.
Expert panel recommendation: 3. Documentation of search process. WWC
should expand the protocol templates to specify more explicit
documentation of the actual search process used in each topic area and
maintain a record of the results of the process that can be used to
guide decision making on future modifications;
IES/WWC response: IES asked the WWC to review the search process. The;
WWC now takes steps to ensure that search records are maintained. Each
team and the library maintain a record of conducted searches. More
documentation on the process will be included in the forthcoming
revision of the handbook;
Implementation status: Implemented.
Expert panel recommendation: 4. Reliability of eligibility screening.
WWC should conduct regular studies of the reliability of the
eligibility screening process, using two independent screeners, and
use the results from these studies to refine the eligibility screening
rules and screening practices;
IES/WWC response: The WWC is undertaking a pilot using five recent
evidence reports in different topic areas. Because WWC screeners are
encouraged to pass to the next stage any study for which they are
uncertain about eligibility, the proportion of eligible studies that
are excluded is the salient error rate (the other source of error is
when screeners include an ineligible study in a review, but this error
is then offset by the review);
IES officials are not aware of any established standards for
acceptable error rates (there are tradeoffs between making Type I vs.
Type II errors relating to cost), but will examine this issue further.
If the screening error rate is larger than the IES and the WWC
believes is acceptable, IES officials will assess whether additional
training or two screeners is appropriate given the different costs and
benefits of each approach;
Implementation status: Under consideration.
Expert panel recommendation: 5. Documentation of screening process.
WWC reports should include a flow chart documenting the flow of
studies through each review and number of studies excluded at each
point, and a table of excluded studies, listing specific reasons for
exclusion for each study;
IES/WWC response: Currently, reference lists for WWC intervention
reports include all studies, both eligible and ineligible, located in
the search process. Ineligible studies are flagged with the primary
reason for not qualifying for further WWC review. Intervention reports
do not list materials such as product descriptions or reviews of
products that are deemed not relevant to the intervention being
reviewed; To make the number of studies (both eligible and ineligible
studies) more apparent to readers, the WWC will add a text box to
intervention reports located in front of the listing of reports. The
text box will summarize the number of studies that met different
conditions (this approach currently is used for reports in which none
of the studies meet standards). The box will serve the same purpose as
a flow chart but the codes used to describe the final status for
reports will be the same ones currently used in the citation appendix.
The WWC plans to begin including the text box in reports released in
2010 and thereafter;
Implementation status: Implemented.
Expert panel recommendation: 6. Misalignment adjustment. In cases
where a study analysis is "misaligned," WWC staff should request that
study authors reanalyze their data correctly, taking into account the
unit of randomization and clustering. The panel recommends that the
results from the process be compared to the adjustment procedure
currently specified, to develop evidence on the validity of the latter;
IES/WWC response: Ideally, the primary source for reanalyses of data
would be study authors. However, as noted above in response to
recommendation 2(v), it is common for authors not to respond to the
WWC's requests for additional information. Reanalyzing the data also
would require additional effort by the authors and would run into
difficulties when studies are dated or are based on data that has been
destroyed to comply with confidentiality or privacy restrictions; The
WWC recently undertook a survey of published clustering estimates. It
found that the WWC's current default clustering correction is
consistent with published estimates for achievement and behavioral
outcomes. The WWC will continue to monitor research developments on
this topic;
Implementation status: Considered but not planning to implement.
Expert panel recommendation: 7. Combining evidence across multiple
studies. WWC should re-evaluate its procedures for combining evidence
across studies, with specific attention to the issue of how the rules
for combining evidence can be optimally tuned, given the objectives of
the WWC review process and the sample sizes in typical studies for a
topic area;
IES/WWC response: There are, of course, many possible ways to
summarize evidence. Given its intended broad and primarily
nontechnical audience, the WWC's current approach is designed to be
transparent and easily explained. IES believes that having the WWC
conduct its own analyses to estimate intervention effects, as
statistical meta-analyses do, would be inconsistent with these goals;
However, as an alternative to modifying the WWC's main approach for
reporting findings, IES is considering having the WWC conduct
supplemental meta-analyses related to specific questions of interest,
and releasing these findings as a separate report that would
complement intervention reports. For example, a report could analyze
whether computer mediated approaches to teaching reading are more or
less effective than approaches that rely solely on teachers, based on
already-released interventions reports. Having a separate report
enables the WWC to continue using its current transparent approach,
while also using statistical techniques that combine evidence in other
ways;
Implementation status: Under consideration.
Expert panel recommendation: 8. Reporting. (i) Published reports on
the Web site should include the topic area protocols, as well as more
information on the screening process results that led to the set of
eligible studies actually summarized in the topic area reports. (ii)
WWC should make available its Standards and Procedures Handbook,
including appendixes, as well as all other relevant documents that
establish and document its policies and procedures;
IES/WWC response: Topic area protocols are available on the topic area
home pages; Just after the expert panel's report, the WWC released its
Procedures and Standards Handbook in December 2008. A revision
currently is under way that will include more detail on the screening
process. See the response to (3) above related to results of the
screening process and the response to (5) regarding the results of the
screening process;
Implementation status: Implemented.
Expert panel recommendation: 9. Practice guides. Clearly separate
practice guides from the topic and intervention reports;
IES/WWC response: IES agrees that these products need to remain
distinct. Practice guides are on a separate Web site tab that
separates them from intervention reports. The next revision of the
handbook (forthcoming in 2010) will include a chapter describing the
practice guide development process and how it is different from that
of the evidence reports; The recently released guide on What Works for
Practitioners also provides more information on reports and practice
guides, and the WWC is preparing a video tutorial that will explain
the differences to users;
Implementation status: Implemented.
Expert panel recommendation: 10. Outreach and collaboration with other
organizations. (i) The WWC should build and maintain a relationship
with national and international organizations focusing on systematic
reviews to engage in the broader scientific community and learn about
the latest standards and practices. (ii) The WWC should convene
working groups with a mixture of researchers (including specialists in
education research and systematic reviews) to address the development
of new standards for the review and synthesis of studies;
IES/WWC response: The WWC tries to keep abreast of developments in the
field, for example, by routinely checking materials from the Cochrane
Collaborative when developing new standards or approaches; Most
recently, the WWC has undertaken the following outreach efforts to
connect with other organizations conducting systematic reviews:
* The WWC sponsored a forum on research methods in December 2008 that
featured speakers from the National Academy of Sciences, the National
Cancer Institute, and the Cochrane Collaboration;
* In June 2009 WWC staff attended the Cochrane conference on practice
guides in June to learn about state of the art methods in research
synthesis and practice guides;
* The WWC is presenting a workshop on WWC standards at the upcoming
annual conference of the Association of Public Policy and Management;
The WWC has also met with six international contacts (from Sweden,
Denmark, Hungary, England, Interamerican Development Bank, and
Trinidad/Tobago) in response to inquiries about how governments or
organizations could implement their own clearinghouse operations;
Recently the WWC began a webinar series to disseminate its new
practice guides. The webinar includes researchers and practitioners in
its audience; The WWC convened two groups of researchers to develop
its forthcoming standards on single-subject designs and regression
discontinuity designs. It will continue to bring together researchers
as needs for new standards are identified. This approach will continue
to be used for developing new standards;
Implementation status: Implemented.
Source: GAO analysis of IES and WWC data.
[End of table]
[End of section]
Appendix IV: Comments from the Department of Education:
United States Department Of Education:
Institute Of Education Sciences:
The Director:
555 New Jersey Ave., NW:
Washington, D.C. 20208:
July 6, 2010:
Ms. Cornelia M. Ashby:
Director, Education, Workforce, and Income Security Issues:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Ms. Ashby:
Thank you for the opportunity to respond to the Government
Accountability Office (GAO) draft report, Improved Dissemination and
Timely Product Release Would Enhance the Usefulness of the What Works
Clearinghouse (GA040-644).
This report indicates that GAO, as well as a congressionally mandated
panel of experts, found that the What Works Clearinghouse's review
process follows accepted standards for evaluating research on the
effectiveness of education interventions. It notes that the
Clearinghouse is responding to recommendations made by the expert
panel to further improve its review and reporting processes, and to
researchers who have criticized the WWC for excluding some types of
research designs that may be appropriate for evaluating certain
education programs, such as special education. It also recognized that
product output and scope of the WWC have increased.
While the report emphasized the need for more timely release of WWC
reports, it noted, but did not highlight, other important findings
from the school district survey that point to important avenues for
improvement. For example, a higher proportion of district respondents
cited having evidence reviews of programs that they might use in their
local areas (68 percent) or that show positive effects (67 percent) as
a reason to use WWC products than cited getting reviews more quickly
(50 percent) (Figure 15, page 36). Given that the timetable of WWC
reports is not posted publicly, we are not sure how to interpret
districts' concerns with timeliness, but we have heard their need for
more relevant evidence reports covering more educational areas and
programs. As GAO noted, a backlog of reports that built up during a
time of senior leadership turnover has been resolved, and many new
reports are soon to be released. In addition, the Institute of
Education Sciences is continuing its efforts to ensure that expansion
into new topic areas is responsive to the needs of the field.
We appreciate the opportunity to comment on the draft report. The
Department of Education has prepared the attached comments in response
to your draft report. If you have any questions regarding this
response, please contact Dr. Rebecca Maynard, Commissioner for
Education Evaluation and Regional Assistance, in the Institute of
Education Sciences at (202) 208-1289.
Sincerely,
Signed by:
John Q. Easton:
Director:
Attachments:
[End of letter]
Attachment 1:
IES Response to GAO Report: Improved Dissemination and Timely Product
Release Would Enhance the Usefulness of the What Works Clearinghouse,
Draft:
Recommendation 1:
To consistently release WWC products in a timely manner, we recommend
the Secretary of Education direct IES to develop and implement
strategies that help avoid future backlogs and ensure that IES' review
and approval processes keep pace with increased contractor production.
Strategies could include shifting IES resources to ensure sufficient
staff time for managing the peer review process and streamlining its
approval process.
Response:
The Department of Education (Department) agrees with GAO's
recommendation and is developing and implementing strategies to ensure
that IES' review and approval processes keep pace with production. The
Director of the Institute of Education Sciences has recently appointed
a Commissioner of the National Center for Education Evaluation and
Regional Assistance who will oversee the content and operations of the
What Works Clearinghouse.
Recommendation 2:
To better track the usefulness and costs of the WWC, we recommend that
the Secretary of Education direct IES to:
* Incorporate findings from its cost studies to develop performance
measures related to costs, such as identifying a range of acceptable
costs per product and using that information to monitor contractor
spending.
* Develop performance measures related to product usefulness and
periodically assess whether WWC products are meeting the needs of
target audiences, by gathering information on product usefulness in
the proposed survey or through other means.
Response:
First, as noted in the report, WWC is currently conducting assessments
for streamlining procedures and costs at IES' request. The results,
due at the end of October 2010, will be incorporated into future WWC
work plans and IES's monitoring of contractor performance and costs.
IES intends to undertake such assessments on an ongoing basis for
continuous improvement.
Second, the WWC will continue to monitor and evaluate consumer use and
assessment of WWC products, as well as gather information regarding
suggestions for improvements. In addition, the WWC will be included as
part of IES' review of its own performance, which will include
surveying customers on satisfaction.
In addition, the Clearinghouse already has in place a number of tools
for users to provide input, and we will continue to emphasize their
availability to the education community. The WWC Help Desk allows
users to contact the WWC to send suggestions and ask questions about
specific products. Users contact the Help Desk through an 800 number
or through the WWC Web site. Since 2007, the WWC has responded to over
2,100 inquiries to the Help Desk.
Users have suggested over 200 interventions for the WWC to review, and
have suggested over 30 different topics for practice guides. This
information is continuously reviewed by WWC staff and informs
priorities. The WWC has used Web-based tools for soliciting specific
feedback from users. For example, the WWC has posted potential
practice guide topics on a Web-based survey, having users "vote" for
their favorite topics. The forthcoming practice guides on Reading
Comprehension, Fractions, and Teaching Writing were all identified
through this process.
In 2007, 2008, and 2009, the WWC conducted focus groups to solicit
feedback from educators on several aspects of its work. These included
such topics as the usability of the WWC Web site, which led to
improvement to the site functionality, and the format of practice
guides, which led to changes in the layout of the guides.
Visits to the WWC Web site have increased substantially over time.
Between fiscal years 2008 and 2009, the number of visits rose by 45
percent to 772,000. And, to date in 2010, there have been 618,000
visits to the Web site, an increase of 21 percent relative to the same
period in fiscal year 2009. IES will continue to track all of these
measures to ensure that we are using solid information on customer use
and satisfaction.
Recommendation 3:
To reach more members of the target audience, we recommend the
Secretary of Education direct IES to assess and improve its
dissemination efforts to promote greater awareness and use of the WWC,
for example, by developing a way to inform school districts of new
products or encouraging educator professional development programs to
focus on research-based practices such as those discussed in practice
guides.
Response:
The Department agrees with GAO's recommendation and is developing
further means to assess and improve dissemination efforts. IES'
monitoring of the WWC contract requires implementation of an Annual
Communication Plan that specifies how the contractor will conduct
outreach, dissemination, and communication activities that include,
for example, the WWCFlash, media and trade organization outreach,
development of dissemination partnerships with organizations and the
education community, WWC sponsored events and forums, and targeted
product outreach. IES has made all WWC products available through the
Department's Educational Resources Information Center (ERIC), an
online digital library of education research and information, and
provides RSS feeds of WWC products to ERIC users. The Department's
National Library of Education makes all WWC products, together with
all items indexed in ERIC, available to Google, MSN, and Yahoo search
engines for harvesting and, therefore, accessible to Web users
worldwide. All WWC products are now available to WorldCat, an
international bibliographic database that is used by libraries around
the world to identify and locate resources.
IES will continue to develop and implement the Communication Plan
vehicle to assess and improve dissemination, as well as fully utilize
other Department dissemination resources. One example of a new vehicle
that the Clearinghouse has in the works to expand its dissemination
partnerships with organizations and the education community is a
monthly WWC Newsletter on up-to-date information on recently released
Clearinghouse products as well as those in line for release in the
month ahead.
[End of section]
Appendix V: GAO Contact and Staff Acknowledgments:
GAO Contact:
Cornelia M. Ashby, (202) 512-7215, ashbyc@gao.gov:
Staff Acknowledgments:
In addition to the contact named above, individuals making key
contributions to this report include Elizabeth Morrison (Assistant
Director), Nagla'a El-Hodiri (analyst-in-charge), James Ashley, Carl
Barden, James Bennett, Valerie Caracelli, Laura Henry, Geoffrey King,
Jill Lacey, Luann Moy, Robert Owens, Cathy Roark, Stephanie Shipman,
Kate Van Gelder, and Craig Winslow.
[End of section]
Related GAO Products:
Program Evaluation: A Variety of Rigorous Methods Can Help Identify
Effective Interventions. [hyperlink,
http://www.gao.gov/products/GAO-10-30]. Washington, D.C.: November 23,
2009.
Teacher Quality: Sustained Coordination among Key Federal Education
Programs Could Enhance State Efforts to Improve Teacher Quality.
[hyperlink, http://www.gao.gov/products/GAO-09-593]. Washington, D.C.:
July 6, 2009.
Teacher Preparation: Multiple Federal Education Offices Support
Teacher Preparation for Instructing Students with Disabilities and
English Language Learners, but Systematic Departmentwide Coordination
Could Enhance This Assistance. [hyperlink,
http://www.gao.gov/products/GAO-09-573]. Washington, D.C.: July 20,
2009.
No Child Left Behind Act: Education Actions Could Improve the
Targeting of School Improvement Funds to Schools Most in Need of
Assistance. [hyperlink, http://www.gao.gov/products/GAO-08-380].
Washington, D.C.: February 29, 2008.
Program Evaluation: Strategies for Assessing How Information
Dissemination Contributes to Agency Goals. [hyperlink,
http://www.gao.gov/products/GAO-02-923]. Washington, D.C.: September
30, 2002.
The Evaluation Synthesis. [hyperlink,
http://www.gao.gov/products/PEMD-10.1.2]. Washington, D.C.: March 1992.
[End of section]
Footnotes:
[1] 20 U.S.C. §§ 6301-7941. The mission and functions to be performed
by IES are set out at 20 U.S.C. § 9511.
[2] This is the second 5-year contract for the Clearinghouse. The
first contract for about $27 million expired in 2007.
[3] H.R. Comm. on Appropriations, 111th Cong., Omnibus Appropriations
Act, 2009: Comm. Print of the Comm. on Appropriations U.S.
Representatives on H.R. 1105/Public Law 111-8, at 1483 (Comm. Print
2009).
[4] Thirty-seven states and the District of Columbia and 74 percent of
surveyed districts responded to our survey.
[5] WWC's mission is consistent with IES's broader mission to bring
rigorous and relevant research, evaluation, and statistics to the
nation's education system.
[6] The current priority for the 2006-2010 REL contract period is
providing policymakers and practitioners with expert advice, training,
and technical assistance on how to interpret the latest findings from
scientifically valid research pertaining to requirements of the ESEA.
[7] This contract was awarded to the American Institutes for Research
and the Campbell Collaboration. In 2007, Education awarded the second
5-year contract to Mathematica Policy Research, Inc. to operate the
WWC.
[8] The initial topic areas chosen in 2003 were: beginning reading, K-
12 math achievement, dropout prevention, adult literacy, peer-assisted
learning, reducing delinquency, and English language acquisition.
[9] IES dropped the peer-assisted learning topic area, which would
have covered interventions related to students learning with and from
other students--generally in the same class and at a similar academic
level.
[10] The WWC intervention reports primarily focus on branded products.
[11] In June 2010 IES made public its standards for two additional
study designs: regression discontinuity and single case design studies.
[12] IES uses some of this information to determine a performance-
based award for the contractor.
[13] GAO, Program Evaluation: A Variety of Rigorous Methods Can Help
Identify Effective Interventions, [hyperlink,
http://www.gao.gov/products/GAO-10-30] (Washington, D.C.: Nov. 23,
2009).
[14] For example, WWC rates the credibility of a study's evidence
along a continuum.
[15] To help educators understand the research behind a WWC report,
the WWC (1) combines information on the size and number of studies
reviewed to rate the extent of evidence as small or medium/large; (2)
includes an overall rating of effectiveness on each measured outcome,
which combines the size and direction of effects, statistical
significance, and the quality of the research designs; and (3) reports
the average improvement index across studies as the expected change in
percentile rank for an average control group student if the student
had received the intervention.
[16] H. Brown, D. Card, K. Dickersin, J. Greenhouse, J. Kling, and J.
Littell, Expert Report on the What Works Clearinghouse, a report
prepared by the National Board for Education Sciences, 2008. The
expert panel was convened by the National Board for Education Sciences
in 2008 in response to the Senate Appropriations Committee. S. Rep.
No. 110-410 at 228-29. The National Board for Education Sciences,
consisting of 15 voting members appointed by the President with the
advice and consent of the Senate, provides IES guidance and oversight.
20 U.S.C. § 9516. The mandate directed the Board to convene leading
experts in rigorous evaluations to assess the WWC, specified that
panel members should be free of conflicts of interest and that a
report with any recommendations was to be submitted within 4 months.
Expert panel members included economists, statisticians, and
professors with expertise in other systematic review efforts in the
fields of health care, and social policy.
[17] The WWC considered but is not implementing three of the panel's
recommendations. One recommendation suggested the WWC develop
standards for documenting the program received by the comparison group
that did not receive the intervention and potentially incorporating
this information when making comparisons across studies and/or
interventions. The other two related to WWC procedures for combining
evidence across studies and asking study authors to reanalyze their
data to correct a common error associated with the use of classrooms
rather than individual students in data analysis. This mismatch can
result in an overstatement of the statistical significance of the
effects of the intervention. The WWC maintains that its current
procedures are consistent with standard practices and has elected not
to ask authors to reanalyze their data. See appendix III for more
detail.
[18] The handbook documents the actions that WWC staff must take when
reviewing research and the items that must be included in the reports,
among other things. It is available at [hyperlink,
http://ies.ed.gov/ncee/wwc/pdf/wwc_procedures_v2_standards_handbook.pdf]
[19] WWC requires reviewers to select one of two levels of attrition
(higher or lower) depending on the topic area and context. WWC allows
a higher level of attrition for topic areas in which it assumes that
attrition is due to factors that are not strongly related to the
intervention. WWC allows a lower level of attrition for topic areas in
which attrition may be due to certain individuals choosing not to
participate in the intervention.
[20] In cases in which a study of a program is funded by the program
developer, the study authors may have incentives to find positive
effects of the program. Such incentives could call the validity of the
study's results into question.
[21] For example, see Robert Slavin and Dewi Smith, "The Relationship
Between Sample Sizes and Effect Sizes in Systematic Reviews in
Education," Educational Evaluation and Policy Analysis, vol. 31, no. 4
(2009): 500-506.
[22] WWC staff also contend that there is no statistical basis for
setting a minimum sample size and doing so would arbitrarily ignore
available evidence and potentially bias the findings of a systematic
review.
[23] Robert E. Slavin and Nancy A. Madden, "Measures Inherent to
Treatments in Program Effectiveness Reviews," paper presented at the
annual meetings of the Society for Research on Effective Education,
Crystal City, Virginia, March 3-4, 2008; and Robert E. Slavin, "What
Works? Issues in Synthesizing Educational Program Evaluations,"
Educational Researcher, vol. 37, no. 1 (Jan./Feb. 2008): 5.
[24] Intervention developers may intentionally or unintentionally
create a test that is more likely to favor the intervention because
they have financial or other interests in the success of the
intervention.
[25] While randomized control trials and quasi-experiments are
considered to be rigorous approaches in assessing program
effectiveness, they are not the only rigorous research designs
available and may not always be appropriate. For example, such
comparison group designs may not be appropriate for research on small
numbers of students receiving special education services in a self-
contained classroom. In such a case, an in-depth case study may be
more appropriate. Examples of other research methods include
statistical analyses of observational data, such as student records,
or analyses of surveys of an intervention's participants.
[26] The WWC produces intervention reports noting when no studies meet
standards.
[27] Specifically, the WWC developed standards--which were made
publicly available in June 2010--for reviewing single-case and
regression discontinuity designs. The WWC anticipates reviewing many
studies with single-case designs--studies that involve repeated
measurement of a single subject (e.g., a student or a classroom)--as
it evaluates interventions for special education. Regression
discontinuity designs compare outcomes for a treatment and control
group that are formed based on the results of a preintervention
measure.
[28] The WWC has published 12 practice guides as of May 2010. The
topics of these 12 practice guides are using data to support decision
making, helping students navigate the path to college, structuring out-
of-school time, assisting students in math, assisting students in
reading, reducing behavior problems, dropout prevention, improving
literacy, turning around low-performing schools, instruction for
English language learners, encouraging girls in math and science, and
organizing instruction and study.
[29] IES requires the contractor to file WWC annual plans that outline
planned product releases and other deliverables. IES and the
contractor update these plans once a year with revised estimates.
[30] Under the first contract (2002 to 2007), the WWC released 89
intervention reports, six topic reports, and three practice guides.
[31] Pub. L. No.111-5, 123 Stat. 115, 182.
[32] The current contract also requires WWC to create and maintain
other resources on its Web site, such as registries of researchers and
randomized trials and the WWC Policy and Procedures Handbook. IES
noted that these deliverables are either new or significantly enhanced
from those produced under the first WWC contract.
[33] IES indicated that the amount of research available meeting WWC
inclusion standards for a given report varied and had an impact on the
number of staff hours required in the production of reports. Reports
based on larger numbers of studies took more staff hours to complete
than those based on less available evidence.
[34] We discussed the backlog and its causes with IES officials in
February and May 2010. For the first six months of 2010, IES completed
the review of 59 report products (intervention reports and quick
reviews)--compared to 46 for the entirety of calendar year 2009--thus
eliminating the backlog.
[35] IES also spends about $200,000 per year on noncontracted WWC
expenses--including internal salaries, independent peer review
honorariums, and Web site support--which have not changed
significantly between the two 5-year contracts. In addition, three
practice guides were completed outside of the WWC contract, at a total
cost of about $319,000. IES noted that these preliminary guides were
produced through a less thorough process than the current process.
[36] Both contractors dedicated the same proportion of funds to WWC
products. The first contractor primarily published products in the
final year of the first contract (2007); however, products were
produced, reviewed, and modified--but not published--prior to that
year. As a result, despite limited publication in the first 4 years, a
large portion of the first contractor's expenditures were designated
for direct product costs.
[37] According to WWC staff, this system allows them to use results
from prior literature searches for related topics, rather than
conducting new searches. The current contractor also designed and
implemented standardized training for staff and subcontractors who
evaluate research. All WWC research evaluators complete 2 days of
training; are tested on WWC products, review standards, and policy;
and have initial reviews with monitored before working independently.
[38] The Organizational Assessment--Education's performance management
system--was developed in response to the requirements of Executive
Order 13450, Improving Government Program Performance, as well as the
Office of Personnel Management's requirement that each federal agency
evaluate its principal offices on an annual basis.
[39] The contract award fee plan includes performance measures related
to production, business management of the contract, and timeliness.
Business management of the contract includes cost management, business
relationships, efforts to meet small business subcontracting goals,
and accurate billing. These measures are linked to work specified in
the WWC contract and annual plans.
[40] While the WWC annually exceeded performance targets, it is
difficult to interpret these results as the performance measures
changed annually and, according to IES officials, the criteria for
meeting them were negotiated well into the fiscal year.
[41] In addition, IES determined that the current and prior
contractors generally met the award fee plan performance measures.
[42] The WWC's award fee plan includes cost management components but
has no cost per product measurements.
[43] This survey would ascertain whether IES has met its goal that at
least 25 percent of decision makers surveyed will have consulted the
Clearinghouse prior to making decisions on interventions in reading,
writing, mathematics, science, or teacher quality by 2013.
[44] WWC staff told us that the listserv had over 10,000 subscribers
and that Web site visits increase after conferences.
[45] IES's Web site hosts an "Ask A REL" page, where educators can
submit questions. "Ask A REL" is described on the Web site as being a
collaborative reference desk service provided by the 10 RELs that
functions much in the same way as a technical reference library.
[46] Thirty-seven states and the District of Columbia responded to our
survey. While the District of Columbia is not a state, we will refer
to the survey respondents as representing 38 states.
[47] The 95 percent confidence interval for this estimate is (36.7,
46.6).
[48] While based on 38 state-level respondents, this analysis provides
the minimum number (33) of states (overall) that have accessed the WWC
Web site. Regardless of whether or not the 13 states that did not
respond to our survey have accessed the Clearinghouse Web site, 33 is
about two-thirds of the 51 states and constitutes a majority of states.
[49] Our survey asked respondents to indicate the number of times they
had accessed the WWC Web site. Answer choices included never; less
than twice a year; between 2 and 6 times per year; between 7 and 11
times per year; monthly; and more than once a month.
[50] The 95 percent confidence interval for this estimate is (29.4 ,
38.8).
[51] The 95 percent confidence interval for this estimate is (8.3,
14.8).
[52] Fourteen states reported accessing the Clearinghouse six or fewer
times a year, as did an estimated 72 percent of districts. In addition
to time constraints (cited by 7 of the 14 states), five states
reported that they did not access the WWC more frequently because its
content was not relevant to their decisions.
[53] The 95 percent confidence interval for this estimate is (94.2,
98.4).
[54] Our survey asked states and districts to report how useful
certain sources of information were in identifying effective education
practices. The sources ranged from general (personal experience,
education periodicals) to specific (RELs, federal outreach centers).
In addition, we listed several research synthesis organizations by
name, including the WWC and the Best Evidence Encyclopedia. See
appendix II for more details.
[55] Between November 2009 and February 2010, we attended a regional
conference for the National Council for Teachers of Mathematics in
Nashville, Tenn., as well as three national conferences: National
Council for Teachers of English, ASCD (formerly the Association of
Supervision and Curriculum Development), and the National Association
of Secondary School Principals.
[56] One state official responded that he and his staff had not used
the WWC to inform any decisions on effective education practices,
while another state official responded "Don't know."
[57] The 95 percent confidence intervals for these estimates are
(55.5, 87.7) and (10.3, 29.8), respectively.
[58] The estimates and their 95 percent confidence intervals were as
follows: inform curriculum decisions--93 percent (85.5, 97.3); inform
professional development of teachers--89.4 percent (81.7, 94.7).
[59] The 95 percent confidence interval for this estimate is (64.3,
82.8).
[60] The estimates and their 95 percent confidence intervals were as
follows: practice guides--54.6 percent (44.4, 64.8); quick reviews--
54.2 percent (44.4, 64.3).
[61] The 95 percent confidence interval for this estimate is (13.7,
30).
[62] The 95 percent confidence interval for this estimate is (5.3, 17).
[63] Twenty-six states that had used the Clearinghouse reported that
they would be somewhat likely or very likely to increase their use of
the WWC if it had a greater number of intervention reports showing
positive effects--a number which depends both on the number of
interventions that the WWC reviews and whether the results of
available research meeting WWC standards show positive effects.
[64] The 95 percent confidence interval for this estimate is (59.8,
76.1).
[65] The 95 percent confidence interval for this estimate is (41.3,
58.6).
[66] GAO, Program Evaluation: A Variety of Rigorous Methods Can Help
Identify Effective Interventions, [hyperlink,
http://www.gao.gov/products/GAO-10-30] (Washington, D.C.: Nov. 23,
2009).
[67] The expert panel was convened by the National Board for Education
Sciences in 2008 in response to a mandate from the Senate
Appropriations Committee. The National Board for Education Sciences,
consisting of 15 voting members appointed by the President with the
advice and consent of the Senate, provides IES guidance and oversight.
The mandate directed the board to convene leading experts in rigorous
evaluations to assess the WWC and specified that panel members should
be free of conflicts of interest. Expert panel members included
economists, statisticians, and professors with expertise in other
systematic review efforts in the fields of health care, social policy,
and education.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
