Education's Data Management Initiative
Significant Progress Made, but Better Planning Needed to Accomplish Project Goals Gao ID: GAO-06-6 October 28, 2005As a condition of receiving federal funding for elementary and secondary education programs, states each year provide vast amounts of data to Education. While the need for information that informs evaluation is important (particularly with the No Child Left Behind Act), Education's data gathering has heretofore presented some problems. It has been burdensome to states because there are multiple and redundant requests administered by a number of offices. In addition, the resulting data supplied by states has not been accurate, timely, or conducive to assessing program performance. To improve the information by which it evaluates such programs and also to ease states' reporting burden, Education in 2002 initiated an ambitious, multiyear plan to consolidate elementary and secondary data collections into a single, department-wide system focused on performance. Given its importance, we prepared a study, under the authority of the Comptroller General, to provide Congress with information on its progress.
Through its Performance-Based Data Management Initiative (PBDMI), Education has consolidated and defined much of the data it anticipates collecting under a unified system. Education reports that many data definitions have been agreed-to and data redundancies eliminated. PBDMI officials also said that to date, however, it has not been able to resolve all remaining differences among the program offices that manage many of the different data collections. PBDMI officials have conducted extensive outreach to the states to advance the initiative. The outreach to states involved regional conferences, two rounds of site visits, and according to officials, $100,000 in grants to most states to help offset their costs. State data providers responding to our survey expressed general satisfaction with the department's outreach, but some were not optimistic that the initiative would ease their reporting burden or enhance their own analytic capacity. The states were not able to produce enough data during test submissions in 2003 and 2004 to enable data quality verification or phasing out the department's multiple data collections. With regard to the lack of sufficient data from many states, Education officials said some lack the technical capacity needed to produce new performance data requirements. State data providers reported having competing demands for their time and resources, given other federal initiatives. Education officials have decided to proceed with the undertaking and have developed a draft interim strategy for moving forward. But they currently have no formal plan for how they would overcome obstacles such as the lack of state data and other technical and training delays to the initiative.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-06-6, Education's Data Management Initiative: Significant Progress Made, but Better Planning Needed to Accomplish Project Goals
This is the accessible text file for GAO report number GAO-06-6
entitled 'Education's Data Management Initiative: Significant Progress
Made, but Better Planning Needed to Accomplish Project Goals' which was
released on October 28, 2005.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
United States Government Accountability Office:
GAO:
October 2005:
Education's Data Management Initiative:
Significant Progress Made, but Better Planning Needed to Accomplish
Project Goals:
GAO-06-6:
GAO Highlights:
Highlights of GAO-06-6, a report to congressional committees:
Why GAO Did This Study:
As a condition of receiving federal funding for elementary and
secondary education programs, states each year provide vast amounts of
data to Education. While the need for information that informs
evaluation is important (particularly with the No Child Left Behind
Act), Education‘s data gathering has heretofore presented some
problems. It has been burdensome to states because there are multiple
and redundant requests administered by a number of offices. In
addition, the resulting data supplied by states has not been accurate,
timely, or conducive to assessing program performance. To improve the
information by which it evaluates such programs and also to ease
states‘ reporting burden, Education in 2002 initiated an ambitious,
multiyear plan to consolidate elementary and secondary data collections
into a single, departmentwide system focused on performance. Given its
importance, we prepared a study, under the authority of the Comptroller
General, to provide Congress with information on its progress.
What GAO Found:
Through its Performance-Based Data Management Initiative (PBDMI),
Education has consolidated and defined much of the data it anticipates
collecting under a unified system. Education reports that many data
definitions have been agreed-to and data redundancies eliminated. PBDMI
officials also said that to date, however, it has not been able to
resolve all remaining differences among the program offices that manage
many of the different data collections.
PBDMI officials have conducted extensive outreach to the states to
advance the initiative. The outreach to states involved regional
conferences, two rounds of site visits, and according to officials,
$100,000 in grants to most states to help offset their costs. State
data providers responding to our survey expressed general satisfaction
with the department‘s outreach, but some were not optimistic that the
initiative would ease their reporting burden or enhance their own
analytic capacity. The states were not able to produce enough data
during test submissions in 2003 and 2004 to enable data quality
verification or phasing out the department‘s multiple data collections.
With regard to the lack of sufficient data from many states, Education
officials said some lack the technical capacity needed to produce new
performance data requirements. State data providers reported having
competing demands for their time and resources, given other federal
initiatives.
Education officials have decided to proceed with the undertaking and
have developed a draft interim strategy for moving forward. But they
currently have no formal plan for how they would overcome obstacles
such as the lack of state data and other technical and training delays
to the initiative.
Reporting to Education: A Sample of Data Collections Seeking
Information on Elementary and Secondary Programs in One State in 2004:
[See PDF for image]
[End of figure]
What GAO Recommends:
GAO recommends that Education (1) develop a strategy to help states
provide quality data, (2) develop a process within the department to
resolve critical, outstanding issues, and (3) develop a clear plan for
completing final aspects of PBDMI, including specific time frames and
indicators of progress toward the initiative‘s goals. Education agreed
with our recommendations.
www.gao.gov/cgi-bin/getrpt?GAO-06-6.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact David Bellis at (415) 904-
2272 or bellisd@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
Education Has Made Progress Defining Data to Be Collected under a
Consolidated System, but Project Officials are Unable to Reconcile All
Differences:
PBDMI Officials Have Worked Extensively with States on Data Preparation
and Submissions, but Most States Cannot Produce the Requested Data:
Education Is Proceeding with Implementation despite the Data Shortage
and without a Detailed Plan of Action:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Comments from the Department of Education:
Appendix III: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Data Collections That Could Be Reduced or Eliminated as a
Result of PBDMI:
Table 2: Education's Outreach Activities to Improve Data Quality under
the PBDMI:
Table 3: State Survey Responses on Education's Outreach Activities
through PBDMI:
Table 4: Percentage of States Reporting Which Goals Were Most
Important, Attainable, and Difficult to Achieve:
Table 5: Percentage of States Reporting the Extent to Which PBDMI Would
Improve Their Analytical Capacity:
Figures:
Figure 1: Time and Money--Estimated Annual State Burden Hours and Costs
for Select Elementary and Secondary Education Data Collections:
Figure 2: Proposed Design for PBDMI's Web-based Network:
Figure 3: Illustration of Key Actors Involved in Development of the
PBDMI:
Figure 4: Fiscal Year 2002-2009 Funding for Education's Data Management
Initiative, including Key Activities Planned for Project Initiation
through Implementation:
Figure 5: School Year 2003-2004 Performance Data Submitted by States to
PBDMI as of June 3, 2005:
Abbreviations:
CIO: chief information officer:
FAPE: free and appropriate public education:
IDEA: Individuals with Disabilities Education Act:
IES: Institute for Education Science:
IG: Inspector General:
NCLBA: No Child Left Behind Act of 2001:
OCR: Office for Civil Rights:
OELA: Office of English Language Acquisition:
OESE: Office of Elementary and Secondary Education:
OMB: Office of Management and Budget:
OSDFS: Office of Safe and Drug-free Schools:
OSERS: Office of Special Education and Rehabilitative Services:
OVAE: Office of Vocational and Adult Education:
PBDMI: Performance-Based Data Management Initiative:
PRA: Paperwork Reduction Act of 1995:
SEA: state education agency:
United States Government Accountability Office:
Washington, DC 20548:
October 28, 2005:
The Honorable Michael B. Enzi:
Chairman:
The Honorable Edward M. Kennedy:
Ranking Minority Member:
Committee on Health, Education, Labor, and Pensions:
United States Senate:
The Honorable John A. Boehner:
Chairman:
The Honorable George Miller:
Ranking Minority Member:
Committee on Education and the Workforce:
House of Representatives:
Each year, state education agencies provide vast amounts of information
to the U.S. Department of Education (Education) in order to fulfill
reporting requirements for federal programs supporting elementary and
secondary education. While this information is important for managing
programs, it has been accompanied by some problems. Reporting has been
burdensome for the state data providers because the department makes
its data requests through multiple, ongoing, and uncoordinated data
collections. By Education's own account, there are currently 200 active
data collections for elementary and secondary programs--each resulting
in approximately 10,000 "person hours" for design, administration,
collection, and reporting. From the vantage point of the department and
its program offices, the information it receives has customarily been
compromised because the schools, districts, and states reporting data
employ their own definitions and, in some cases, report data that is
inaccurate, incomplete, and not timely. Finally, in terms of program
evaluation, much of the data that Education has traditionally requested
has not necessarily focused on program performance. Yet the need for
evaluative data has grown, particularly with passage of laws such as
the No Child Left Behind Act of 2001, which requires states receiving
assistance under the act to report on, among other things, the
achievement of their students on academic assessments required under
that law.
To address these problems and better evaluate its programs, Education
in 2002 began an initiative to consolidate and improve the information
it requests from states on elementary and secondary education and to
seek more consistency and quality in the data states supply. The
Performance Based Data Management Initiative (PBDMI) is a large-scale
effort within the department to combine more than a dozen separate data
collections into a single collection system, and better focus the
information Education requests from states by eliminating duplication,
conflicting definitions, and information that is not useful for the
evaluation of its programs. The PBDMI represents an important step
forward for Education in its efforts to monitor the performance of the
nation's elementary and secondary schools. The initiative is also a
large-scale undertaking for state education agencies, which are
volunteering to help develop uniform data and test the new data
collection system while they continue to meet their ongoing reporting
requirements. The PBDMI was scheduled to begin phasing out the old data
collections by September 2005, following final testing of the new
system and training of department staff.
In view of its importance and the inherent challenges, therefore, we
have prepared a study under the authority of the Comptroller General to
provide Congress with information about Education's progress with the
PBDMI. We have examined Education's work to (1) define what performance-
related data it will collect from states on behalf of the program
offices, (2) assist states in their efforts to submit quality
information, and (3) utilize performance-related data to provide
enhanced analytic capacity within the program offices.
To address our objectives we reviewed relevant documents, including
Education's business plans, information collected by Education on
states' capacity to supply data, various contracts for key pieces of
the initiative, Education's submissions to the Office of Management and
Budget (OMB) justifying the various data collections, the department's
concept of operations, and other information related to the development
of PBDMI. We also interviewed Education officials overseeing PBDMI,
officials from most of the participating program offices, and key
stakeholders in PBDMI, including a standards-setting organization, an
advocacy group, and contractors, to obtain their perspectives on the
progress of the initiative and to verify the information we reviewed.
Finally, we surveyed 52 state data coordinators, including the District
of Columbia and Puerto Rico, about their experiences with PBDMI, and we
received 50 responses. We performed this work between April 2004 and
September 2005 in accordance with generally accepted government
auditing standards. See appendix I for additional information on our
scope and methodology.
Results in Brief:
Through its PBDMI, Education officials have said that they have
identified and defined much of the data to be collected under a unified
data collection system. To determine what data will be collected,
project officials have engaged in an ambitious effort with the program
offices to identify data needed for program administration and
oversight. They also developed performance-related data that would meet
those needs, particularly for evaluating the effectiveness of federal
programs. They further worked to develop common definitions and
eliminate redundancy for data that would be collected through the
system. The end result of this work was a body of performance-based
data elements designed to better position the department to monitor the
performance of its elementary and secondary education programs.
However, officials responsible for the initiative told us that they
were unable to resolve all data differences among Education's program
offices, given the traditional, diffused control of information
collected throughout the department. PBDMI officials estimated that the
majority of the work to define these data elements had been completed,
although we found that they did not develop baseline data which would
allow them to track the full extent of their progress. We were also
told that these hard to resolve differences that remain would
ultimately be settled at higher levels within the department, but the
department has no formally agreed upon process for how or when such
decisions would occur.
PBDMI officials have conducted extensive outreach to the states to help
them meet Education's data request and in some cases upgrade their
collection and submission systems; but after 2 years of testing, most
states have not, for a variety of reasons, been able to provide
Education with enough reliable data to proceed with the initiative.
This outreach involved two rounds of site visits to all the
participating states to confer about data elements developed with the
programs and offer technical assistance, and $100,000 in grants to most
states to help offset their costs. In addition, Education sponsored
regional conferences and developed a call center to help states prepare
and submit data. The department's activities were focused largely on
state-level agencies, but did involve some educational organizations.
State data providers responding to our survey expressed general
satisfaction with the department's outreach, but about 75 percent
nevertheless predicted that the burden of collecting and reporting data
would increase or remain the same once PBDMI was completed. Many states
also expressed doubts that PBDMI could enhance their analytic
abilities. Only about 20 percent of states expected PBDMI to improve or
greatly improve their analytic capacity. Despite the extensive
outreach, most states were not able to produce enough data during test
submissions in 2002-2003 and again in 2003-2004 for the department to
validate its quality and consider phasing out its standing collection
systems. Thus, the department has decided to keep the latter collection
phase open longer. According to PBDMI officials, some states lack the
technical capacity to collect and report the requested data
electronically and others need to modify their existing processes to
meet the new specifications. Still others wanted clarification from the
department for data definitions. State data providers also reported
having competing demands for their time and attention, given other
federal initiatives.
Education officials have decided to proceed with PBDMI's implementation
despite a shortage of data, other delays, and reservations among a few
program offices; however, they do not have a specific plan for
addressing these obstacles. Currently the department expects to
complete its systems development efforts, which includes the full
implementation of its data analysis and reporting system by the spring
2006--1 year later that its initial completion date--primarily due to
the lack of state data and the failure of some of Education's
contractors to meet scheduled delivery dates. To the degree that it has
been able to proceed, the department has begun developing a set of
quality checks, although a few program offices expressed concern about
their adequacy for maintaining the value of the data. Meanwhile,
Education officials have said they are developing strategies to address
these obstacles, including exempting states from certain reporting
requirements, but they had no specific plan for providing further
assistance to the states or for meeting state expectations for phasing
out multiple data collections.
We are making recommendations to Education to improve its planning and
decision-making processes supporting PBDMI. In responding to a draft of
this document, Education's Assistant Secretary for the Office of
Planning, Evaluation and Policy Development provided written comments
on a draft of this report. In its comments, Education agreed with our
findings and recommendations. Copies of the written comments are in
appendix II.
Background:
The Department of Education annually administers data collections to
gather information from states about elementary and secondary education
programs receiving federal assistance. When it administers a data
collection, Education, like most federal agencies, is required to
follow the provisions of the Paperwork Reduction Act (PRA)[Footnote 1]
in order to maximize the utility of information to the federal agency
and minimize the level of burden incurred by the states and agencies
from whom it solicits the information. Traditionally, the department's
program offices, which have responsibility for the administration and
oversight of federal education programs, have developed and operated
similar data collections independent of one another, in a continuous
year-round process. In addition, much of the data requested from states
has been focused on compliance and procedural matters, and overlooked
performance and the impact of programs in the classroom. Moreover, the
collection of this data has been complex and prone to error, given that
it typically passes from about 94,000 public schools to more than
14,000 school districts and then to state education agencies before
Education receives it.
Collecting data can be both time-intensive and costly. Education
estimated that in 2004, for example, that states spent approximately
45,000 hours and nearly $1.2 million responding to the department's
requests for certain elementary and secondary education data. (See fig.
1.) Data collections are costly for Education also. Over $5 million was
spent in 2004 administering certain data collections that included
allocating federal funds for both the staff to administer the
collections and in many instances for contractors to analyze these
data.
Figure 1: Time and Money--Estimated Annual State Burden Hours and Costs
for Select Elementary and Secondary Education Data Collections:
[See PDF for image]
[A] State Education Agencies (SEA).
[B] Individuals with Disabilities Education Act (IDEA).
[C] Free and appropriate public education (FAPE).
Note: Figure includes burden estimates for ongoing collections for
which data were available.
[End of figure]
Initiated in 2002, the Education's PBDMI has four goals: to improve the
quality of the data Education collects about elementary and secondary
education in terms of accuracy, consistency, and timeliness; to reduce
the burden that states incur in reporting data to the department; to
improve the focus of data analysis on program performance; and to
improve Education's data-sharing relationship with the states. While
this initiative is not the department's first attempt to overhaul the
way it collects data, it nonetheless represents a fundamental change to
its data management in that it is agencywide as opposed to program
specific.[Footnote 2] As envisioned, the new collection would
consolidate 16 separate collections heretofore conducted by seven
program offices.[Footnote 3] Given the additional reporting effort that
development and testing of the system would require of states,
Education sought and received OMB approval to collect data from the
states through PBDMI.[Footnote 4] (See table 1 for a list of the
separate collections the PBDMI is designed to supplant.)
Table 1: Data Collections That Could Be Reduced or Eliminated as a
Result of PBDMI:
Program office: Office of Elementary and Secondary Education (OESE);
Data collections:
1. Consolidated State Performance Report;
2. State Data Collection for the McKinney-Vento Homeless Assistance
Act;
3. Elementary and Secondary Education Act Title I, Part C Migrant Child
Count Report.
Program office: Institute of Education Science (IES);
Data collections: 4. Common Core of Data Surveys.
Program office: Office of English Language Acquisition (OELA);
Data collections: 5. Title III Biennial Evaluation Report Required of
State Education Agencies Regarding Activities under the No Child Left
Behind Act of 2001;
6. Biennial Report Form for the Emergency Immigration Education
Program.
Program office: Office of Special Education and Rehabilitative Services
(OSERS);
Data collections: 7. Report of Children with Disabilities Receiving
Special Education under Part B of the Individuals with Disabilities
Education Act;
8. Part B, Individuals with Disabilities Act Implementation of "Free
and Appropriate Public Education" (FAPE) Requirements;
9. Personnel Employed to Provide Special Education and Related Services
for Children with Disabilities;
10. Report of Children with Disabilities Exiting Special Education
During the School Year;
11. Report of Children with Disabilities Unilaterally Removed or
Suspended/Expelled for More than 10 Days;
12. Part B of the Individuals with Disabilities Education Act Annual
Performance Report;
13. Consolidated Data Collection on Students with Disabilities [A].
Program office: Office of Safe and Drug-free Schools (OSDFS);
Data collections:
14. Gun-Free Schools Act Report.
Program office: Office for Civil Rights (OCR);
Data collections:
15. Elementary and Secondary School Civil Rights Compliance Report[ A].
Program office: Office of Vocational and Adult Education (OVAE);
Data collections:
16. Carl D. Perkins Vocational and Technical Act Annual Performance
Report.
Source: GAO analysis of Department of Education documents.
[A] Collections that have been replaced by PBDMI.
[End of table]
In addition to defining the information to be collected, the initiative
involves the development of a Web-based, data exchange network that
will provide states and others with the ability to submit school-based
data into one unified system to be stored in a data repository. The
network will comprise three separate, but interrelated systems--the
first system, the submission system, developed in late 2004, is used to
collect data from states, check data for quality, and store the data in
the data repository. The second system, the survey tool, which was also
developed in 2004, enables Education to collect supplemental data from
states and others that is also stored in the data repository. The third
system, the data analysis and reporting system, which is not yet
operational, will allow users (i.e., program office staff and the
public) to among other things, query the data repository to analyze
retrieved data and generate ad hoc reports. Education envisions that
states and school districts would be able to use the data to assess
their own program performance while also providing an opportunity for
them to verify the quality of data submitted through the system. Figure
2 depicts the system design for the data network.
Figure 2: Proposed Design for PBDMI's Web-based Network:
[See PDF for image]
[End of figure]
Education had originally planned to have all components of the data
exchange network fully operational in the spring of 2005 following the
completion of key activities, such as (1) defining the data to be
collected through in-depth consultations with department program
offices and with state data providers,(2) populating the database with
school-based data submitted by the states so that the quality of the
stored data can be checked, and (3) training program staff on how to
use the new network.
PBDMI's efforts to define what data were to be collected included
forging agreements among Education's individual program offices about
which data would be essential to administration and oversight,
particularly as performance indicators, and also developing common
definitions for those elements that had been redundant. As a
collaborative project, this involved developing consensus and receiving
feedback from many parties-program offices, state policymakers and data
providers, and organizations that develop data standards in the field
of Education. Within the department, the office responsible for the day-
to-day work of the project and for ensuring its success is the
Strategic Accountability Service, which also has responsibility for
developing and disseminating agencywide performance indicators.
However, a number of other offices and boards within the department
have been charged with providing oversight and guidance: a steering
committee convened to share information on the development of the
initiative consisting of the PBDMI managers and other senior officials
within the participating program offices, the Chief Information Officer
(CIO), a data information working group, and Education's investment
review board. The data information working group, which is headed by
Education's CIO, has responsibility for ensuring the consistency and
quality of new data collections and for facilitating the integration
and sharing of information between program offices. The department's
investment review board has overall responsibility for reviewing and
approving and prioritizing department investments in technology,
including the new network. As voluntary participants, stakeholders such
as data coordinators from each of the 50 state education agencies, the
District of Columbia, and Puerto Rico were provided with opportunities
to give their input and feedback on the development of the initiative.
The Education Information Advisory Committee established by Council for
Chief State School Officers facilitates this exchange. Figure 3 depicts
the various groups involved in the initiative.
Figure 3: Illustration of Key Actors Involved in Development of the
PBDMI:
[See PDF for image]
[End of figure]
Once departmental data requirements were identified, Education planned
a series of data collections to be followed by extensive testing of the
quality of that data by the program offices. Specifically, Education
planned to have states submit the newly defined data for the 2002-2003
and 2003-2004 school years. (States would voluntarily make these
submissions to PBDMI while also maintaining their current multiple
reporting obligations under Education's program offices.) In
conjunction with the program offices, PBDMI officials then anticipated
validating and verifying the quality of the new data submitted using a
number of checks and evaluations. Also at this time the development of
the system that staff would use to analyze data and generate reports
was to be finalized. Once these activities were completed, the program
offices were to assess whether the new system would be an adequate
substitute for their existing data collections.
Education has projected that it would spend just over $30 million
through 2005 and initial estimates indicate that the data network will
cost--beginning in 2006--just over $4 million annually to maintain. See
figure 4 for project time frames and projected costs through 2009.
Figure 4: Fiscal Year 2002-2009 Funding for Education's Data Management
Initiative, including Key Activities Planned for Project Initiation
through Implementation:
[See PDF for image]
[A] Includes $2 million to develop a Web-based survey tool designed to
collect supplemental data from schools, districts, and states.
[End of figure]
Education Has Made Progress Defining Data to Be Collected under a
Consolidated System, but Project Officials are Unable to Reconcile All
Differences:
Education officials spearheading PBDMI told us they have made progress
defining the data to be collected. To do this, project officials worked
with the program offices to identify their existing data needs. They
also worked with program offices to translate these needs into
performance-related data, such as math and reading achievement scores
for different groups of students. Officials told us they had eliminated
data elements collected by the program offices that are more indicative
of process than performance. PBDMI officials encouraged program offices
to identify performance-related data by using requirements specified in
laws such as the No Child Left Behind Act and using the goals in the
department's strategic plans.
PBDMI officials also worked with the program offices to reach agreement
on common definitions for the data elements selected and to eliminate
redundancy. For example, some programs needed information on charter
schools, and PBDMI officials coordinated efforts within the department
to develop one standard definition for them. The end result of these
efforts is a unified body of data elements that includes definitions
for each of the data elements and identifies the program with primary
stewardship over decisions about that element. According to one
department official managing the initiative, this collection will
improve the quality of the data by assuring more consistency in what
states provide.
Although PBDMI officials reported progress in identifying performance-
related data and establishing common data definitions, project
officials have not fully documented these achievements by establishing
a baseline and thus cannot be certain of the full extent of the
progress made toward achieving their goal to enhance the department's
focus on outcomes and accountability. For example, while PBDMI
officials were able to provide a list of 161 data elements focused on
performance they were unable to provide us with a comprehensive list of
"process-oriented" elements that had been eliminated. Similarly, while
PBDMI managers reported that the program offices had agreed to
definitions for the bulk of the data elements--one official estimated
that they reached agreement for about 90 percent of the data--they
could not provide us with a complete list of redundant elements that
had been eliminated or those that remain because they had not tracked
them.
While PBDMI officials could not provide a full list of disputed data
elements, they reported that some differences still remain among
program offices. Although PBDMI officials encouraged the use of
strategic plans and statutory requirements to justify the selection of
performance-based data, they told us that program offices had final say
over what data to collect. For example, one office uses similar
although somewhat broader criteria that allow it to collect "data that
can be reliably obtained from states or that Education has a documented
need for."[Footnote 5] Additionally, according to initiative officials,
some differences remain due to differences in legislative requirements
for the particular programs, while others resulted from preferences of
some offices to continue using the same definitions as in the past.
Officials responsible for carrying out the PBDMI told us they were
unable to reconcile all differences. Officials told us they were
working with the program offices to reach agreements, but said the
programs maintain primary control for defining their data needs and
would make final decisions. Additionally, we were told by Education's
CIO, who is required to review all data collections and who has a
primary role within the Data Information Working Group, that this
office does not have a role in resolving data disputes between program
offices in order to ensure uniformity. However, an official also said
that any differences that could not be resolved between the program
offices would ultimately be arbitrated at the assistant secretary level
within Education.
PBDMI Officials Have Worked Extensively with States on Data Preparation
and Submissions, but Most States Cannot Produce the Requested Data:
PBDMI officials have conducted extensive outreach to the states to help
unify their data definitions and upgrade their collection and
submission systems. State data providers responding to our survey
expressed general satisfaction with the department's outreach. However,
the majority thought that the burden of data collection and reporting
would either increase or remain the same with implementation of the
PBDMI. In addition, less than half expected the initiative to improve
their ability to conduct their own in-state analyses only somewhat.
Despite the extensive outreach, the states were not able to produce
enough data during test submissions in 2002-2003 and again in 2003-2004
for the department to validate its quality and consider phasing out its
standing collection systems.
Education Has Conducted Extensive Outreach to States to Improve Data
Quality:
In order to ensure that states could meet Education's requests for
quality data required as part of PBDMI, officials conducted extensive
outreach to state agencies, their data providers, and to data standards
organizations. After Education developed its body of data elements, it
consulted in 2002 with a task force consisting of a small number of
state data providers to advise the department on the availability of
the data it intended to collect. The department then conducted site
visits beginning in April 2003 to 50 states, the District of Columbia,
and Puerto Rico to obtain feedback on the ability of states to provide
needed data and to prepare for testing the states' ability to submit
data. Education officials said they also made $50,000 grants to all 52
states to offset costs of overhauling information systems or obtaining
additional staff. At the culmination of these visits, Education
originally planned for states to transmit 2002-2003 school year data
that could be tested for quality.
However, Education scaled back the scope of this first data collection
after recognizing that states would not, as yet, be able to offer
certain types of data, such as data needed to meet requirements of the
NCLBA. Consequently, Education delayed its plans to assess the quality
of the data states submitted and focused instead on the ability of
states to electronically transmit as much PBDMI data as they could to
the department. Also, Education decided to remove from PBDMI's
prospective collection some data elements that states reported were not
available at that time. Under this transmission pilot test, 50 states,
including the District of Columbia and Puerto Rico, were able to submit
some data to Education demonstrating that PBDMI was technically
feasible.
After establishing this technical feasibility, Education began
preparing in 2004 for its data collection of 2003-2004 school year by
providing additional outreach to the states. Project officials
conducted a second round of site visits beginning in April and provided
further guidance to help states align their data definitions with PBDMI
standards. By aligning definitions with PBDMI, Education attempted to
minimize possible confusion about what data to submit and when, further
assisting the department's efforts to improve data quality. Department
officials have said that establishing a unified body of data elements
across the department and states--so that all involved parties use the
same "language" when analyzing and sharing data[Footnote 6]--is a
priority. Education officials attribute the lack of quality in the data
it currently collects from states and others to a variety of reasons,
such as the lack of common data definitions that developed over time in
response to the specific information needs of the program offices and
data requirements arising at the state level.
Officials with the initiative also conducted a limited number of
quality assessments of state information systems to identify better
ways of collecting and reporting data to the department. To serve
states on a broader scale, Education conducted regional meetings,
providing them with updates and feedback on the progress of the
initiative. Officials also established a call center to answer states'
questions about the data to be submitted. Most states also received
another $50,000 in grants for their continued participation in the
initiative.[Footnote 7] Education began collecting 2003-2004 school
year data in November 2004.
To increase the likelihood that its definitions would be adopted by
states and other data providers, PBDMI officials also collaborated with
advocacy groups that establish data and influence the development of
technical standards. For example, PBDMI officials contracted with the
Council of Chief State School Officers to coordinate PBDMI conferences,
help states prepare and submit data, and provide feedback as PBDMI
developed data definitions. Education also collaborated with the
Schools Interoperability Framework, a group that develops data-sharing
standards and software primarily designed for schools and districts. By
working with the Schools Framework, Education officials said they could
improve data quality by increasing the likelihood that departmental
definitions and other standards would be incorporated into software
used by schools and districts. This interaction with the Schools
Framework is Education's primary attempt to deal with the long-standing
problem of poor data provided by schools and districts.[Footnote 8]
(See table 2 for a list of some of Education's outreach activities.)
Table 2: Education's Outreach Activities to Improve Data Quality under
the PBDMI:
Activity: Site visits (1st round);
Description: Met with states in summer 2003 to discuss data definitions
and availability for test data collection in November 2003;
Purpose: To introduce standards, encourage consistency, and assess data
availability and technical capacity of state information systems.
Activity: Site visits (2nd round);
Description: Met with states in spring 2004 to confirm data definitions
and availability for initial data collection in November 2004.
Activity: $50,000 PBDMI Participation Grants;
Description: Officials report awarding 52 state education agencies with
funds in 2003-2004; and grants to 46 states received funds in 2004-
2005;
Purpose: To obtain state buy-in and offset costs.
Activity: State taskforce;
Description: This advisory group, made up of a small number of states,
has provided input concerning available data that can be submitted by
states and how collections could yield better quality data;
Purpose: To solicit initial state input on data issues such as
availability and capacity.
Activity: Technical assistance;
Description: Call center available to all states, meetings,
conferences, and data quality assessments provided to 10 volunteer
states;
Purpose: To provide states with answers to questions, updates on the
status of the initiative, and information to help improve data systems.
Activity: Outreach to software vendors;
Description: Education coordinates with key standard setting
organizations such as the Schools Framework;
Purpose: To work collaboratively to develop educational data standards.
Source: Department of Education.
[End of table]
Most States Expressed Satisfaction with Education's Outreach, but Had
Mixed Views on PBDMI's Potential Benefits:
States were generally satisfied with Education's outreach activities.
(See table 3.) Most state data providers--72 percent--rated Education's
site visits effective in improving the partnership with the states. One
state data provider characterized his exchanges with the department as
open and non-defensive, and further reported that the department had
been responsive. More than half rated as effective or very effective
Education's technical assistance (57 percent) and regional meetings (52
percent). While most states thought Education's activities to improve
its partnership with states were effective, some suggested areas for
improvements. For example, 72 percent thought the site visits provided
only some or little information on successes achieved in other states.
Table 3: State Survey Responses on Education's Outreach Activities
through PBDMI:
[See PDF for image]
Source: GAO, Survey of States on the Department of Education's
Performance Based Data Management Initiative (PBDMI).
[End of table]
In their survey responses half of the states expressed the view that
reducing their reporting burden was the most important PBDMI goal;
however, fewer than a third of the states said they believe the
initiative will do so. (See table 4.) Some states emphasized their
burden had increased in the short term as they continued dual reporting
in order to meet the still ongoing data collection requirements of the
program offices. Three states reported to us their cost estimates of
systems development projects needed to support PBDMI, which ranged from
approximately $120,000 to as much as $5 million. Moreover, about 75
percent of the states reported that they thought the burden to collect
data would remain the same or increase once PBDMI was implemented. Some
state respondents expressed the opinion that until there is a firm
commitment by Education to halt multiple data collections their
reporting burden would not likely lessen. "We are asked from the
federal government for more and more information—. [which] opens the
flood gate for more and more reporting," noted one official, adding
that it is currently "hard to see the benefit at this time.":
Table 4: Percentage of States Reporting Which Goals Were Most
Important, Attainable, and Difficult to Achieve:
PBDMI goals: Reducing collecting and reporting burden;
Most important goal: 50%;
Most attainable goal: 30%;
Most difficult goal to achieve: 36%.
PBDMI goals: Improving data quality;
Most important goal: 26;
Most attainable goal: 16;
Most difficult goal to achieve: 30.
PBDMI goals: Improving the partnership with states based on common data
standards;
Most important goal: 12;
Most attainable goal: 24;
Most difficult goal to achieve: 14.
PBDMI goals: Focusing on outcomes;
Most important goal: 0;
Most attainable goal: 16;
Most difficult goal to achieve: 8.
PBDMI goals: Not sure;
Most important goal: 12;
Most attainable goal: 14;
Most difficult goal to achieve: 12.
Source: GAO, Survey of States on the Department of Education's
Performance Based Data Management Initiative (PBDMI).
[End of table]
Some states also had reservations about the benefits of PBDMI for
evaluation. One respondent cautioned, for example, that support within
his state had weakened because of the lack of perceived benefits. Only
about 20 percent of states expected PBDMI to improve or greatly improve
their analytic capacity--that is the ability to meet their own state
reporting requirements, analyze program effectiveness, analyze student
outcomes, and to compare outcomes within states. Their reasons varied.
For example, five states reported that they would continue to use their
own systems. A few elaborated that their own information systems allow
more detailed analyses of state performance than the information to be
collected through PBDMI. Additionally, an almost equal number of states
saw PBDMI as an effective tool to inform stakeholders as not. Table 5
lists the extent to which state data providers expect PBDMI to enhance
their analytical capacity in a variety of areas.
Table 5: Percentage of States Reporting the Extent to Which PBDMI Would
Improve Their Analytical Capacity:
Inform stakeholders;
Very great/great extent: 35%;
Little to no/some extent: 33%.
Meet state reporting requirements;
Very great/great extent: 22;
Little to no/some extent: 59.
Analyze student outcomes;
Very great/great extent: 22;
Little to no/some extent: 53.
Make budgetary decisions;
Very great/great extent: 16;
Little to no/some extent: 61.
Analyze program effectiveness;
Very great/great extent: 20;
Little to no/some extent: 45.
Compare outcomes within states;
Very great/great extent: 20;
Little to no/some extent: 51.
Source: GAO, Survey of States on the Department of Education's
Performance Based Data Management Initiative (PBDMI).
[End of table]
Many States Are Not Prepared to Meet Education's Data Requests:
As of June 3, 2005, only 9 states had submitted more than half of the
requested 2003-2004 school year data, while 29 states had submitted
less than 20 percent (see fig. 5). Although PBDMI officials said they
will wait until August 2005 for states to submit the 2003-2004 data,
they also acknowledged that many states would not be able to provide
significant portions. The lack of state data is particularly acute in
some programmatic areas. For example, many states have been unable to
provide data on homeless and migrant students or students with limited
English proficiency. States told Education officials early in the
process that changes to state data collection processes, systems, and
definitions would be needed to provide these types of information.
Figure 5: School Year 2003-2004 Performance Data Submitted by States to
PBDMI as of June 3, 2005:
[See PDF for image]
[End of figure]
We found that there were various reasons why states could not provide
data. Some states reported that they wanted better documentation from
the department in areas such as clarifying established data definitions
and file format specifications needed to transmit data. States needed
to make major modifications to their existing data collection and
reporting processes in order to provide new information required by
PBDMI. States also reported that they would not provide certain data
elements that were inapplicable, hard to collect, or available
elsewhere. Some also reported that there was still some confusion over
multiple or unclear definitions. Department officials said that many
states had initially overestimated their capabilities and that the data
states said would be available differed greatly from what they have
produced thus far. States have also noted competing demands for their
time and resources stemming from NCLBA. Some states reported they
lacked resources, such as staff and money, to implement changes
specific to the initiative. Specifically, 56 percent of the state
survey respondents said that all or a portion of the $50,000 in grants
they received from Education were used to contract for additional
personnel, a quarter of the states said that these funds were used to
improve their information systems. Some states noted, however, that
these funds were insufficient to make changes necessary for their
participation in PBDMI.
Recognizing that obtaining state data has been problematic, Education
has recently developed a preliminary strategy for working more closely
with states to ensure that it obtains 100 percent of data from all the
states. While not finalized, Education is currently considering actions
such as issuing regulations requiring states to submit PBDMI data and
allowing those states that provide acceptable amounts of "high quality"
data under PBDMI to be exempt from existing data collections. For
example, states that submit data to PBDMI that are also currently
collected through the Consolidated State Report--one of many data
collections required under the NCLBA--would not have to submit the same
data under this data collection. Officials have also tentatively
proposed collecting data of lesser quality that are readily available
and obtaining data through other systems to supplement what has been
provided thus far. It is not clear the extent to which this proposal
would undermine efforts to improve data quality and maintain program
office buy-in. Another option under consideration at Education is to
target departmental resources, such as $25 million in grants for system
improvements from the Institute of Education Sciences, at states that
actively participate in PBDMI.
Education Is Proceeding with Implementation despite the Data Shortage
and without a Detailed Plan of Action:
Education is proceeding with efforts toward full implementation of
PBDMI--using the data for analysis and reporting--despite the limited
amount of data collected. To do so, program offices decide whether the
quality of the data (in terms of accuracy, consistency, timeliness, and
utility) collected through PBDMI meets their needs. Once program
offices validate the quality of the data, Education would begin to
phase out existing data collections. Additionally staff will be trained
on how to access and use the data collected to date. Originally
Education expected to complete all of these activities by the spring of
2004. To the degree that it has been able to proceed, the department
has developed a set of quality checks designed to ensure the accuracy
and completeness of the data states submit.
Nevertheless, two program offices, which as members of the seven
principle offices included in the initiative and have a role in
determining whether the data are accurate and complete for their
purposes, expressed concern that PBDMI's procedures to ensure data
quality may not be adequate. An official in the Office of Special
Education and Rehabilitative Services (OSERS), which has collected
almost 30 years of longitudinal data about the effectiveness of the
nation's special education programs, told us that PBDMI had been
provided with information about the nearly 200 data quality checks used
in special education collections, but was not sure that PBDMI adopted
them all. PBDMI officials said they adopted those that were universally
relevant. Further, this official expressed concern that PBDMI would not
meet its special needs. Specifically, unlike other program offices,
OSERS programs bases student assessment on age as opposed to grade
level attained. Additionally, this official was concerned about the
timeliness of the data collected through PBDMI because that office
generated a number of congressionally mandated reports at specific
times of the year. Consequently, this office plans to compare the
quality of its own data with the data collected through PBDMI.
Officials in the Office for Civil Rights also expressed similar
reservations with PBDMI's administration of its large elementary and
secondary survey of schools and districts used to assess compliance
with civil rights laws and identify trends. Historically, district
superintendents have responded to this survey in large enough numbers
allowing Education to generalize on any findings with a high degree of
confidence.[Footnote 9] However, when PBDMI administered the survey,
fewer superintendents responded and, according to the Office for Civil
Rights, PBDMI did not have a readily available plan that adequately
outlined steps needed to raise the response rate. As of June 10, 2005,
the response rate for this survey was lower than previous surveys.
Final implementation has also been hampered by delays in training and
delivery of the analysis and reporting system. Both are more than a
year behind schedule. An official responsible for overseeing the
training efforts told us they could not focus on the delays because
considerable time was spent addressing state problems submitting data
through PBDMI. The data analysis and reporting system is more than a
year behind scheduled due to the lack of data and the failure of
Education's contractor to meet its scheduled delivery of the system.
Education officials now expect to fully implement the system by March
31, 2006. In lieu of developing its data analysis and reporting system
and training, PBDMI has offered presentations of these tools as a
preview for staff to see the new system's capabilities and to keep them
apprised of the initiative's progress.
Despite the many obstacles confronting the PBDMI, Education officials
said they expect to proceed with implementation of the initiative,
albeit with some activities postponed. In August, project officials
developed a preliminary strategy designed to address the problem of
collecting data from the states, such as providing exemptions from
certain reporting requirements for some states. However, this strategy
has not been finalized, and Education has not developed a specific plan
of action for how they might (1) help states that are deficient, (2)
deal with state expectations for phasing out the multiple data
collections, or (3) meet the expectations of their own program offices.
Conclusions:
The PBDMI represents an important step forward for the Department of
Education in its efforts to monitor the performance of the nation's
elementary and secondary schools. By developing the ability to collect
data that are more accurate, timely, consistent, and focused on key
national performance indicators, Education will be much better informed
to make its many policy and programmatic decisions. The initiative, by
asking for a clearly defined set of information that is to be submitted
only one time, has the potential to substantially reduce state
reporting burden for elementary and secondary programs as well as to
help states to develop better data systems. However, PBDMI is an
ambitious and risky undertaking that requires the continued cooperation
of a number of internal and external stakeholders.
In order for PBDMI to be successful, the department must rely on states
to provide new information at a time when they are busy implementing
large new federal initiatives, such as the No Child Left Behind Act.
While some states have been able to provide significant amounts of
data, others continue to lag far behind. In order for PBDMI to be
successful, it is important for all states to submit timely, reliable,
accurate, and consistent data. Consequently, it is important for the
department to have a clear plan for addressing states with problems
providing data and to continue to provide a proper combination of
support and incentives for states to participate. By having worked
closely with the states on their collection systems, PBDMI officials
have the information they would need to develop a plan of action to
help move them forward.
Because PBDMI represents a significant change in the way the Department
of Education conducts business, it can only be accompanied effectively
and efficiently by a change in management practices. However, program
offices still retain much discretion over what data they will collect,
how they will define it, and whether or not PBDMI's data will meet
their needs. While it is the initiative's responsibility to make sure
it collects data that meets the program offices' requirements, PBDMI is
also responsible for developing a data collection system focused on
program performance and quality data. To the extent that programmatic
differences, such as those over data definitions, inhibit PBDMI's goals
there should be a clear process for reconciling those differences. If
PBDMI truly represents a new way of doing business, Education should be
able to ensure that its organizational units go along. It is difficult
to see PBDMI achieving its full potential without a clear process for
furthering the initiative's goals.
Fundamental to any large, complex effort's success is a well thought
out plan that tracks its progress against a set of clearly defined and
measurable goals. PBDMI has not put in place such a planning and
tracking system. State governments and Education's program offices have
devoted much time, effort, and money participating in PBDMI with the
idea that they would see benefits as a result. A lack of demonstrated
progress and benefits potentially erodes state support, undermining the
viability of this important initiative. Some states are already
beginning to lose sight of the potential benefits of PBDMI. As the
department goes past its original completion deadline, it is important
for it to lay out a clear plan for how it will now proceed.
Recommendations for Executive Action:
To address the issues we have identified with regard to planning,
decision-making, and improving data quality, we recommend that the
Secretary of Education develop:
* a strategy to help states improve their ability to provide quality
data given the challenges that many states face in providing data;
* a clear process for reconciling differences between the program
offices and the PBDMI oversight office to ensure that decisions
critical to the success of PBDMI are made; and:
* a clear plan for completing final aspects of PBDMI, including
specific time frames and indicators of progress toward the initiative's
goals.
Agency Comments and Our Evaluation:
We received written comments on a draft of this report from the
Department of Education. Education agreed with our findings and
recommendations and stated that it has devoted additional resources to
the initiative and plan to issue a detailed project plan that outlines
the steps needed to complete the initiative. These comments are
reprinted in appendix II.
Education also provided technical corrections and comments that we
incorporated where appropriate.
We are sending copies of this report to the Secretary of Education, the
Office of Strategic Accountability Services, the Director of the Office
of Management and Budget, and appropriate congressional committees.
Copies will also be made available to other interested parties upon
request. Additional copies can be obtained at no cost from our Web site
at www.gao.gov.
If you or your staff should have any questions, please call me at 415-
904-2272 or bellisd@gao.gov. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last
page of this report.
GAO staff who made major contributions to this report are listed in
appendix III.
Signed by:
David Bellis:
Director, Education, Workforce and Income Security Issues:
[End of section]
Appendix I: Scope and Methodology:
The objective of our review of the Performance Based Data Management
Initiative (PBDMI) was to assess the progress Education has made in its
implementation of the initiative, particularly with regard to (1)
defining what performance-related data it will collect from states on
behalf of the program offices, (2) assisting states in their efforts to
submit quality information, and (3) utilizing performance-related data
to provide enhanced analytic capacity within the program offices. We
conducted our review between April 2004 and September 2005 in
accordance with generally accepted government auditing standards.
Overall Approach:
To assess the department's progress in each of these areas, we reviewed
documents relating to the implementation of the initiative, relevant
laws, and information provided by the office responsible for PBDMI--the
Strategic Accountability Service (SAS) and others. We interviewed key
staff responsible for the initiative as well as officials in each of
the offices that are participating in PBDMI. We also interviewed senior-
level Education officials to determine their role in the implementation
of PBDMI. To gain insight into state perspectives on the initiative, we
administered a Web-based survey to state officials responsible for
providing these data to Education. We received responses from 50 states
including Puerto Rico. We also interviewed a variety of external
stakeholders, a data standards organization, and three contractors
involved in the initiative, including an official from the Council of
Chief State School Officers. We also reviewed previously issued reports
by Education's Office of the Inspector General (IG) as well as GAO
reports and testimonies.
In addition to interviewing departmental officials, we also reviewed
documentation on the initiative to gain a better understanding of what
actions Education was undertaking to implement the goals of the
initiative, including its data quality contract, data dictionary, its
business plans as well as justification reports to the Office of
Management and Budget (OMB) required under the Paperwork Reduction Act
to collect data. We also reviewed summary information about state
performance data that was obtained as a result of site visits to states
conducted in 2004 in order to analyze what data was obtained from
states as a result of their efforts.
Education provided information on states' submission of requested data
elements to PBDMI as of June 3, 2005. States were expected to provide
data for 64 data elements ranging from dropout rates, student
performance on reading, science, and writing assessments, teacher
certification, and many others. For each of these elements, Education
determined whether each state had submitted the information, had not
submitted the data, or did not collect the information. We incorporated
into our report Education's calculated percentages of elements
submitted for each state. We determined that these data were sufficient
for the purposes of this engagement.
In order to document the burden hours associated with certain
elementary and secondary data collections, we accessed 14 data
collection justifications authored by each of the department's program
offices and submitted to the chief information officer. These reports
had received OMB approval or were seeking approval to collect data from
states and others. We talked with an official responsible for
maintaining these documents at the department's Web site to verify that
these were the most recent data available for analysis. From each
document we obtained the estimated state burden hours and costs and
federal administrative costs associated with each data collection. Each
estimate was based on a formula that we adjusted to reflect these costs
for the 52 states participating in the initiative. In some instances
where an average was used, we assumed that the 52 states were similar
in characteristics to the overall population of states included in
Education's estimates. However, we did not find it feasible to prorate
the formulas for the federal administrative costs (based on 52 states)
for each of the collections. A statistician verified each of the
calculated estimates for accuracy.
We also surveyed all 52 state data coordinators using a Web-based
survey instrument in order to obtain their perspectives on various
aspects of the initiative. Our survey instrument was developed based on
information obtained during interviews with state data coordinators in
Pennsylvania, Virginia, Washington, and Oregon. Additionally, other
internal stakeholders specializing in technology and education were
asked to review and comment on our draft survey instrument. The survey
was pre-tested with Wyoming, North Carolina, and Illinois to determine
if the questions were clear and unbiased and whether the terms were
accurate and precise. We included these three states in our pretests
because they varied in size and technical capacity for data
transmission as determined by an earlier Education survey. Based on
their comments, we refined the questionnaire as appropriate.
Our final survey instrument asked a combination of questions that
allowed for closed-ended as well as open-ended responses and included
questions about state perspectives on PBDMI's ability to achieve its
goals. The survey was conducted using self-administered electronic
questionnaire posted on the Internet. We sent e-mail notifications
about the upcoming survey to all 52 state data coordinators (50 states,
the District of Columbia, and Puerto Rico) on November 15, 2004, and
activated the survey shortly thereafter. Each potential respondent was
provided a unique password and username by e-mail to limit
participation to members of the target population. To encourage
respondents to complete the questionnaire, we sent an e-mail message to
prompt each non-respondent approximately 2 weeks after the survey was
activated and followed up by e-mail or phone with each non-respondent
several times thereafter. We closed the survey on January 21, 2005,
after the 50th respondent had replied.
Because this was not a sample survey, there are no sampling errors.
However, the practical difficulties of conducting any survey may
introduce errors, commonly referred to as non-sampling errors. For
example, difficulties in how a particular question is interpreted, in
the sources of information that are available to respondents, or in how
the data are entered into a database or were analyzed can introduce
unwanted variability into the survey results. We took steps in the
development of the survey instrument, the data collection, and the data
analysis to minimize these non-sampling errors. For example, a survey
specialist designed the survey instrument in collaboration with GAO
staff with subject matter expertise. Then, as stated earlier, it was
pre-tested to ensure that the questions were clear, unbiased, and
accurate. When the data were analyzed, a second, independent analyst
checked all computer programs. Because this was a Web-based survey,
respondents entered their answers directly into the electronic
questionnaire, eliminating the need to have the data keyed into a
database, thus removing an additional source of error. [Footnote 10]
[End of section]
Appendix II: Comments from the Department of Education:
UNITED STATES DEPARTMENT OF EDUCATION:
ASSISTANT SECRETARY:
OFFICE OF PLANNING, EVALUATION AND POLICY DEVELOPMENT:
October 17, 2005:
David D. Bellis:
Director, Education, Workforce, and Income Security Issues:
United States Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Mr. Bellis:
Thank you for providing the Department of Education with a draft copy
of the U.S. Government Accountability Office's report entitled,
"Education's Data Management Initiative: Significant Progress Made but
Better Planning Needed to Accomplish Project Goals" (GAO-06-6). We
agree with your recommendations and value your observations on this
continuing initiative.
The Department's performance-based data management initiative is
essential to the Department's efforts to improve the use of data to
inform policy making and program management, to increase the focus on
student achievement outcomes rather than on process, to reduce the
burden on States of reporting data to the Department, and to improve
the accuracy, timeliness, and utility of data collected. As you note in
the report, the initiative is an ambitious undertaking, and we
recognize the important challenges to achieving our goals.
You recommend that the Department develop 1) a strategy to help States
improve their capability to provide quality data, 2) a process within
the Department to resolve critical, outstanding issues, and 3) a clear
plan for completing final aspects of the initiative. We have also
identified these as priority focus areas and began to address them
shortly after I joined the Department this summer. Since that time, we
have committed additional leadership and financial resources to the
initiative, among other actions, and have begun developing the detailed
project plan needed to successfully complete the initiative. We will
include a full description of planned actions in our corrective action
plan.
Sincerely,
Signed by:
Tom Luce:
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
David Bellis (415) 904-2272:
Acknowledgments:
In addition to the contact named above the following individuals made
important contributions to this report: Bryon Gordon, Assistant
Director; Carla Craddock, Analyst-in-Charge; Susan Bernstein; David
Dornisch; Mary Dorsey; Kimberly Gianopoulos; Brandon Haller; Stuart
Kaufman; Jonathan McMurray; Valerie Melvin; James Rebbe; Gloria
Hernandez Saunders; Kimberly Siegel; Michelle Verbrugge; and Elias
Walsh.
FOOTNOTES
[1] The PRA was originally enacted in 1980 and most recently
reauthorized and amended in 1995 (Pub. L. No. 104-13, May 22, 1995).
Generally, the law requires each agency's chief information officer
(CIO) to review program offices' proposed collections to ensure that
they meet PRA standards before submission to the Office of Management
and Budget (OMB) for its approval. See also GAO, Paperwork Reduction
Act: New Approach May be Needed to Reduce Government Burden on Public,
GAO-05-424 (Washington, D.C.: May 20, 2005). The scope of this report
did not include a review of Education's compliance with the PRA.
[2] One earlier attempt, known as the Integrated Performance
Benchmarking System, was a two-state demonstration project designed to
consolidate department reporting requirements, but was terminated in
2000 without an assessment of its feasibility.
[3] Two collections that were formerly administered by the offices for
Civil Rights and Special Education and Rehabilitative Services have
already been subsumed into PBDMI.
[4] This approval will expire on September 30, 2005. Currently,
Education is seeking approval for further PBDMI data collections
beginning in 2006 through 2008. OMB's approval of this extension of
PBDMI data collection efforts was pending as of the end of August 2005.
Additionally, any subsequent data collections would also be subject to
the PRA process, including CIO review and OMB approval.
[5] Office of Elementary and Secondary Education, "Criteria for
Inclusion of Data Elements in the Consolidated State Report," 2005.
[6] Some of the work to establish a consistent standard has been
ongoing throughout the department prior to the development and
implementation of PBDMI such as the work undertaken by Education's
National Center on Education Statistics to establish a departmentwide
data dictionary. PBDMI has a role in contributing to some of these
other efforts.
[7] According to department officials, all 50 states, the District of
Columbia, and Puerto Rico accepted grants in 2003; and in 2004, 46
states acquired another $50,000. The grants were awarded solely on the
basis of state participation in PBDMI, and states were allowed wide
latitude in their usage.
[8] Past reports issued by GAO (published jointly with other education
agencies) and Education's Inspector General (IG) document that
inadequate data quality practices by schools and districts have
adversely affected the states' ability to produce quality data. In
2002, GAO and Education's IG reported that states had problems entering
accurate data and lacked sufficient supervisory review procedures to
check data received from schools and districts. GAO et al., A Joint
Audit Report on the Status of State Student Assessment Systems and the
Quality of Title I Accountability Data, SAO-02-064, (Austin, Tex.:
2002). OIG, Department of Education, Improving Title I Data Integrity
for Schools Identified for Improvement, ED-OIG/A03-B0025 (Philadelphia,
Pa.) March, 2002.
[9] Officials have told us that historically 97 percent or more of
randomly selected school districts have responded to this survey.
[10] Source: GAO Intranet, ARM Guidance, "Evaluating and Reporting of
Non-sampling Errors in Surveys."
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
