Title I
Education Needs to Monitor States' Scoring of Assessments Gao ID: GAO-02-393 April 1, 2002Concerned that Title I of the Elementary and Secondary Education Act (ESEA) had not significantly improving the educational achievements of children at risk, Congress mandated major changes in 1994. States were required to adopt or develop challenging curriculum content and performance standards, assessments aligned with content standards, and accountability systems to measure progress in raising student achievement. In return, states were given greater flexibility in the use of Title I and other federal funds. The No Child Left Behind Act of 2001 augments the assessment and accountability requirements that states must implement and increases the stakes for schools that fail to make adequate progress. The 1994 legislation required states to comply with the requirements by January 2001 but allowed the Department of Education to extend that deadline. Education has granted waivers to 30 states to give them more time to meet all requirements. If states fail to meet the extended timeliness, they are subject to the withholding of some Title I administrative funds. Title I directors indicated that a state's ability to meet the 1994 requirements improved when both state leaders and state agency staff made compliance a priority and coordinated with one another. Most directors said that inadequate funding hindered compliance. Many of the states reported taking action to ensure that Title I assessments were scored accurately, that any exemptions for students with limited English proficiency were justified, and students with disabilities were receiving appropriate testing accommodations. As of March 2002, 17 states had complied with the 1994 assessment requirements; 35 states had not.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-02-393, Title I: Education Needs to Monitor States' Scoring of Assessments
This is the accessible text file for GAO report number GAO-02-393
entitled 'Title I: Education Needs to Monitor States‘ Scoring of
Assessments' which was released on April 1, 2002.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
United States General Accounting Office:
GAO:
Report to the Secretary of Education:
April 2002:
Title I:
Education Needs to Monitor States‘ Scoring of Assessments:
GAO-02-393:
Contents:
Letter:
Results in Brief:
Background:
Most States Are Not in Compliance with the 1994 Title I Accountability
and Assessment Requirements:
States Cited Support and Coordination as Furthering Compliance:
Most States are Taking Some Action To Ensure Accurate Scoring,
Justification for Exemptions, and Appropriate Accommodations, but
Actions are Limited:
Conclusions:
Recommendation:
Agency Comments:
Appendix I: Scope and Methodology:
Appendix II: Comments from the Department of Education:
Tables:
Table 1: Accountability and Assessment Requirements under the 1994 and
2001 Reauthorizations of Title I:
Table 2: Status of States‘ Compliance with 1994 Title I Assessment
Requirements as of March 2002:
Table 3: Timeline Waiver Status for 30 States in Meeting the 1994 Title
I Assessment Requirements as of March 2002:
Abbreviations:
ESEA: Elementary and Secondary Education Act:
GAO: General Accounting Office:
LEA: local education agency:
[End of section]
United States General Accounting Office:
Washington, DC 20548:
April 1, 2002:
The Honorable Roderick R. Paige:
The Secretary of Education:
Dear Mr. Secretary:
Title I of the Elementary and Secondary Education Act (ESEA), the
largest source of federal funding for education, will provide states
with $10.3 billion in fiscal year 2002 to improve the educational
achievement of children at risk. Title I serves about 12.5 million
children in all 50 states, the District of Columbia, and Puerto Rico.
In response to concerns that Title I funding was not significantly
improving the educational achievement of at-risk students, in 1994
Congress mandated major changes to Title I, including changes to how
states measure student performance. As part of the 1994 ESEA
reauthorization, states were required to adopt or develop challenging
curriculum content and performance standards, assessments aligned with
content standards, and accountability systems to assess schools‘ and
districts‘ progress in raising student achievement. In exchange for
meeting these requirements, states were given increased flexibility in
the use of Title I and other federal funds. New legislation that
reauthorizes ESEA – The No Child Left Behind Act of 2001 – has since
been passed. The new legislation does not reduce or eliminate any of
the 1994 requirements. Instead, it augments the assessment and
accountability requirements that states must implement and increases
the stakes for schools that fail to make adequate progress. This report
provides a snapshot of how close states had come to meeting the 1994
requirements when the 2001 requirements were signed into law.
In light of the increasingly important role played by the required
assessment and accountability measures, we have collaborated with other
audit organizations to review aspects of how states are implementing
these measures and ensuring that data used to assess schools‘ progress
in raising student achievement are complete and accurate. This report
is one of several on this topic to be issued separately by the various
collaborating audit organizations, which include the U.S. Department of
Education‘s Office of the Inspector General (OIG), the Texas State
Auditor‘s Office, the Pennsylvania State Auditor‘s Office, and the City
of Philadelphia Controller‘s Office. In this report, we provide GAO‘s
findings regarding (1) the status of states‘ compliance with key 1994
Title I requirements, (2) factors that have helped or hindered states
in meeting the requirements, and (3) the actions states are taking to
ensure that Title I assessments are scored accurately, exemptions for
students with limited English proficiency are justified and students
with disabilities are appropriately accommodated during testing
according to Title I regulations.
To assess states‘ progress in implementing important assessment and
accountability requirements introduced by Title I, we (1) interviewed
Education officials and reviewed Education memoranda and reports
regarding states‘ compliance[Footnote 1] with the 1994 requirements,
(2) surveyed State Education Agency (SEA) Title I directors regarding
the challenges faced by states with and without approved assessment
systems, and (3) conducted telephone interviews with Title I and other
state officials in states that are in compliance with Title I to
develop detailed information on strategies for overcoming key barriers.
We also conducted similar interviews in some states still working to
attain compliance with the 1994 requirements to determine factors that
have hindered their progress. We sent surveys to the 50 states, the
District of Columbia, and Puerto Rico and obtained 50 responses (96
percent). We conducted our work in accordance with generally accepted
government auditing standards between June 2001 and March 2002. (See
app. I for specifics on our scope and methodology.)
Results in Brief:
As of March 2002, 17 states were in compliance with the 1994 Title I
assessment requirements; however, 35 states and instrumentalities were
not.[Footnote 2] The 1994 legislation required states to be in full
compliance with the requirements by January 2001 but allowed the
Department of Education to extend that deadline. For states that have
not met the requirements, Education distinguishes between states that
are near compliance and those that still have a significant amount of
work remaining, granting timeline waivers to the former and compliance
agreements to the latter. Education has granted timeline waivers to 30
states to give them more time to meet all requirements. According to
Education, by January 31, 2004, all of these states will be in
compliance and meet all 1994 Title I requirements, none of which are
reduced or eliminated by the 2001 legislation. If states fail to meet
extended timelines for implementing the 1994 requirements, they are
subject to the withholding of some Title I administrative funds.
Education has asked the remaining five states to enter into compliance
agreements that will establish the final date by which they must meet
all requirements. Education has held public hearings in each of these
states. Under the law, without a compliance agreement, these states may
lose funding. Noncompliant states most frequently have not met the
specific requirements to assess all students and to report the data by
subgroups of students, but some also have more intractable problems,
such as assessments that are not aligned with state standards. Because
the majority of states have not met the requirements of the 1994 law,
many states may not be well positioned to meet the deadlines for
implementing the additional requirements in the 2001 legislation.
Title I directors indicated that a state‘s ability to meet the 1994
requirements improved when the necessary players”both state leaders and
state agency staff”made compliance a priority and coordinated with one
another to achieve it; most directors indicated that state compliance
was hindered by inadequate funding. Each compliant state that we
interviewed said that the state‘s governor, legislature, department of
education leaders, or business leaders first prompted compliance
initiatives, such as establishing a blue ribbon committee to address
the issue. According to Title I directors, state leader initiatives
that endorsed compliance as a high-priority, the backing of
requirements by state and local staff involved in implementation, and
technical expertise at the state level were among the types of support
that contributed most to states‘ compliance. Coordination between staff
in different offices and levels of government was another factor
identified as important to compliance. Inadequate funding was the most
often cited factor that hindered state compliance with requirements,
according to our survey of Title I directors. In interviews, the
directors said that their investment of time and money in systems of
assessment that predated and conflicted with the requirements of the
1994 legislation was an obstacle to compliance.
Most of the states reported taking some action to ensure that Title I
assessments were scored accurately, that any exemptions for students
with limited English proficiency were justified, and students with
disabilities were receiving appropriate testing accommodations. Almost
all states hire a contractor to score the Title I assessments; however,
16 of these states reported that they did not monitor the scoring done
by the contractor. Most of those who did monitor the scoring reported
they did so by selecting a sample of answer sheets to compare with the
contractor‘s results to ensure their accuracy. Others compared school
and district test results with the results from previous years to
identify large discrepancies that might suggest a problem. Several
states have experienced errors in scoring done by contractors. For some
states, these errors resulted in the incorrect identification of
schools in need of improvement and students in need of additional
services. Thirty-three states also reported taking some actions to
ensure that any exemptions for students with limited English
proficiency were justified and 41 reported actions to ensure students
with disabilities received appropriate accommodations during testing.
For example, states reported they had developed standards for districts
to follow in accommodating these students so that assessments can yield
valid measures of their performance. However, states reported few
actions that would ensure that these guidelines were being followed.
Many states are still developing procedures to ensure that any
exemptions are justified and accommodations are appropriate. Education
performs compliance reviews of grantee programs and is in the process
of redesigning this review process. However, the redesigned reviews do
not specifically include the monitoring of states‘ actions with regard
to contractors‘ scoring of assessments.
To reduce the potential for undetected errors in test scoring that
could have material effects on educational decisions or damage
confidence in the test results, we are recommending that Education
specifically include monitoring of state actions regarding contractors
and scoring provisions in its state compliance reviews. Education
agreed to our recommendation in its official agency comments. A copy of
the comments is printed in Appendix II.
Background:
The original Title I legislation was passed in 1965, but the 1994
reauthorization of ESEA mandated fundamental changes to the Title I
program. One of the key changes involved the development of state
systems of standards and assessments to ensure that students served by
Title I were held to the same standards of achievement as all other
children. Prior to 1994, some states had already implemented assessment
systems, but these tended to be norm-referenced”students‘ performance
was judged in relation to the performance of other students. The 1994
legislation required assessments that were criterion-based”students‘
performance was to be judged against an objective standard.[Footnote 3]
Every state applying for Title I funds since 1994 agreed to implement
the changes described in the 1994 law and to bring its assessment
systems into compliance. States are also required to develop a
definition of adequate yearly progress based on the assessments to hold
schools accountable for educational progress. To help states that could
not meet the proposed 2001 timeline, Education had authority to grant
timeline waivers and compliance agreements to states under certain
conditions. In its 2001 ESEA reauthorization, Congress increased
testing requirements for states as well as the consequences for not
improving test scores in schools and did not eliminate any of the
requirements of the 1994 legislation. As shown in table 1, the 1994 and
2001 legislative requirements for assessment and accountability concern
developing standards for content and performance; measuring
improvement; implementing and administering assessments, including
assessing students with limited English proficiency; reporting
assessment data; and applying consequences for not meeting performance
goals.
Table 1: Accountability and Assessment Requirements under the 1994 and
2001 Reauthorizations of Title I:
Developing standards for content and performance:
1994 requirements: Develop challenging standards for what students
should know in math and reading or language arts. In addition, for each
of these standards, states should develop performance standards
representing three levels: partially proficient, proficient, and
advanced. The standards must be the same for all children. If the state
does not have standards for all children, it must develop standards for
Title I children that incorporate the same skills, knowledge, and
performance expected of other children.
2001 requirements: In addition, develop standards for science content
by 2005-06. The same standards must be used for all children.
Implementing and administering assessments:
1994 requirements:
* Develop and implement assessments aligned with the content and
performance standards in at least math and reading or language arts.
2001 requirements:
* Add assessments aligned with the content and performance standards in
science by the 2007-08 school year. These science assessments must be
administered at some time in each of the following grade ranges: from
grades 3 through 5, 6 through 9, and 10 through 12.
1994 requirements:
* Use the same assessment system to measure Title I students as the
state uses to measure the performance of all other students. In the
absence of a state system, a system that meets Title I requirements
must be developed for use in all Title I schools.
2001 requirements:
* Use the same assessment system to measure Title I students as the
state uses to measure the performance of all other students. If the
state provides evidence to the secretary that it lacks authority to
adopt a statewide system, it may meet the Title I requirement by
adopting an assessment system on a statewide basis and limiting its
applicability to Title I students or by ensuring that the Title I local
educational agency (LEA) adopts standards and aligned assessments.
1994 requirements:
* Include in the assessment system multiple measures of student
performance, including measures that assess higher-order thinking
skills and understanding.
2001 requirements:
* Unchanged.
1994 requirements:
* Administer assessments for math and reading in each of the following
grade spans: from grades 3 through 5, 6 through 9, and 10 through 12.
2001 requirements:
* Administer reading and math tests annually in grades 3 through 8,
starting in the 2005-06 school year (in addition to the assessments
previously required sometime within grades 10 through 12). States do
not have to administer math and reading or language arts tests annually
in grades 3 through 8 if Congress does not provide specified amounts of
funds to do so, but states have to continue to work on the development
of the standards and assessments for those grades. Have students in
grades 4 and 8 take the National Assessment for Educational Performance
(NAEP) exams in reading and math every other year beginning in 2002-03,
as long as the federal government pays for it.
1994 requirements:
* Implement controls to ensure the quality of the data collected from
the assessments.
2001 requirements:
* Unchanged.
Including students with limited English proficiency and with
disabilities in assessments:
1994 requirements:
* Assess students with disabilities and limited English proficiency
according to standards for all other students. Provide reasonable
adaptations and accommodations for students with disabilities or
limited English proficiency, to include testing in the language and
form most likely to yield accurate and reliable information on what
they know and can do.
2001 requirements:
* By 2002-03, annually assess the language proficiency of students
with limited English proficiency. Students who have attended a U.S.
school for 3 consecutive years must be tested in English unless an
individual assessment by the district shows testing in a native
language will be more reliable.
Reporting data:
1994 requirements:
* Report assessment results according to the following: by state, LEA,
school, gender, major racial and ethnic groups, English proficiency,
migrant status, disability, and economic disadvantage.
2001 requirements:
* Unchanged.
1994 requirements:
* LEAs must produce for each Title I school a performance profile with
disaggregated results and must publicize and disseminate these to
teachers, parents, students, and the community. LEAs must also provide
individual student reports, including test scores and other information
on the attainment of student performance standards.
2001 requirements:
* Provide annual information on the test performance of individual
students and other indicators included in the state accountability
system by 2002-03. Make this annual information available to parents
and the public and include data on teacher qualifications. Compare
high- and low-poverty schools with respect to the percentage of classes
taught by teachers who are ’highly qualified,“ as defined in the law,
and conduct similar analyses for subgroups listed in previous law.
Measuring improvement:
1994 requirements:
* Use performance standards to establish a benchmark for improvement
referred to as ’adequate yearly progress.“ All LEAs and schools must
meet the state‘s adequate yearly progress standard, for example, having
90 percent of their students performing at the proficient level in
math. LEAs and schools must show continuous progress toward meeting the
adequate yearly progress standard. The state defines the level of
progress a school or LEA must show. Schools that do not make the
required advancement toward the adequate yearly progress standard can
face consequences, such as the replacement of the existing staff.
2001 requirements:
In addition to showing gains in the academic achievement of the overall
school population, schools and districts must show that the following
subcategories of students have made gains in their academic
achievement: pupils who are economically disadvantaged, have limited
English proficiency, are disabled, or belong to a major racial and
ethnic group. To demonstrate gains among these subcategories of
students, school districts measure their progress against the state‘s
definition of adequate yearly progress. States have 12 years for all
students to perform at the proficient level.
Consequences for not meeting the adequate yearly progress standard:
1994 requirements:
* LEAs are required to identify for improvement any schools that fail
to make adequate yearly progress for 2 consecutive years and to provide
technical assistance to help failing schools develop and implement
required improvement plans. After a school has failed to meet the
adequate yearly progress standard for 3 consecutive years, LEAs must
take corrective action to improve the school.
2001 requirements:
* New requirements are more specific as to what actions an LEA must
take to improve failing schools. Actions are defined for each year the
school continues to fail leading up to the 5th year of failure when a
school must be restructured by changing to a charter school, replacing
school staff, or state takeover of the school administration. The new
law also provides that LEAs offer options to children in failing
schools. Depending on the number of years a school has been designated
for improvement, these options may include going to another public
school with transportation paid by the LEA or using Title I funds to
pay for a private tutor.
Source: P.L. 103-382 and P.L. 107-110.
[End of table]
Almost all states employ contractors to perform services to help them
meet these requirements. Among states that we interviewed, contractors
included private companies, universities, nonprofit organizations, and
individual consultants. These entities were hired to provide services
that may include assessment development, administration, scoring,
analysis, and reporting of results. Some of these entities can provide
combinations of services to states, such as test development and test
scoring. States are responsible for monitoring contractor performance.
Congress allowed states to phase in the 1994 ESEA requirements over
time, giving states until the beginning of the 2000-01 school year to
fully implement them with the possibility of limited extensions.
Education is responsible for determining whether or not a state is in
compliance with these requirements and is authorized under ESEA, to
give states more time to implement the requirements as long as states
are making adequate progress toward this goal. States submit evidence
to Education showing that their system for assessing students and
holding schools accountable meets Title I requirements. Education has
contracted with individuals with expertise in assessments and Title I
to review this evidence. The experts provide Education with a report on
the status of each state regarding the degree to which a state‘s system
for assessing students meets the requirements and deserves approval.
Using this and other information, the Secretary sends each state a
decision letter that summarizes the experts‘ review and communicates
whether a state is in full compliance, in need of a timeline waiver, or
more seriously, a compliance agreement. Education may withhold funds if
a state does not meet the terms of its compliance agreement. The 1994
legislation was not specific in the amount of administrative funds that
could be withheld from states failing to meet negotiated timelines, but
the 2001 legislation states that Education must withhold 25 percent of
state administrative funds until the state meets the 1994 requirements
including the terms of any timeline waivers or compliance agreements.
In June 2000, we issued a report on states‘ efforts to ensure
compliance with key Title I requirements.[Footnote 4] At that time, we
expressed concern about the number of states that were not positioned
to meet the deadlines in the 1994 law. To increase compliance, we made
two recommendations. We recommended that the Department of Education
should (1) facilitate among states the exchange of information and best
practices regarding the implementation of Title I requirements and (2)
implement additional measures to improve research on the effectiveness
of different services provided through Title I to improve student
outcomes. Education continues to work on the implementation of these
recommendations. In addition, we said that Congress should consider
requiring that states‘ definitions of adequate yearly progress apply to
disadvantaged children, as well as to the overall student population.
The 2001 legislation does require that states apply adequate yearly
progress requirements and report on the results by subgroups, including
students in poverty, with disabilities, and with limited English
proficiency.
Most States Are Not in Compliance with the 1994 Title I Accountability
and Assessment Requirements:
As of March 2002, 17 states were in compliance with the 1994 Title I
assessment requirements; however, 35 were not. (See table 2.)
Departmental approval of timeline waivers to give states more time to
reach compliance has been granted for 30 states. Education has asked
five states to enter into compliance agreements that will establish the
final date by which they must be in compliance before losing Title I
funding. Among other requirements, states that are not in compliance
have most frequently not met the specific requirements to assess all
students and break out assessment data by subcategories of students.
The 2001 legislation requires states to implement additional
assessments through 2008, thus substantially augmenting current
assessment requirements. Education has published a notice of proposed
negotiated rulemaking in the Federal Register and has solicited
comments from outside parties in preparation for establishing state
compliance standards for the 2001 legislation.
Table 2: Status of States‘ Compliance with 1994 Title I Assessment
Requirements as of March 2002:
Compliant (17):
Colorado:
Delaware:
Indiana:
Kansas:
Louisiana:
Maine:
Maryland:
Massachusetts:
Missouri:
North Carolina:
Oregon:
Pennsylvania:
Rhode Island:
Texas:
Vermont:
Virginia:
Wyoming.
Noncompliant (35):
Alabama:
Alaska:
Arizona:
Arkansas:
California:
Connecticut:
District of Columbia:
Florida:
Georgia:
Hawaii:
Idaho:
Illinois:
Iowa:
Kentucky:
Michigan:
Minnesota:
Mississippi:
Montana:
Nebraska:
Nevada:
New Hampshire:
New Jersey:
New Mexico:
New York:
North Dakota:
Ohio:
Oklahoma:
Puerto Rico:
South Carolina:
South Dakota:
Tennessee:
Utah:
Washington:
West Virginia:
Wisconsin.
Source: Department of Education.
[End of table]
When Education determines that a state is not in compliance with the
1994 Title I assessment requirements, it may grant the state a timeline
waiver for meeting those requirements. A waiver may not exceed 3 years.
Education officials indicate that the agency grants waivers to states
that have a history of success in implementing significant portions of
their assessment systems, have a clear plan with a definite timeline
for complying with the Title I requirements, and have the capacity to
carry out the plan and thus meet those requirements. When a state
requests a waiver, it must provide Education with a plan that includes
a timeline for addressing deficiencies in the state‘s assessment
system. Education reviews this information to decide whether the waiver
should be granted and its duration. So far, Education has granted
timeline waivers to 30 states. (see table 3.)
Table 3: Timeline Waiver Status for 30 States in Meeting the 1994 Title
I Assessment Requirements as of March 2002:
1-year (9):
Arkansas:
Connecticut:
Florida:
Illinois:
Kentucky:
New Hampshire:
New York:
Utah:
Washington.
2-year (18):
Alaska:
Arizona:
California:
Georgia:
Hawaii:
Iowa:
Michigan:
Mississippi:
Nebraska:
Nevada:
New Jersey:
New Mexico:
North Dakota:
Oklahoma:
South Carolina:
South Dakota:
Tennessee:
Wisconsin.
3-year (3):
Minnesota:
Ohio:
Puerto Rico.
Note: Each timeline waiver has an exact ending date.
1 year = to be complete December 31, 2002 or sooner.
2 year = to be complete December 12, 2003 or sooner.
3 year = to be complete January 31, 2004 or sooner.
Source: Department of Education.
[End of table]
A compliance agreement is deemed necessary when Education determines
that a state will not complete the implementation of its assessment
system in a timely manner. According to Education officials, a state
requiring a compliance agreement generally does not have a history of
successful implementation, has not met a significant number of Title I
requirements, and does not have a plan in place for meeting those
requirements. Education recommends a compliance agreement so that a
state may continue to receive Title I funds. Before Education may enter
into a compliance agreement, a public hearing must be held in which the
state has the burden of persuading Education that full compliance with
the Title I requirements is not feasible until a future date. The state
must be able to attain compliance within 3 years of the signing of the
compliance agreement by the state and Education. The state then
negotiates the terms of the agreement with Education. Education‘s
written findings and the terms of the compliance agreement are
published in the Federal Register. A state that enters into a
compliance agreement to address requirements of the 1994 Title I law
and subsequently fails to meet the requirements of the agreement can be
subject to loss of some state Title I administrative funds. Education
is presently working on five compliance agreements (Alabama, Idaho,
Montana, West Virginia, and the District of Columbia) and has held
public hearings for each.
The 2001 reauthorization of ESEA was signed into law on January 8,
2002. The act provides states not in compliance with the 1994 Title I
requirements at the time of the signing of the 2001 legislation with a
90-day period that started on January 8, 2002 to negotiate changes in
the dates by which they must be in compliance with the 1994
requirements. After the conclusion of this 90-day period, the
legislation prohibits further extensions for compliance with the 1994
requirements. States failing to meet these negotiated timelines will be
subject to loss of some of their Title I administrative funds.
According to senior Education officials, this loss could be significant
to states, as many use federal program administrative funds to pay the
salaries of state department of education staff.
A review of documents from Education shows that noncompliant states
have most commonly not met two Title I requirements”assessing all
students and breaking out assessment data by subcategories of students.
Title I does not permit states to exempt any student subgroup from
their assessments and Education‘s guidance states that individual
exemptions may be permitted by the states in extraordinary
circumstances. Nonetheless, many states allow substantial exemptions
for students with disabilities and limited English proficiency. Several
states reported that they have only recently amended laws that
prohibited testing of some students with limited English proficiency.
Title I also requires states, local districts, and schools to report
the performance of students overall and in a variety of subcategories.
These categories are gender, race, ethnicity, English proficiency
status, migrant status, disability status, and economic disadvantage.
Many states disaggregated data for some but not all of these
categories. Documents from Education show that data for the disabled,
migrant, and economically disadvantaged subcategories are the most
common subgroups excluded from state, district, and school reports. In
addition, many states lag in other areas, such as aligning assessments
to state content standards.
To achieve compliance with the 2001 legislation, states will need to
add new standards and increase assessment efforts, as detailed in table
1. In responding to our survey, 48 states reported that they have
developed content standards in science, but only 16 reported having
annual assessments for math and 18 reported annual assessments for
reading in all grades 3 through 8. In addition, states will not have
the 2 to 3 year timeline waivers available to them as they had when
they worked to meet the 1994 requirements. New 2001 requirements listed
in table 1 have deadlines that vary according to the requirement, and
the Secretary of Education can give states 1 additional year from those
deadlines to meet the new requirements, but only in case of a ’natural
disaster or a precipitous and unforeseen decline in the financial
resources of the state.“ Since the majority of states have not met the
requirements of the 1994 law, it appears that many states may not be
well-situated as they work to meet the schedule for implementing new
requirements that build upon the 1994 requirements.
States Cited Support and Coordination as Furthering Compliance:
States successful in meeting key Title I requirements attributed their
success primarily to four factors. These factors were (1) the efforts
of state leaders to make Title I compliance a priority; (2)
coordination between staff of different agencies and levels of
government; (3) obtaining buy-in from local administrators, educators,
and parents; and (4) the availability of state-level expertise. Survey
respondents identified inadequate funding as an obstacle to compliance.
The state Title I officials we interviewed said that their states‘
commitment of resources to norm-referenced assessments that conflicted
with the 1994 Title I requirements contributed to this obstacle.
Almost 80 percent of the respondents identified state leaders‘ efforts
as a factor that facilitated their meeting the 1994 Title I
requirements. In every state that had attained compliance with the
Title I requirements, the officials that we interviewed said that the
governor, legislators, or business leaders made compliance with the
Title I requirements a high-priority. States described the development
of high-level committees, new state legislation, and other measures to
raise the visibility and priority of this issue. For example, one
governor spearheaded a plan that used commissions to develop content
standards and assessments aligned with those standards. Some state
officials we interviewed reported that efforts by state department of
education leaders resulted in major organizational changes in the state
education department. For example, according to one Title I Director,
the state changed the organizational structure and reporting
relationships of state offices to organize them by function rather than
by funding streams and to enhance coordination; according to another
Title I Director, state leaders who did not support changes necessary
to achieve compliance with Title I were replaced with staff that did
support the changes.
In responding to our survey, over 80 percent of the Title I officials
identified the ability of staff or agencies to coordinate their efforts
with one another as a factor that helped them meet requirements. In our
interviews, state officials cited the necessity of coordination between
state and local staff working in the areas of assessment, instruction,
and procurement. Two of the states we interviewed specifically noted
that when the assessment office shared a physical location with the
Title I office, coordination was easier and the ability to achieve
compliance with Title I was enhanced.
Title I and other officials we interviewed in those states that had met
the 1994 Title I assessment requirements noted that they had made great
efforts to obtain buy-in from other state officials, local
administrators, educators, and the public. They said that efforts to
ensure buy-in paved the way for changes meant to ensure compliance with
the assessment and accountability requirements of the1994 legislation.
Several officials we interviewed reported holding public meetings and
focus groups to obtain input from parents, teachers and local
administrators regarding how the state should implement Title I
requirements. They also reported conducting public relations campaigns
to educate the public about the importance of complying with Title I
requirements for standards-based assessment. One state, for example,
conducted 6 years of focus groups and hearings and conducts a
conference annually to allow local education officials to gain advice
from experts regarding any concerns or problems they are having in
implementing Title I requirements.
In responding to our survey, over 80 percent of state Title I directors
identified the availability of state level expertise as a factor that
facilitated their efforts to meet Title I requirements. State officials
we interviewed reported that training for teachers and district
personnel was often needed to apply new content standards in the
classroom and to administer assessments correctly. Two states, for
example, used regional centers to educate local staff on assessments
and standards.
Fifty percent of survey respondents identified inadequate funding as an
obstacle in moving toward compliance and noncompliant states cited this
problem more often than compliant ones. In our interviews, Title I and
assessment officials from noncompliant states reported that progress
toward compliance with Title I requirements was stalled because of
investments they had made in assessment systems that predated and
conflicted with the requirements of the 1994 Title I reauthorization.
Respondents said that they had made substantial investments of time and
money in systems of assessment that often relied upon norm-referenced
assessments and did not meet the 1994 requirement for criterion-based
tests. They noted that it took their states several years to change
from the old system of assessment to one meeting the requirements
specified by the 1994 reauthorizing legislation. According to the
officials we interviewed, building support to start again on another
system and obtaining the funding made it more difficult to make the
necessary changes in a timely manner. In addition, one survey
respondent from a very small state noted that due to the state‘s size
it has a small number of staff and does not have the technical
expertise needed to develop a new system, thus hampering the state‘s
ability to meet the requirements.
Most States are Taking Some Action To Ensure Accurate Scoring,
Justification for Exemptions, and Appropriate Accommodations, but
Actions are Limited:
Most states are taking some action to ensure that Title I assessments
are scored accurately, that any exemptions for students with limited
English proficiency are justified, and that students are receiving
appropriate accommodations when these are needed to gather an accurate
assessment of their abilities. Most states hire a contractor to score
Title I assessments and about two-thirds of these states monitored the
scoring performed by the contractor. Some states that hire contractors
have found errors in the scoring the contractors did, and in some
cases, these errors have had serious negative consequences for schools
and students. Most states reported taking some actions to ensure that
students with limited English proficiency and disabilities received
appropriate accommodations during testing. Education is redesigning its
current compliance and monitoring program to better monitor states‘
implementation of Title I.
Problems Found in Contractor Scoring:
According to our survey results, most states (44) hire a contractor for
test scoring, but 16 of these states identified no monitoring mechanism
to ensure the accuracy of their contractor‘s scoring and reporting.
Among those states that did report one or more monitoring mechanisms,
15 reported that they monitored the contractor‘s scoring by comparing a
sample of original student test results to the contractor‘s results. A
few states also reported, in interviews with us, that they compared
their most recent test scores with those from previous years and looked
for significant variations that suggested potential errors in scoring.
However, in our interviews, some assessment officials indicated that
they use this type of monitoring rather informally.
The problems identified in assessment scoring suggest that these
approaches do not always provide adequate assurance of complete and
accurate results. Indeed, several of the states that use contractors to
score tests reported that they have had problems with errors in scoring
whether or not they had monitoring measures in place. In some cases,
contractors marked correct answers as incorrect and in other cases the
contractors calculated the scores incorrectly. The errors were
discovered by a number of individuals, including local district
officials, parents, and state agency staff. These scoring errors had
impacts on students, families, and school and district resources. Based
on erroneous scores calculated by a contractor, one state sent
thousands of children to summer school in the mistaken belief that
their performance was poor enough to meet the criterion for summer
intervention. In addition to disrupting families‘ summer plans and
potentially preventing student promotions, this may have drawn
resources away from other necessary activities. In another case, based
on a contractor‘s erroneous scoring, a state incorrectly identified
several schools as ’in need of improvement,“ a designation that carries
with it both bad publicity and extra expense, for example, districts
may have to fund the needed improvements.
A few state officials that we interviewed told us that they have begun
instituting processes to check the accuracy of scoring. For example,
three states said that they had hired individuals who were experts in
test scoring or they hired other third parties to conduct independent
audits. States that were in compliance with 1994 Title I assessment
requirements generally had more complete monitoring systems, including
measures such as technical advisory committees to review results,
conduct site visits, and follow a sample of tests through the scoring
and reporting process. In contrast, several states indicated they are
still relying on contractor self-monitoring to ensure accurate scoring.
Although Education is obligated under the Federal Managers‘ Financial
Integrity Act of 1982[Footnote 5] and the Single Audit Act[Footnote 6]
to ensure that states that receive federal funds comply with statutory
and regulatory requirements to monitor contractors, it currently takes
limited action regarding states‘ monitoring of assessment contractors.
Education‘s inspector general has reported deficiencies in an important
vehicle for such oversight - -Education‘s compliance reviews of state
programs.[Footnote 7] The compliance reviews are conducted on a 4-year
cycle and include an on-site visit that lasts 1 week. Specifically, the
OIG cited insufficient time to conduct the reviews, lack of knowledge
among Education staff about areas they were reviewing, and a lack of
consistency in how the reviews were conducted. Senior Education
officials told us the department is redesigning the current compliance
and monitoring program used for its on-site visits to better focus on
outcomes and accountability in Title I and that it is addressing the
OIG‘s recommendations. However, a senior Education official who is
working on the redesign of the compliance reviews told us that the
current draft plans did not include specific checks on state monitoring
of assessment scoring. Confidence in the accuracy of test scoring is
critical to acceptance of the test results‘ use in assessing school
performance.
Some Actions Taken To Ensure Exemptions Justified and Accommodations
Appropriate:
According to our surveys and interviews, 33 states have taken at least
minimal actions to ensure any exemptions for students with limited
English proficiency are justified and 41 states take actions to ensure
accommodations for students with disabilities are appropriate.[Footnote
8] Most states reported that they had developed standards for districts
to follow in accommodating these students so that assessments could
yield accurate measures of their performance. However, states reported
few actions that would ensure that these guidelines were being
followed. For example, 17 states reported that they compare the number
of students with limited English proficiency tested within a given year
against the number for the previous year. They used this comparison as
their means of verifying that the numbers of students receiving
exemptions were reasonable. As the pool of students in a particular
school can change substantially from year to year, this comparison has
obvious limitations. Moreover, students‘ status, for example with
respect to English proficiency, can change from year to year.
Similarly, 37 states reported using an annual comparison of the number
of students with disabilities being tested as a check for appropriate
accommodations. However, it is not evident how such comparisons would
allow states to ascertain the appropriateness of the accommodations.
Survey results and interviews did indicate that more states are taking
actions to monitor accommodations for students with disabilities than
for students with limited English proficiency. For example, while 25
states reported that they had standards for accommodating students with
limited English proficiency, 36 had standards for accommodating
students with disabilities. The state officials that we interviewed
told us that this was because districts built upon steps they had taken
under the Individuals with Disabilities Education Act (IDEA)[Footnote
9] to document the accommodations needed by students with disabilities.
In general, states said that the districts have more experience and
technical expertise for assessing and supporting students with
disabilities because of working under IDEA for many years. In contrast,
some states lacked consistent standards for identifying students with
limited English proficiency and more states were still working to
develop alternate assessments or accommodations for these students.
Augmenting the 1994 requirements, the new 2001 legislation requires
that states annually assess the language proficiency of students with
limited English proficiency by the 2002-03 school year. States do
conduct cyclical monitoring of the implementation of all their programs
that might be used to assess the appropriateness of district policy and
practice with regard to testing accommodations. However, in a recent
review,[Footnote 10] we found that states varied dramatically in the
frequency of their on-site visits. The average time between visits to
districts ranged from 2 years or less (6 states) to more than 7 years
(17 states).
Conclusions:
This snapshot of the states‘ status with respect to the 1994 Title I
requirements suggests that many states may not be well-positioned to
meet the requirements added in 2001. Only 17 states were in compliance
with the assessment requirements of the 1994 law in March of 2002;
therefore, the majority of states will still be working on meeting the
1994 requirements as they begin work toward meeting the new
requirements.
In addition, despite the enhanced emphasis on assessment results,
states still appear to be struggling with ensuring that assessment data
are complete and correct. The 1994 and 2001 ESEA reauthorizations
raised student assessments to a new level of importance. The
assessments are intended to help ensure that all students, including
those who have disabilities and those who have limited English
proficiency are meeting challenging standards. In addition, assessment
results are a key part of the mechanism for holding both schools and
states accountable for improving educational performance. Thus,
ensuring the completeness and accuracy of assessment data is central to
measuring students‘ progress and ensuring accountability. Without
adequate oversight of assessment scoring, efforts to identify and
improve low-performing schools could be hindered by lack of confidence
in assessment results or uncertainty regarding whether particular
schools have been appropriately identified for improvement. Education‘s
current monitoring does not include specific oversight of how states
ensure the quality of scoring contractors‘ work, but Education‘s
revision of its monitoring process provides the agency with an
opportunity to help states ensure that scoring done by contractors is
accurate.
Recommendation:
To enhance confidence in state assessment results, we recommend that
when the Department of Education monitors state compliance with federal
programs, it include checks for contractor monitoring related to Title
I, Part A. Specifically, Education should include in its new compliance
reviews a check on the controls states have in place to ensure proper
test scoring and the effective implementation of these controls by
states.
Agency Comments:
We provided Education with a draft of this report for review. The
Department‘s official comments are printed in appendix II. In its
comments, Education agreed with our recommendation. Education also
provided us with technical comments that we incorporated in the report
as appropriate.
We are sending copies of this report to appropriate congressional
committees and other interested parties. If you have any questions
about this report, please contact me on (202) 512-7215 or Betty Ward-
Zukerman on (202) 512-2732. Key contributors to this report were Mary
Roy, G. Paul Chapman II, Laura Pan Luo, Corinna Nicolaou, and Patrick
DiBattista.
Sincerely yours,
Signed by:
Marnie S. Shaul, Director:
Education, Workforce, and Income Security Issues:
[End of section]
Appendix I: Scope and Methodology:
We conducted this review in conjunction with our partners in the
Domestic Working Group. The Domestic Working Group‘s objective is to
allow officials in the federal, state, and local governmental audit
communities to interact on a personal and informal basis on various
topics of mutual concern. The group consists of 18 (6 federal, 6 state,
and 6 local) top officials and is intended to complement the work of
the intergovernmental audit forums and other professional associations.
For this review, the Texas State Auditor's Office conducted a detailed
assessment of data quality at the state and local levels in Texas,
while the Department of Education's Office of Inspector General did so
at the state and local levels in California and conducted additional
work on control processes at the Department of Education. In
Pennsylvania, the Pennsylvania Department of Auditor General conducted
work at the state level and the Philadelphia Controller's Office
pursued the same goal within the city of Philadelphia.
To complement these efforts, GAO surveyed all states and conducted
detailed interviews with several regarding their experiences in
implementing major provisions of Title I. Specifically, we reviewed
three key questions: (1) the status of states‘ compliance with key 1994
Title I assessment requirements; (2) factors that have hindered or
helped states move toward meeting the requirements; and (3) the actions
states are taking to ensure that Title I assessments are scored
accurately, exemptions for students with limited English proficiency
are justified, and students with disabilities are accommodated during
testing according to federal regulations.
We obtained information on the first objective from the Department of
Education. We met with Education officials and obtained updated
listings of compliance throughout the audit. In addition, we reviewed
state decision letters, peer reviews of state assessment systems, and
reports completed or commissioned by Education‘s Planning and
Evaluation Service.
To address the second and third questions, we used both a state survey
of Title I directors and detailed interviews with state Title I
officials and other state officials who played a key role in Title I
compliance - often assessment officials and sometimes Special
Education, program evaluation, and information technology officials. We
sent the survey to all 50 state directors and to the District of
Columbia and Puerto Rico. We received 50 completed surveys. We followed
up with 19 states to clarify and expand on questions in the interview
related to contracting for the scoring of tests. We interviewed
officials from 5 states that had assessment systems approved by
Education and 3 states that were still trying to attain compliance. We
also interviewed two expert reviewers, Education officials with
responsibility for Title I and program review, three officials at
Education‘s regional assistance centers, and officials at the Council
of Chief of State School Officers. We coordinated our work and findings
with our audit partners, who provided us with information relative to
their states‘ activities.
[End of section]
Appendix II: Comments from the Department of Education:
United States Department Of Education:
Office Of Elementary And Secondary Education:
The Assistant Secretary:
600 Independence Ave., S.W.
Washington, D.C. 20202-6100:
"Our mission is to ensure equal access to education and to promote
educational excellence throughout the Nation."
Mamie S. Shaul:
Director:
Education, Workforce and Income Security Issues:
U.S. General Accounting Office:
Washington, DC 20548:
Dear Ms. Shaul:
This is in response to your letter to Secretary Paige requesting
Department of Education comments on the draft report Title I: Education
Needs to Monitor States' Scoring of Assessments (GAO/02-393). Thank you
for the opportunity to comment.
The draft document recommends that the Department monitor States'
actions with respect to their contractors' scoring of assessments. As a
matter of clarification, scoring of assessments is not typically a
State function. Often, the State enters into a contract with a test
developer who, as a part of the contractual agreement, is responsible
for scoring. States have the primary responsibility to oversee the work
of the test developer and to verify the accuracy of results. Typically,
States include scoring in their contracts with test developers, as well
as other safeguards related to administration, analysis, and reporting
procedures.
The Department, as recommended, will include in its monitoring
activities a review of a State's compliance in monitoring the technical
quality of products delivered by its contractor, and its procedures to
ensure accuracy of assessment data prior to dissemination.
If I can be of further assistance, please do not hesitate to contact me
at (202) 401-0113.
Sincerely,
Signed by:
Susan B. Neuman, Ed.D.
[End of section]
Footnotes:
[1] In this report, we refer to a state as compliant when Education has
fully approved its assessment system for meeting the 1994 final
assessment requirements. We do not intend to imply that these states
are necessarily in compliance with all requirements of the 1994 law.
[2] In this report, when we refer to states we will be including the
instrumentalities of the District of Columbia and Puerto Rico.
[3] A norm-referenced test is an objective test that is standardized on
a group of individuals whose performance is evaluated in relation to
the performance of others. Criterion-referenced tests are assessments
that measure the mastery of specific skills or subject content and
focus on the performance of an individual as measured against a
standard or criterion rather than the performance of others taking the
test.
[4] General Accounting Office, ’Title I Program: Stronger
Accountability Needed for Performance of Disadvantaged Students“
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/HEHS-00-89],
(Washington, D.C.: June 1, 2000).
[5] U.S. General Accounting Office, ’Standards for Internal Control in
the Federal Government“ [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO/AIMD-00-21.3.1], (Washington, D.C.: Nov. 1999).
[6] P.L. 98-502.
[7] See ’Review of the Office of Elementary and Secondary Education‘s
Monitoring of Formula Grants: Final Audit Report“ (ED-OIG/A04-A0013),
November 2001.
[8] Accommodations for testing modify the circumstances for the student
while taking a test. For example, students with learning disabilities
may be allowed more time to take a test.
[9] IDEA was enacted in 1970 and, among other things, requires schools
to provide assessments for students with disabilities and provide
whatever accommodations they may need, including specific accommodation
for testing, such as more time to take tests, to ensure that the tests
provide valid reflections of their abilities.
[10] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/HEHS-00-89],
June 1, 2000.
[End of section]
GAO‘s Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO‘s commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO‘s Web site [hyperlink,
http://www.gao.gov] contains abstracts and full text files of current
reports and testimony and an expanding archive of older products. The
Web site features a search engine to help you locate documents using
key words and phrases. You can print these documents in their entirety,
including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ’Today‘s Reports,“ on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
[hyperlink, http://www.gao.gov] and select ’Subscribe to daily E-mail
alert for newly released products“ under the GAO Reports heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov:
(202) 512-4800:
U.S. General Accounting Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
