District of Columbia Opportunity Scholarship Program
Additional Policies and Procedures Would Improve Internal Controls and Program Operations Gao ID: GAO-08-9 November 1, 2007The D.C. School Choice Incentive Act created the first private kindergarten-through-grade-12 school-choice program supported by federal funds. The program was named the D.C. Opportunity Scholarship Program (OSP). GAO was asked to assess the (1) accountability mechanisms governing the use of funds, (2) results of the grantee's efforts to meet the program's recruiting priorities and eligibility requirements and inform parents of their choices, and (3) extent that the evaluation of OSP reflects statutory requirements and the implementation of the program supported the detection of useful and generalizable findings. To assess the program, GAO analyzed financial, program, and evaluation data. GAO did not assess the performance of participating private schools nor did GAO evaluate the impact of the program.
Following the Department of Education's selection of the Washington Scholarship Fund (WSF) to operate the District of Columbia Opportunity Scholarship Program, WSF greatly expanded its operations from $150,000 in federal and foundation grants in fiscal year 2004 to $12.9 million in 2006 without sufficient accountability mechanisms to govern the use of the funds. With such rapid expansion, WSF had little time to design and implement the needed systems, procedures, and internal controls for managing such a major increase in its operations. WSF's accountability was further weakened by high staff turnover, a lack of detailed fiscal policies, and nonintegrated accounting functions. We found that WSF did not adhere to its own procedures for making scholarship payments, and WSF's use of OSP funds to pay tuition for students attending schools that do not normally charge students tuition is not in accordance with the Act. Additionally, on the basis of a lack of documentation about whether before-and-after care included academic support, GAO was unable to determine if use of OSP funds to pay these fees was in accordance with the Act. Despite recruitment efforts and efforts to inform parents of their choices, WSF faced challenges recruiting students from schools designated as in need of improvement, ensuring private school quality and placement opportunities, and providing parents with accurate information regarding private schools. Students who were offered scholarships generally reported income consistent with OSP's financial eligibility standards, but, among students offered scholarships, students from schools in need of improvement were underrepresented relative to their presence in the population of District students. Although most private schools in the District officially participated in the program, the schools varied widely in the number of openings available to scholarship students, and few openings were available at the secondary level. The characteristics of participating schools varied, and some schools did not meet basic requirements to operate in the District, but the information WSF provided to parents to help them choose schools for children was not always complete and correct. The evaluation contractor developed a strong evaluation design that reflected the statutory requirements and used random assignment to strengthen comparisons between students offered and not offered scholarships. However, factors related to program implementation limited the ability to perform comparisons directed by the Act and the usefulness and generalizability of findings. For example, the Act directed the evaluation contractor to use the same test as the District used to measure achievement and to compare the achievement of students offered scholarships with students in District public schools. The contractor did use the test used by the District, but District officials adopted a new testing program the second year of the evaluation, making it infeasible to compare students offered scholarships with students in District public schools.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-08-9, District of Columbia Opportunity Scholarship Program: Additional Policies and Procedures Would Improve Internal Controls and Program Operations
This is the accessible text file for GAO report number GAO-08-9
entitled 'District of Columbia Opportunity Scholarship Program:
Additional Policies and Procedures Would Improve Internal Controls and
Program Operations' which was released on November 2, 2007.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
GAO:
United States Government Accountability Office:
Report to Congressional Requesters:
November 2007:
District Of Columbia Opportunity Scholarship Program:
Additional Policies and Procedures Would Improve Internal Controls and
Program Operations:
District of Columbia Opportunity Scholarship Program:
GAO-08-9:
GAO Highlights:
Highlights of GAO-08-9, a report to congressional requesters.
Why GAO Did This Study:
The D.C. School Choice Incentive Act created the first private
kindergarten-through-grade-12 school-choice program supported by
federal funds. The program was named the D.C. Opportunity Scholarship
Program (OSP). GAO was asked to assess the (1) accountability
mechanisms governing the use of funds, (2) results of the grantee‘s
efforts to meet the program‘s recruiting priorities and eligibility
requirements and inform parents of their choices, and (3) extent that
the evaluation of OSP reflects statutory requirements and the
implementation of the program supported the detection of useful and
generalizable findings. To assess the program, GAO analyzed financial,
program, and evaluation data. GAO did not assess the performance of
participating private schools nor did GAO evaluate the impact of the
program.
What GAO Found:
Following the Department of Education‘s selection of the Washington
Scholarship Fund (WSF) to operate the District of Columbia Opportunity
Scholarship Program, WSF greatly expanded its operations from $150,000
in federal and foundation grants in fiscal year 2004 to $12.9 million
in 2006 without sufficient accountability mechanisms to govern the use
of the funds. With such rapid expansion, WSF had little time to design
and implement the needed systems, procedures, and internal controls for
managing such a major increase in its operations. WSF‘s accountability
was further weakened by high staff turnover, a lack of detailed fiscal
policies, and nonintegrated accounting functions. We found that WSF did
not adhere to its own procedures for making scholarship payments, and
WSF‘s use of OSP funds to pay tuition for students attending schools
that do not normally charge students tuition is not in accordance with
the Act. Additionally, on the basis of a lack of documentation about
whether before-and-after care included academic support, GAO was unable
to determine if use of OSP funds to pay these fees was in accordance
with the Act.
Despite recruitment efforts and efforts to inform parents of their
choices, WSF faced challenges recruiting students from schools
designated as in need of improvement, ensuring private school quality
and placement opportunities, and providing parents with accurate
information regarding private schools. Students who were offered
scholarships generally reported income consistent with OSP‘s financial
eligibility standards, but, among students offered scholarships,
students from schools in need of improvement were underrepresented
relative to their presence in the population of District students.
Although most private schools in the District officially participated
in the program, the schools varied widely in the number of openings
available to scholarship students, and few openings were available at
the secondary level. The characteristics of participating schools
varied, and some schools did not meet basic requirements to operate in
the District, but the information WSF provided to parents to help them
choose schools for children was not always complete and correct.
The evaluation contractor developed a strong evaluation design that
reflected the statutory requirements and used random assignment to
strengthen comparisons between students offered and not offered
scholarships. However, factors related to program implementation
limited the ability to perform comparisons directed by the Act and the
usefulness and generalizability of findings. For example, the Act
directed the evaluation contractor to use the same test as the District
used to measure achievement and to compare the achievement of students
offered scholarships with students in District public schools. The
contractor did use the test used by the District, but District
officials adopted a new testing program the second year of the
evaluation, making it infeasible to compare students offered
scholarships with students in District public schools.
What GAO Recommends:
The Secretary of Education should direct the grantee to improve
internal controls, continue to integrate its financial systems, improve
monitoring, and provide accurate information to parents. The Secretary
and the Mayor of the District of Columbia should ensure participating
schools are in compliance with District requirements. The Secretary
should ensure that programs are implemented in accordance with any
required evaluation. The entities providing comments on this report
generally agreed with our recommendations but disagreed with many
findings.
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.GAO-08-9]. For more information, contact
Cornelia M. Ashby at 202-512-7215 or ashbyc@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
WSF's Accountability and Internal Control Were Inadequate for OSP, and
Some Payments Were Not Made in Accordance with the Act While Others
Raise Questions about Compliance:
While OSP Students Generally Met the Program's Eligibility
Requirements, WSF Had Limited Success Meeting Recruiting Priorities and
Did Not Provide Parents Complete Information:
Evaluation Contractor Adopted a Strong Methodology, but Program
Implementation and Other Factors Limited the Usefulness and
Generalizability of Findings:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Laws Authorizing Related Voucher Programs:
Appendix III: Comments from the Office of the Mayor of the District of
Columbia:
Appendix IV: Comments from the Department of Education:
Appendix V: Comments from the Washington Scholarship Fund:
Appendix VI: Contact and Staff Acknowledgment:
Related GAO Products:
Tables:
Table 1: Key Events in Implementing and Operating the OSP:
Table 2: Opportunity Scholarship Program Federal Funding for Fiscal
Years 2004-2006:
Table 3: Growth in WSF from Fiscal Years 2003--2006:
Table 4: Types of Outreach Methods Used to Reach Potentially Eligible
Families:
Table 5: Summary of Tuition Rates by Grade Level for Participating
Private Schools, School Year 2006-07:
Table 6: Unweighted Percentage of Students with Missing Achievement
Test Scores at Baseline and after 1 Year of Program Participation:
Table 7: Number of Total Private School Placements for OSP Students:
Table 8: Eligible Applicants Offered and Not Offered Scholarships and
Numbers Included in and Excluded from the Evaluation:
Table 9: Analysis of Laws Authorizing Voucher Programs in the District
of Columbia and the State of Ohio:
Table 10: Analysis of Laws Authorizing Voucher Programs in Cleveland,
Ohio, and Milwaukee, Wisconsin:
Figures:
Figure 1: WSF Staff Turnover between May 2006 and June 2007 Based on
Positions in Place in August 2006:
Figure 2: Reporting Structure Among WSF Staff as of August 2007:
Figure 3: Examples of Outreach Materials Grantee Used To Recruit Low-
Income Families:
Figure 4: Percentages of Students Offered OSP Scholarships Who Attended
Schools Designated as in Need of Improvement Were Consistently Below
the Percentages of Students in District of Columbia Schools Who
Attended Such Schools:
Figure 5: Examples of Schools That Operated without a Certificate of
Occupancy Specifying Use as a Private Day School:
Abbreviations:
OSP: Opportunity Scholarship Program:
MOU: memorandum of understanding:
MPCP: Milwaukee Parental Choice Program:
SSP: Signature Scholarship Program:
WSF: Washington Scholarship Fund:
United States Government Accountability Office:
Washington, DC 20548:
November 1, 2007:
The Honorable Edward M. Kennedy:
Chairman, Committee on Health, Education, Labor, and Pensions:
United States Senate:
The Honorable Richard J. Durbin:
Chairman, Subcommittee on Financial Services and General Government:
Committee on Appropriations:
United States Senate:
The Honorable Eleanor Holmes Norton:
House of Representatives:
In an effort to enhance educational opportunity for children from low-
income families in the District of Columbia (the District), Congress
passed the D.C. School Choice Incentive Act in January 2004.[Footnote
1] The Act created a private school scholarship program, subsequently
called the D.C. Opportunity Scholarship Program (OSP), for low-income
students living in the District of Columbia, setting a precedent for
the use of federal funds to provide scholarships to finance private
education for students in kindergarten through grade 12. The purpose of
the Act is to provide low-income parents of students in the District,
particularly parents of students who attend public schools identified
as in need of improvement under the No Child Left Behind Act,[Footnote
2] "with expanded opportunities for enrolling their children in higher
performing schools in the District" by providing annual scholarships of
up to $7,500 per child. The D.C. School Choice Incentive Act directed
the Secretary of Education to award a grant for up to 5 years to an
eligible entity or entities to operate the program, giving priority to
grant applicants who would most effectively give priority to students
from schools in need of improvement, target resources to students and
families that lack financial resources to take advantage of available
educational options, and provide students and families with the widest
range of educational options. The Act also directed the Secretary and
the Mayor of the District of Columbia to select an independent
evaluator to compare outcomes, including academic achievement, for
several groups of children, such as participating eligible students and
eligible students who were not offered scholarships and participating
eligible students and students in District Public Schools. The
evaluation was also to examine the effect of the program on District
public schools.
You asked us to assess the implementation of the program, including the
(1) accountability mechanisms in place governing the use of funds, (2)
results of the grantee's efforts to meet the program's recruiting
priorities and eligibility requirements and inform parents about their
choices, and (3) extent to which the evaluation reflects statutory
requirements and the implementation of the program supported the
detection of useful and generalizable findings.
To address these issues, we used multiple strategies. To assess the
accountability mechanisms governing the use of funds, we interviewed
officials of the Washington Scholarship Fund (WSF), the grantee
operating the program, of the Department of Education's Office of
Innovation and Improvement and of the District of Columbia; reviewed
documents describing the grantee's processes for implementing the
program; analyzed WSF's financial statements; obtained and reviewed
detailed financial data from WSF; and tested specific internal controls
over payments. To assess the results of the grantee's efforts to meet
the program's recruiting priorities and eligibility requirements and
inform parents about their choices, we interviewed officials at the
Office of Innovation and Improvement and the grantee; examined
pertinent documentation; and analyzed students' demographic data,
characteristics of participating schools, and information provided to
parents. We reviewed information on private schools from WSF and public
sources, but we did not assess specific private schools or their
performance. We did not evaluate the impact of the program, which was
the subject of Education's evaluation. We reviewed WSF's process for
collecting financial eligibility information, the records maintained by
WSF on household income data reported by parents of scholarship
recipients, and the findings on eligibility made by the grantee's
auditor. We did not perform independent testing of eligibility. To
assess the extent to which the evaluation reflects statutory
requirements and program implementation supported the detection of
useful and generalizable findings, we interviewed officials at
Education's Institute of Education Sciences and at Westat--the
evaluation contractor--reviewed Westat's and its affiliates'[Footnote
3] reports for year one and year two of the OSP that described the
participation of students and schools in the program, analysis plan,
and evaluation of the results of the program after 1 year of
participation. In assessing data reliability, we found problems with
WSF's and the evaluation contractor's program data systems, which we
discuss in appendix 1, that limited the analyses that we could perform.
We brought these issues to the attention of Education, WSF, and the
evaluation contractor. However, we were able to perform sufficient
analyses to answer the study's questions. We conducted our work between
September 2006 and September 2007 in accordance with generally accepted
government auditing standards. (For more detail on our scope and
methodology, see app. 1.)
Results in Brief:
Following the Department of Education's selection of WSF to operate the
District of Columbia Opportunity Scholarship Program, WSF greatly
expanded its operations from $150,000 in federal and foundation grants
in fiscal year 2004 to $12.9 million in 2006 without sufficient
accountability mechanisms to govern the use of the OSP funds. Rapid
expansion and new demands put at risk WSF's accountability for its use
and safeguarding of federal funds. This accountability risk was
increased by high turnover in key staff positions. WSF did not develop
the comprehensive and detailed fiscal policies and procedures needed
for internal control over its financial and programmatic operations.
WSF's draft policies and procedures lacked key control activities and
did not detail procedures for approving scholarship payments or require
documentation that procedures were carried out as required. WSF's
automated financial systems were not interfaced, and reliance on manual
processes between automated systems increased the risk of error.
Serious weaknesses in the implementation of internal controls affected
the processing of OSP scholarship payments, reconciliations between
financial records and bank statements, and assurance that participating
schools were financially responsible. For example, while WSF
established policies to help ensure that OSP funds are used to pay for
services in accordance with the Act establishing the program, WSF's use
of OSP funds to pay tuition for students attending schools that do not
normally charge students tuition is not in accordance with the Act.
However, WSF maintains that students attending these schools are
supported by scholarship dollars just as those who receive OSP funds.
Furthermore, our review questions certain other payments, but we were
unable to determine if they were allowable because these payments
lacked documentation.
Despite efforts to recruit eligible applicants and private schools and
inform parents, WSF faced challenges recruiting students from schools
in need of improvement, ensuring private school placement opportunities
and quality, and providing parents with accurate information regarding
private schools. WSF designed procedures to ensure that scholarship
recipients met program financial eligibility standards and income data
reported by parents of participants were broadly consistent with income
guidelines. While students from schools in need of improvement had
priority for receiving scholarships, for each year the program has
operated, the percentage of students from schools in need of
improvement who were offered scholarships has been smaller than the
percentage of students in District public schools who attended such
schools. Although most private schools in the District officially
participated in the program, the schools varied widely in the number of
openings available to scholarship students, and very few openings were
available at the secondary level. In school year 2006-2007, more than
half the scholarship students were concentrated in 19 participating
private schools, and 16 private schools had enrolled five or fewer OSP
students. The extent to which private schools had characteristics
associated with high-quality educational programs has varied. For
example, some schools reported that some of their teachers lacked at
least a bachelor's degree. Furthermore, some participating schools did
not meet basic requirements to operate in the District. For example, a
few had no certificate of occupancy on file with the District or had
certificates that did not specify educational use. Despite important
variation among schools, the grantee did not always provide parents
with complete and accurate information on their characteristics. For
example, WSF provided parents inaccurate information on teachers'
qualifications and tuition for some schools.
The evaluation contractor developed a strong methodology that used
random assignment to strengthen comparisons between students offered
and not offered scholarships, but factors related to program
implementation limited the ability to perform the comparisons directed
by the Act and the usefulness and generalizability of evaluation
findings. For example, although the Act directed the evaluators to
compare the achievement of participating eligible students who used
scholarships with students in the same grade in District public
schools, the evaluation contractor could not reasonably or cost-
effectively make these comparisons. Although the evaluation contractor
administered the same tests to measure achievement as were given by the
District in the first year, the District adopted a new achievement
testing program during the second year of the program for students in
grades 3-8 and grade 10. This meant that a common test was no longer
available for the program's 5 years of operation and made it necessary
to pay the evaluation contractor to test more students than planned
because, in addition to testing students with OSP scholarships in
private schools, the contractor also had to test eligible applicants in
District public schools who were not offered scholarships. Also, the
Act called for comparisons of dropout rates, graduation rates, and
college admission rates of scholarship program participants and
nonparticipants, but the limited placements offered by high schools
restricted the number of high school students and graduates in the
evaluation and eliminated the evaluator's ability to make these
comparisons. Because the WSF awarded many scholarships nonrandomly and
to some students who already attended private schools, the number of
students who could be included in the evaluation was diminished,
reducing the ability of the evaluation's statistical tests to identify
differences between groups and the strength of possible findings. Other
factors arising from the operation of the program, including the
inability of some students who were offered scholarships to find
placements and the provision of school or community based supplemental
services to some scholarship recipients might have made the
interpretation of findings difficult and also might have limited the
usefulness of the evaluation to programs in other jurisdictions.
To improve OSP, we are making several recommendations to the Secretary
of Education and the Mayor of the District of Columbia. To better focus
resources on achieving the program's goals of providing accountability,
quality educational experiences, and allowing parents to make an
informed school choice for their child, we recommend that the Secretary
direct the grantee to establish policies and procedures that improve
certain internal controls, continue efforts to integrate its financial
systems, improve monitoring, and provide accurate and complete
information to parents. We are also recommending that the Secretary and
the Mayor of the District of Columbia take specific steps to improve
private school compliance with applicable District requirements and
improve oversight of the program. Finally, we recommend that, in
planning for future programs for which Congress has required an
evaluation, Education should oversee the program to ensure consistency
between program implementation and the evaluation design.
We sent this report to the Office of the Mayor of the District of
Columbia, Education and WSF for review and comment. (Comments are
reprinted in appendixes III, IV, and V, respectively.) These entities
generally agreed with our recommendations, but Education and WSF
disagreed with many of our findings. Education officials raised
concerns related to scope of the assignment and stated that the draft
report did not present a complete and balanced picture of issues such
as the extent to which OSP students previously attended schools in need
of improvement. In addition, Education provided comments indicating
that GAO should have focused on the percentage of students whose former
schools were identified for improvement by the end of the students'
first year in the OSP. We disagree. Scholarship lottery organizers
could not be expected to give priority to students from schools that
had not yet been identified. Moreover, regardless of which year is
used, the percentage of students from schools in need of improvement
was smaller among scholarship recipients than among the general
population of District students from whom scholarship recipients could
have been recruited. Education also indicated we should have provided
more information on the speed with which the program was implemented,
satisfaction among parents, demand for scholarships, and participating
schools' traditional independence in areas such as hiring and
establishing teacher qualifications. As discussed further in our
response beginning on page 46, we believe that these subjects were
addressed in a complete and balanced fashion to the extent that they
were included in the scope of our review. Moreover, we found that lack
of reliable data on applicants limited the analysis we could perform to
determine the demand for scholarships. We agree that participating
schools had independence in areas such as hiring and establishing
teacher qualifications, but note that participating schools are also
required to be in compliance with District education and safety
requirements. In its comments, WSF also acknowledged that some of GAO's
recommendations, findings, and observations were valid and useful and
that it had taken or will take action to address some of them but
disagreed with many of our observations, interpretations, and findings.
While WSF implies that the deficiencies noted in this report are minor
in nature and disagreed with our conclusion that the Act prohibited
payments to schools that customarily charged no tuition, we stress that
the issues we identified are indicative of potential problems and if
not addressed could have a material, detrimental effect on WSF's
accountability over federal funds, especially when combined with the
accounting systems and cash reconciliation weaknesses that existed. As
we noted above in our response to Education, we believe the findings
and interpretations are accurate as stated in the draft and therefore
have not made changes other than technical corrections.
Background:
The D.C. School Choice Incentive Act:
The D.C. School Choice Incentive Act of 2003 directs the Secretary of
Education to award a grant on a competitive basis for up to 5 years to
an entity or entities to use to make scholarship payments to parents of
eligible students to be used for private school tuition, fees, and
transportation expenses. The Act directs the grantee to ensure that the
amount of tuition or fees that a participating private school charges a
scholarship student does not exceed the tuition or fees that the school
customarily charges to students not participating in the scholarship
program. Under the Act, Education could not approve a request for a
grant unless the entity's application detailed how it would, among
other things:
* address the priorities identified in the Act--that is, giving
priority to students who attended schools identified as in need of
improvement,[Footnote 4] targeting resources to families that lacked
resources to take advantage of available educational options, and
providing students and families with the widest range of educational
choice;
* ensure that if more eligible students seek admission in the program
than the program can accommodate, eligible students are selected for
admission through a random selection process that gives weight to the
program priorities;
* notify parents of eligible students of expanded choice opportunities
and ensure that parents receive sufficient information about their
options to allow the parents to make informed decisions;
* determine the amount that would be provided to parents for the
tuition, fees, and transportation payments, if any;
* seek private elementary and secondary schools in the District to
participate in the program, and ensure that participating schools would
meet the application and reporting requirements of this Act;
* ensure that participating schools were financially responsible and
would use funds effectively; and:
* address the renewal of scholarships to participating eligible
students, including continued eligibility.
The Act limits payments for each scholarship student to $7,500 per
year. If tuition exceeds the $7,500 cap, schools may supplement the
$7,500 with private scholarships or donated funds, absorb the
difference, or work out payment plans with parents or guardians.
According to the program requirements established by WSF, participating
private schools may apply their own admission standards and decide
whether to admit students who have received an OSP scholarship. The
program's funding allowed scholarships for about 2,000 District
students out of a public school population of about 72,000. (App. II
presents a comparison of the District of Columbia school choice program
and programs in several states and cities.)
The Department of Education's Office of Innovation and Improvement
awarded a grant to operate the program to WSF, a nonprofit organization
in the District of Columbia that had experience providing privately
funded scholarships to low-income students. WSF offered its first
scholarships under OSP for the 2004-2005 school year, and the program
recently completed its third year of operation with the conclusion of
the 2006-07 school year. The Act required that scholarships be awarded
through lotteries if the number of eligible applicants exceeded the
number of private school placements available or the number of
scholarships that the program could accommodate.
In these lotteries, students were assigned different priorities for
receiving a scholarship. Students attending schools in need of
improvement received the highest priority, followed by students who had
attended a public school not in need of improvement, followed by
students from private schools. The number of District schools assigned
the designation of needing improvement has increased from 2003 to 2006,
and the designations are updated following the school year.
The Act also required that the Secretary award a contract to an entity
jointly selected by the Secretary and the Mayor of the District of
Columbia for an independent evaluation that would use "the strongest
possible research design" to determine the effectiveness of the
program. The evaluators were to provide (1) a comparison of the
academic achievement of participating eligible applicants who were
offered scholarships and those who were not; (2) a comparison of the
academic achievement of applicants who were offered scholarships and
children attending District of Columbia Public Schools; (3) comparisons
of drop-out rates, graduation rates, and college admission rates of
students who participate in the program with students in District
schools who do not participate; (4) a comparison of the safety of
schools attended by participants in the program and the schools
attended by students who do not participate in the program; (5)
measures of the effect of the program on District public schools and
their students; and (6) the success of the program in expanding choice
for parents. The Secretary awarded the evaluation contract to Westat
and its affiliates.
Additional Laws and Agreements Governing Administration of the OSP:
The Act required that Education and the District enter into a
memorandum of understanding (MOU) regarding the implementation of the
D. C. School Choice Incentive Program. It directed that the MOU conform
to requirements contained in the conference report accompanying the
Act. In that report, the conferees stated their expectation that the
MOU would include (1) strong accountability measures and specifications
for program performance evaluations; (2) specifications for a lottery
system providing for fair and unbiased acceptance of students into the
program; (3) provisions for joint oversight of the program's operations
by the Mayor of the District of Columbia and the Secretary of
Education; (4) the methodology for the evaluation and selection of
participating schools that have met the District of Columbia's current
licensure requirements; (5) the methodology for determining the tuition
and fees of participating schools, including the actual cost; (6)
requirements for the development of appropriate oversight and
accountability measures; and (7) teacher quality criteria.
The MOU reflected this language in part. It identified Education's
Deputy Under Secretary of Innovation and Improvement as having lead
responsibility for implementation of the OSP, Education's Institute of
Education Sciences as having lead responsibility for implementation of
the evaluation, and the Mayor or his designee as having lead
responsibility for carrying out cooperative activities between the
District and Education. It also specified that Education and the Mayor
would jointly oversee the program and ensure it was carried out in
accordance with statutory requirements, the approved grant application,
and sound management and educational principles. Also, the MOU
conditioned schools' participation, in part, on "operating lawfully
within the District of Columbia," -meaning that participating schools
meet all the District of Columbia's current requirements. For example,
the Code of District of Columbia Municipal Regulations requires that
for purposes of the District's compulsory attendance law, private
schools must provide evidence that they are currently accredited by or
undergoing the accreditation process from one of seven named
organizations or any other accrediting body approved by the Board of
Education, or that they submit evidence of their educational soundness
acceptable to the Board.[Footnote 5] Accreditation provides assurance
that a school's course credits can be counted toward a District diploma
or accepted by other schools. Operating lawfully in the District of
Columbia also entails meeting additional District requirements, such as
securing a certificate of occupancy to demonstrate compliance with
health, safety, and fire code requirements. WSF's grant application
indicated that it would require participating private schools to
"attach their Certificate of Occupancy [to their letter of intent to
participate] and document that they are in compliance with all relevant
D.C. health, safety, and fire code requirements." Further, the MOU
stated that the grantee was to provide information on each
participating private school to parents of students offered
scholarships including, but not limited to, teacher qualifications, the
academic achievement of the school's students, and the safety and
environment of the school. However, the MOU did not include a
methodology for determining actual tuition and fees, or specify
oversight and accountability measures and teacher quality criteria.
Initiation and Funding of the Opportunity Scholarship Program:
Soon after the D.C. School Choice Incentive Act was passed, Education
awarded the grant to operate the OSP to the Washington Scholarship
Fund, which was the only entity that submitted a complete proposal.
Table 1 shows the timing of key events in the implementation and
operation of OSP.
Table 1: Key Events in Implementing and Operating the OSP:
Date: January 23, 2004;
Event: Congress enacted the D.C. School Choice Incentive Act of 2003.
Date: February 3, 2004;
Event: The District's Mayor selected Fight for Children to serve as the
interim, nonprofit entity charged with conducting early outreach and
publicity efforts.
Date: March 24, 2004;
Event: Education awarded the grant to Washington Scholarship Fund to
operate the DC School Choice Program.
Date: March 26, 2004;
Event: Education awarded a contract to provide technical assistance for
placement lotteries.
Date: Late March 2004;
Event: Outreach efforts and preliminary data on interested families
were transferred from Fight for Children to WSF.
Date: Late March-late August 2004;
Event: WSF developed application and eligibility determination
processes, and program management systems and continued multifaceted
recruitment and outreach efforts.
Date: April 16, 2004;
Event: The evaluation contractor and its affiliates submitted a
proposal to conduct the evaluation.
Date: Mid-April -Mid-May 2004;
Event: WSF collected its first applications from potential scholarship
families and held orientation meetings at the Washington Convention
Center.
Date: April-May 2004;
Event: Families submitted applications, and WSF and its contractor
processed applications and conducted eligibility determination.
Date: May 11, 2004;
Event: Deadline for private schools to report available number of
openings by grade for first school year of program.
Date: June 17, 2004;
Event: WSF conducted lottery for awarding year one scholarships.
Date: July 21, 2004;
Event: Education awarded a contract for the evaluation of the program
to the firm contracted to provide technical assistance on placement
lotteries.
Date: August/September 2004;
Event: First school year of Opportunity Scholarship program began with
1,027 OSP students enrolled in private schools.
Date: May 2005;
Event: WSF held lotteries for awarding scholarships.
Date: August/September 2005;
Event: Second school year began;
1,661 OSP students were enrolled in private schools.
Date: March 2006;
Event: Deadline for parents to apply for a scholarship for the 2006-07
school year; only children entering grades kindergarten through 5th
grade could apply for new scholarships.
Date: August/September 2006;
Event: Third school year began, with about 1,743 OSP students enrolled
in private schools.
Date: December 20, 2006;
Event: Congress passed legislation raising the eligible income level to
300 percent of the poverty line for students who received scholarships
during the first two years of the program.
Date: June 2007;
Event: Education released Evaluation of the DC Opportunity Scholarship
Program: Impacts After One Year.
Source: GAO analysis of information from Education and WSF.
[End of table]
As shown in table 2, $14 million was initially appropriated to
Education in each of fiscal years 2004 through 2006. Each year's
appropriation was reduced through subsequent rescissions. Table 2 also
shows the flow of funds from Education to the program. Specifically, of
the funds for fiscal year 2006, Education received $13.86 million. Of
this amount, about $12.45 million was awarded to WSF in the form of a
federal grant to operate the OSP. The amount of funding retained by
Education, $1.4 million, was provided to Education's Office of
Innovation and Improvement for program administration and oversight
($0.1 million) and to Education's Institute of Education Sciences for
an evaluation of the program ($1.3 million) as mandated by the Act.
Funds disbursed to the grantee, WSF, are to be expended for tuition,
fees, transportation, and program administration. According to WSF, the
majority of funds donated by private sources are used for eligibility
determination, infrastructure, and other technical systems development
and maintenance, and student placement.
Table 2: Opportunity Scholarship Program Federal Funding for Fiscal
Years 2004-2006:
Sources and disbursement of funds: Appropriation to Education: Federal
appropriation;
2004: $14,000,000;
2005: $14,000,000;
2006: $14,000,000.
Sources and disbursement of funds: Appropriation to Education:
Rescission[A];
2004: (82,600);
2005: (112,000);
2006: (140,000).
Sources and disbursement of funds: Appropriation to Education: Net
appropriations to the U.S. Department of Education;
2004: 13,917,400;
2005: 13,888,000;
2006: 13,860,000.
Sources and disbursement of funds: Disbursement of grant funds:
Department of Education: Institute of Education Sciences;
2004: 1,344,822;
2005: 1,308,640;
2006: 1,305,877.
Sources and disbursement of funds: Disbursement of grant funds:
Department of Education: Office of Innovation and Improvement (OII);
2004: 66,800;
2005: 100,000;
2006: 99,923.
Sources and disbursement of funds: Disbursement of grant funds:
Department of Education: Grant provided to WSF;
2004: 12,505,778;
2005: 12,479,360;
2006: 12,454,200.
Sources and disbursement of funds: Disbursement of grant funds:
Department of Education: Total funds disbursed to Education and WSF;
2004: 13,917,400;
2005: 13,888,000;
2006: 13,860,000.
Source: GAO analysis of appropriations laws and allotments data.
[A] Congress enacted across-the-board rescissions for appropriations
made for fiscal years 2004, 2005, and 2006. The rescissions were 0.59
percent for fiscal year 2004, 0.80 percent for fiscal year 2005, and 1
percent for fiscal year 2006.
[End of table]
The Act provided that the grantee may not use more than 3 percent of
the total amount provided under the grant each year for the
administrative expenses of carrying out the program during the year,
including (1) determining the eligibility of students to participate,
(2) providing information about the program and the schools involved to
parents of eligible students, (3) selecting students to receive
scholarships, (4) determining the amount of scholarships and issuing
the scholarships to eligible students, (5) compiling and maintaining
financial and programmatic records, and (6) providing funds to assist
parents in meeting expenses that might otherwise preclude the
participation of their child in the program. WSF officials defined
administrative expenses as any program expenses other than tuition,
fees, and transportation. These administrative expenses generally
included direct expenses, such as student eligibility reviews and
payroll, and indirect expenses, such as rent and utilities.
WSF's Accountability and Internal Control Were Inadequate for OSP, and
Some Payments Were Not Made in Accordance with the Act While Others
Raise Questions about Compliance:
After being selected as the OSP grantee, WSF struggled with developing
and implementing accountability mechanisms and internal controls in the
management of federal and foundation grants that increased from
$150,000 in fiscal year 2004 to $12.9 million in fiscal year 2006.
Further, with the addition of OSP to its operations, WSF was challenged
to find systems and approaches for community outreach, school
placement, family and school support, and financial management while
implementing the program. Due to its need to quickly implement the
program, WSF had little time to develop internal control policies and
procedures, systems, and internal control activities needed to provide
accountability over the use of grant funds for this new
program.[Footnote 6] WSF had to rely on the knowledge of its staff to
perform the daily operations for OSP, and its ability to rely on its
key staff was greatly affected by high staff turnover. The discovery of
fraud allegedly perpetrated by a WSF employee affecting one of its
private scholarship programs highlighted the need for strengthened
internal control activities. The lack of comprehensive policies and
procedures, the use of nonintegrated systems, and the outsourcing of
basic bookkeeping functions increased risk and weakened WSF's
accountability over OSP. Moreover, we found that WSF did not
consistently adhere to the basic accountability and internal control
procedures it had established for scholarship payments. In reviewing
OSP scholarship payments, we found that WSF's practice of paying for
tuition scholarships to schools that do not normally charge students
tuition is a violation of the Act because the Act provides that the
tuition charged OSP students cannot be more than the tuition charged
for similarly situated non-OSP students. Furthermore, we could not
determine whether payments for before-and-after care programs were
appropriate because the documentation provided by WSF did not indicate
that it had verified that the care programs met the criteria for
determining whether a fee is allowable. Based on the documentation
provided by WSF, we could not determine that WSF verifies that before-
and-after school care programs are tied to the student's academic
program and part of customary fees charged by the school, which are
Education and WSF's key criteria under the Act for determining whether
a fee is allowable.
WSF's Rapid Growth and High Staff Turnover Created Accountability Risk
for the Opportunity Scholarship Program:
Following its selection as a grantee in March 2004, WSF rapidly
expanded its scholarship operations to provide OSP scholarships to
students for the upcoming 2004-2005 school year.[Footnote 7] This rapid
expansion can be seen in the increase of WSF's annual federal and
foundation grants from $150,000 in fiscal year 2004 to $8.6 million in
fiscal year 2005, and to $12.9 million in fiscal year 2006. (See table
3.) WSF, charged with running OSP, had to create systems for community
outreach, school placement, and family and school support while
implementing the program, and WSF faced the challenge of accomplishing
school search and enrollment activities before the beginning of the
2004-2005 school year. WSF was also challenged to find ways in which
thousands of OSP checks could be correctly generated, accounted for,
and monitored for compliance with statutory requirements. With this
rapid expansion, WSF had little time to design and implement the needed
systems, policies, procedures, and internal controls for managing such
a major increase in its operations. While WSF was able to process the
OSP scholarship payments for school years 2004-2005 and 2005-2006, it
used systems that were not integrated, resulting in data being manually
transferred, and thereby increasing the risk of error. Also, WSF's
basic bookkeeping and accounting functions, including the printing of
OSP scholarship checks and preparation of bank reconciliations, were
not integrated, contributing to the fragmentation of WSF's OSP
operations and weakened accountability.
Table 3: Growth in WSF from Fiscal Years 2003--2006:
Financial statement line item: Federal and foundation grants[A];
Fiscal year ending June 30: 2003: --[B];
Fiscal year ending June 30: 2004: $150,000;
Fiscal year ending June 30: 2005: $8,563,995;
Fiscal year ending June 30: 2006: $12,893,898.
Financial statement line item: Contributions and grants;
Fiscal year ending June 30: 2003: $1,801,251;
Fiscal year ending June 30: 2004: $1,929,707;
Fiscal year ending June 30: 2005: $1,576,706;
Fiscal year ending June 30: 2006: $1,793,409.
Total;
Fiscal year ending June 30: 2003: 1,801,251;
Fiscal year ending June 30: 2004: 2,079,707;
Fiscal year ending June 30: 2005: 10,140,701;
Fiscal year ending June 30: 2006: 14,687,307.
Financial statement line item: Scholarships expense[C];
Fiscal year ending June 30: 2003: 1,967,480;
Fiscal year ending June 30: 2004: 2,268,942;
Fiscal year ending June 30: 2005: 9,043,594;
Fiscal year ending June 30: 2006: 14,108,731.
Financial statement line item: Expenditures of federal awards;
Fiscal year ending June 30: 2003: --[D];
Fiscal year ending June 30: 2004: -- [D];
Fiscal year ending June 30: 2005: 6,570,407;
Fiscal year ending June 30: 2006: 11,664,898.
Source: WSF audited financial statements for fiscal years ending June
30, 2003, 2005, and 2006.
[A] While OSP is a major federal program for WSF, WSF also has received
a 2-year "Next Generation" grant from the Corporation for National and
Community Service. This 2-year grant is designed to help organizations
build programs in community service and volunteerism.
[B] Beginning with fiscal year 2004, WSF reclassified contributions and
grants for comparative purposes to show the federal and foundation
grants separately from contributions.
[C] This expense includes scholarships paid by WSF under both its
publicly funded OSP and its Signature Scholarship Program, which is
privately funded.
[D] Single audits, which include the schedule of expenditures of
federal awards, are required for grant recipients who expend $500,000
or more in federal awards in each year. These amounts are not available
because WSF expended less than $500,000 in federal grants and therefore
was not required to submit a single audit report for fiscal years 2003
and 2004.
[End of table]
WSF's rapid growth, the changing nature of its systems and processes
for OSP scholarship payments, and the lack of comprehensive and
detailed procedures meant that WSF had to rely on the knowledge of key
staff to perform daily operations. However, the ability of WSF to rely
on institutional knowledge was greatly affected by staff turnover in
key positions. (See fig.1.) The Chief Financial and Operations Officer
resigned in May 2006. WSF's Accounting and Grants Manager resigned in
January 2007 and the Director, Finance and Operations resigned in
February 2007. The Senior Director Education and Outreach left WSF
during that time. The President and Chief Executive Officer then left
in May 2007. The Senior Director for Scholarship Operations--a key
person who understood WSF's scholarship payment processes for the OSP-
-resigned in June 2007. Staff turnover at WSF, combined with the lack
of established accountability procedures and systems, has significantly
increased WSF's overall fiscal and program operations risk.
Figure 1: WSF Staff Turnover between May 2006 and June 2007 Based on
Positions in Place in August 2006:
This figure is a chart showing WSF staff turnover between May 2006 and
June 2007.
[See PDF for image]
Source: Washington Scholarship Fund.
[End of figure]
WSF has recently increased its staff significantly. A comparison of
figure 1 and figure 2 shows this increase in staff resources from 11
staff in May 2006 to 17 staff in August 2007. Staffing is supplemented
by interns who work with the full-time staff on both of WSF's
scholarship programs.
Figure 2: Reporting Structure Among WSF Staff as of August 2007:
This figure is a chart showing reporting structure among WSF staff as
of August 2007.
[See PDF for image]
Source: Washington Scholarship Fund.
[End of figure]
WSF's Accountability Was Further Weakened by a Lack of Comprehensive
Fiscal Policies and Procedures and Use of Nonintegrated Systems for
Basic Bookkeeping and Accounting Functions:
WSF's accountability over OSP scholarship payments was further weakened
by a lack of comprehensive and detailed policies and procedures to
guide its processing of receipts and payment of scholarships and
administrative expenses. For the fiscal year ended June 30, 2005, WSF's
external auditor reported that WSF was in the process of developing a
formal accounting manual documenting its procedures. The auditor stated
in a letter to WSF management that this type of accounting manual is
necessary to ensure that transactions are treated in a standardized
manner. In its letter, the auditor also noted that written procedures,
instructions, and assignments of duties will prevent or reduce
misunderstandings, errors, wasted effort, duplicated or omitted
procedures, as well as other situations that can result in inaccurate
or untimely accounting records. The auditor recommended that WSF
include in its accounting manual descriptions of each fiscal procedure-
-including payment of invoices, maintenance of accounts receivable
subsidiary ledgers, and payroll procedures.
In February 2007, WSF provided us with a copy of its Internal Systems
Review, which was still in draft form and provided only general process
descriptions. The document was created in response to the
recommendation from its external auditor. WSF officials told us that
its operations team would be continually writing and editing this
document to further encourage organizational and financial integrity in
its daily operations. The officials added that they expected the fiscal
policies and procedures document to be final once WSF's new nonprofit
financial management software is operational. As of late June 2007, WSF
officials stated that this new software would be operational for
processing the 2007-2008 scholarship payments. The February 2007 draft
provides a high-level description of the processes for receipts, OSP
and other scholarship payments, administrative expenses, human
resources, and budgeting/ bank reconciliations. However, without
extensive on-the-job training, the descriptions in this document are
not adequately detailed for use in carrying out the procedures or for
use in holding individuals responsible for key internal controls and
accountability.
The policies described in the draft Internal Systems Review are not
specific and do not include controls over key activities for handling
WSF's funds and payments or the specific internal controls needed to
help ensure that key control activities are being carried out. For
example, the procedures state that bills submitted by WSF's vendors are
to be approved by the financial team, but the procedures do not include
how the approval is to be documented and how the payment of the invoice
is to be recorded in WSF's accounting software. Also, in the section on
procedures for sending the OSP scholarship payments data to the
contractor for printing checks, there is no mention of controls and
procedures to verify the accuracy of the payment amounts and the
printed checks. While segregation of duties[Footnote 8] is mentioned in
the Internal Systems Review document for processing receipts and for
issuing checks to WSF's vendors, these procedures do not clearly
outline responsibilities for (1) authorizing transactions, (2)
processing and recording them, (3) reviewing the transactions, (4)
handling any related access to cash, and (5) how these duties are to be
separated. The control achieved through segregation of duties is
important for small as well as large organizations. Where the staff is
not large enough to segregate all duties, a compensating control, such
as supervisory review, can be established. Segregation of duties is
among the controls designed to prevent such fraudulent acts as those
allegedly committed by a former WSF staff member in processing WSF's
private scholarship funds (no OSP funds were involved, according to WSF
officials) between October 2006 and January 2007. The manager diverted
funds from WSF's Signature Scholarship Program (SSP) bank account and
made personal charges to a WSF credit card. A WSF official subsequently
identified the fraudulent activity through reviews of bank
reconciliations and credit card charges. WSF management told us that it
has implemented additional controls to help ensure that access to any
accounts or funds is secure and that these controls will be tested as a
part of WSF's next financial statement audit.
WSF processed OSP scholarship payments using systems that were not
interfaced, resulting in data being manually transferred, a procedure
that increases the risk of error. For school years 2004-2005 and 2005-
2006, due to limitations in WSF's accounting system, WSF used an Excel
spreadsheet to track information about individual OSP scholarship
payments. However, the detailed payment and check information in the
Excel spreadsheet could not be uploaded or interfaced with the
accounting software used by WSF. As a result, only the lump sum totals
for batches of payments were recorded in the WSF accounting software,
not the individual scholarship check information. This system was used
as the basis for WSF's financial reports.
WSF enhanced its scholarship payment process for 2005-2006 through the
implementation of a Web-based, online billing system that enabled
participating schools to electronically enter the tuition and fee
charges and submit invoices for all OSP students attending that
school.[Footnote 9] WSF also created a student database that included
key information about each student. However, because the online billing
system was not integrated with the student database, key information
about each OSP student had to be uploaded electronically into the
online billing system to enable participating schools to submit
invoices.
WSF's overall process for key bookkeeping and accounting functions,
such as check printing and bank reconciliation, was not well defined or
integrated. Further, these processes were outsourced, requiring
monitoring and oversight of contractor's work to ensure against errors.
For example, the bank reconciliation process--a key cash management
control--was made more difficult by the tracking of individual payments
in the Excel spreadsheet or student database, neither of which was
integrated with the accounting system. The lump sum recording of
payments also made it difficult to track the separate manual entries in
its accounting software for voided checks, reissued checks, and other
adjustments, because individual OSP scholarship checks were not
directly matched with corresponding entries. Therefore, the preparation
of bank reconciliations was difficult. In 2007, WSF officials noted
errors in recent bank reconciliations. These errors, combined with
WSF's detection of fraud allegedly perpetrated by the former WSF
employee in its nonfederal program, have prompted WSF to begin a
process of redoing all bank reconciliations, starting with 2004. As of
the end of our field work, this effort had not been completed and
therefore the outcome is unknown.
WSF officials have contracted to purchase a financial management
software package designed for nonprofit organizations that they believe
will facilitate the recording of the individual scholarship checks.
According to WSF, the system should be in place for the 2007-2008
school year. As WSF continues to implement integrated systems and
processes, the need for manual data transfers will be reduced.
WSF Did Not Consistently Adhere to Basic Accountability and Internal
Control Procedures:
WSF did not consistently adhere to its established OSP scholarship
payment procedures and made payments without the required documentation
or approvals, and it did not fulfill its oversight role for ensuring
that participating schools are financially responsible. Lack of
appropriate accountability and internal control increases the risk that
program funds will not be used in accordance with program requirements.
On the basis of our examination of supporting documentation for
payments for a random sample of 50 students receiving scholarships at
25 schools in school year 2005-2006, we found the following problems
with scholarship payments made for 46 of the 50 students. Payments for
some students had more than one of these problems.
* For 39 of the 50 student files we reviewed, WSF had paid fees based
on incomplete fee approval forms from the participating schools, which
lacked authorizing signatures from the participating school, dates, or
both. Participating schools submit these forms to obtain WSF's approval
of fees that are not included in the school's published list of tuition
and fees. According to WSF's procedures, these forms must be reviewed
and approved by WSF officials before payment can be made. Without
having properly approved fee information, WSF is at risk of paying
unapproved fees.
* For 9 of the 50 students, we found related student or school files
were incomplete. Some of the items that were missing included
disbursement and payment detail reports as well as other documents such
as school placement letters. Disbursement reports provide a summary of
the tuition and fees that have been billed to the student's scholarship
account for the year. Payment detail reports list all tuition and fees
for each OSP student attending the participating school, and the
correctness of the information on these reports is to be certified by
officials of the participating schools. Without this information, we
could not assess whether the controls were operating as intended.
* In 23 of the 50 student files we reviewed, disbursement reports did
not include the WSF student identification number next to the student's
name or the student identification or name was incorrect. Although WSF
maintains that its student identification numbers are not used as key
identifiers, these student numbers are the only numbers that link the
disbursement reports and the scholarship checks. Without a consistent
student identifier to link students in the program with the actual
payments being made, WSF is at risk of losing accountability for
specific payments made for individual students in the program.
* We found a scholarship payment that WSF paid for a fee that was not
included on the payment detail report and a fee payment for another
student that was based on a payment detail report that had not been
certified by a school official. Without this certification, WSF
officials did not have assurance from the participating school that the
payment was made based on correct and complete information.
Some Payments Are Not in Compliance with the Act, and Others Raise
Questions about Compliance:
Under the Act, the grantee is to use the funds to provide eligible
students with scholarships to pay tuition, fees, and transportation to
enable students to attend the D.C. private school of their choice.
Neither the Act nor its legislative history defines the terms
"tuition," "transportation," or "fees." According to Education
officials, the Department left the determination of what fees are to be
included in a scholarship payment to WSF. WSF established the policy
that any fee for an activity, service, or product that contributes to
the academic success of a student is allowable if two key criteria are
met. One criterion is that fees paid must be for services offered to
all similarly situated students, not just OSP students. The other
criterion was that the rates of all fees for OSP students must be the
same as rates charged for similarly situated, non-OSP students. WSF
commonly uses OSP scholarship funds to pay fees for school uniforms,
books, cafeteria services, tutoring, before-and-after care, physical
education, and enrichment (music, dance, and art).
During the 2006-2007 school year, WSF offered scholarships up to $7,500
to students enrolled in schools that do not normally charge students
tuition. For school year 2006-2007, WSF offered such scholarships to
about 30 students. At these schools, families of non-OSP students
typically pay a small monthly fee as a sign of commitment to the
school, but tuition and other expenses are supported by private
donations to the school. OSP is providing scholarship payments of
$7,500 per year to pay tuition for OSP scholarship children to attend
these schools while the families of non-OSP students pay no tuition.
Section 307(a)(1) of the Act requires that the grantee ensure that the
amount of any tuition or fees that the school customarily charges OSP
students not exceed the amount of tuition or fees that the school
customarily charges non-OSP students. In reviewing OSP scholarship
payments, we found that WSF's practice of paying for tuition
scholarships to schools that do not normally charge students tuition is
a violation of the Act because the Act provides that the tuition
charged OSP students cannot be more than the tuition charged for
similarly situated non-OSP students. The small monthly fee, because it
is charged to all students, would be an allowable fee under the OSP.
WSF maintains that there is no violation of the Act because students
attending these schools tuition-free are supported by scholarship
dollars in the same way as those who receive OSP funds.
During school year 2005-2006, WSF was billed for before-and-after
school care for 31 of the 50 students whose payments we reviewed. On
the basis of the documentation that WSF provided to us for the 25
schools in our sample that offered before-and-after care, in 22 cases
we could not determine whether these services included any academic
support activities. Absent this characteristic, before-and-after care
would not meet the requirement that a fee must contribute to a
student's academic success.
While OSP Students Generally Met the Program's Eligibility
Requirements, WSF Had Limited Success Meeting Recruiting Priorities and
Did Not Provide Parents Complete Information:
WSF had limited success in recruiting students from schools in need of
improvement, ensuring private school placement opportunities and
quality, and providing parents accurate information on private schools.
Students who were offered OSP scholarships generally met the program's
income eligibility requirements, and applicants from schools in need of
improvement had priority for receiving scholarships, but low numbers of
scholarship recipients came from such schools. While a large number of
schools were recruited to participate, they varied widely in the
numbers of placements they made available and characteristics
associated with quality, such as teachers' qualifications, and
information WSF provided parents was not always complete and accurate.
Grantee Recruited Eligible Students, but Low Numbers Were from Schools
in Need of Improvement:
Data reported to the grantee generally indicated students who were
offered scholarships met the financial eligibility criteria. Despite
considerable outreach efforts and priority given to students from
schools in need of improvement, they were underrepresented among
recipients of scholarship offers relative to their presence in the
population of District students, owing to both challenges in recruiting
applicants and school openings.
Under the terms of the memorandum of understanding between Education
and the District of Columbia regarding the operation of the program,
the District, in consultation with Education, agreed to identify a
community-based nonprofit organization to conduct activities, including
outreach to parents and private schools, prior to the award of the
grant in order to promote a successful launch of the program for the
2004-2005 school year. This entity--Fight for Children--met with
private school officials, collected preliminary information about these
schools, and initiated community outreach activities. After the grant
was awarded, WSF assumed responsibility for these efforts and built
upon them by engaging in such activities as holding community meetings,
running radio advertisements, conducting mail campaigns, and reaching
out to minority communities. However, WSF encountered obstacles in
recruiting students from schools in need of improvement, and less than
a quarter of all students who were offered scholarships by 2006 came
from these schools. In fact, 4 percent of students who were offered
scholarships the first year were from schools in need of improvement,
while about 11 percent of District students attended regular public and
charter schools in need of improvement that year. Table 4 describes
some of WSF's outreach activities, and figure 3 shows examples of some
outreach materials used.
Table 4: Types of Outreach Methods Used to Reach Potentially Eligible
Families:
Method: Paid advertising;
Examples:
* Direct mail;
* Newspapers;
* Public transportation ads;
* Movie theater advertisements;
* Radio.
Method: Neighborhood meetings at a variety of locations;
Examples:
* District of Columbia Armory and convention center;
* Community centers;
* Churches;
* Housing developments;
* Libraries;
* Public housing;
* School-sponsored events;
* Homes.
Method: Direct mail to targeted audiences;
Examples:
* Temporary Assistance for Needy Families recipients;
* Families with students attending District of Columbia public schools
designated as in need of improvement;
* Commercial list of 27,000 low-income families;
* District of Columbia Public School and charter school mailings.
Method: Posters at a variety of locations;
Examples:
* Community centers;
* Clinics;
* Churches;
* Homeless shelters;
* Government services intake offices;
* Outreach centers targeting minority populations;
* Retail outlets (grocery stores, beauty salons).
Method: Flier distribution at a variety of locations;
Examples:
* Schools in need of improvement;
* Public transportation stops;
* Targeted neighborhoods;
* Retail outlets (grocery stores, beauty salons).
Method: Office accessibility;
Examples:
* Evening and weekend hours;
* Satellite offices;
* Bilingual interpreters, primarily Spanish;
* Volunteers;
* School and application fairs.
Source: GAO summary of WSF information.
[End of table]
Figure 3: Examples of Outreach Materials Grantee Used To Recruit Low-
Income Families:
This figure shows a combination of different outreach materials used to
recruit low-income families.
[See PDF for image]
Source: Washington Scholarship Fund.
[End of figure]
Various factors hindered WSF's efforts to recruit more students from
schools in need of improvement for the first year, according to WSF
officials. Officials said that many families were skeptical about the
program--stating that it was "too good to be true" or that private
schools would be reluctant to accept their children-- and had heard
misinformation, such as that receiving a scholarship could reduce
social service benefits. In addition, WSF officials said they only had
a short time after receiving the grant to recruit a large number of
potentially eligible families and help them through the complicated
application and school admissions process before they had to conduct
the first lottery.
By the second school year, 2005-2006, the percentage of students from
schools in need of improvement who were offered scholarships increased
to 38 percent. According to WSF officials, several factors contributed
to this growth: More families knew about the program and were less
skeptical than they had been in the beginning, and WSF had more time to
recruit eligible families. In addition, the percentage of children who
attended District public and charter schools designated as in need of
improvement increased, therefore enlarging the pool of potentially
eligible applicants from these schools.
WSF decided to reduce the extent of its outreach efforts for the third
year, 2006-2007, because of the limited availability of new
scholarships for students.[Footnote 10] Most of the grant funds were
needed to support students awarded scholarships in years one and two.
According to WSF officials, conducting extensive outreach efforts when
only a limited number of scholarships were available would have
unrealistically raised some families' hopes of receiving scholarships.
WSF cut back on media advertising, but mailed flyers to families with
children in schools in need of improvement and held community meetings
and events, such as school fairs.[Footnote 11] About 24 percent of
students who were offered new scholarships for the third year were from
schools in need of improvement. (See fig. 4.)
Lotteries for scholarship awards incorporated a mechanism to afford
required priority to students from schools in need of improvement, but
an applicant's probability of selection was also influenced by the
number of private school openings offered at his or her grade level.
This is significant because private school openings were most plentiful
at the elementary level, while improvement designations had been more
common among middle and secondary schools. In order to give priority to
students attending schools in need of improvement, the evaluator
assigned highest probabilities to those students that attended District
schools with this designation, lower priority to students from public
schools without this designation, and, for the first year of operation,
lowest priority to students from private schools. Because the lottery
also awarded scholarship offers in proportion to the number of possible
openings reported by participating schools at particular grade levels,
students in kindergarten through grade 8 had considerably better
chances of receiving scholarship offers than high school students
because many more openings were available for them.[Footnote 12] While
District middle and secondary schools were more likely to be in
improvement status than elementary schools, private school openings at
the secondary level were so scarce that WSF restricted application in
later years of the program to students in the elementary grades.
Figure 4: Percentages of Students Offered OSP Scholarships Who Attended
Schools Designated as in Need of Improvement Were Consistently Below
the Percentages of Students in District of Columbia Schools Who
Attended Such Schools:
This figure is a bar chart showing percentages of students offered OSP
scholarships who attended schools designated as in need of improvement
were consistently below the percentages of students in District of
Columbia schools who attended such schools.
The X axis represents the school/program year, and the Y axis
represents the percentage. In 2004-2005, 4% of students offered OSP
scholarships who were from schools in need of improvement. That same
school year, 11% of District public and charter school students
enrolled in schools in need of improvement.
In 2005-2006, 38% of students of students offered OSP scholarships who
were from schools in need of improvement. That same school year, 51% of
District public and charter school students enrolled in schools in need
of improvement.
In 2006-2007. 24% of students of students offered OSP scholarships who
were from schools in need of improvement. That same school year, 52% of
District public and charter school students enrolled in schools in need
of improvement.
Source: GAO analysis of data from WSF and the District of Columbia
Public Schools.
Note: The District of Columbia issued its list of schools in need of
improvement each August for the prior school year. Because WSF and the
evaluation contractor had to conduct scholarship lotteries in the
spring before the District issued the lists, they had to use prior year
designations. For example, they conducted the scholarship lotteries for
the 2004-2005 school year in spring 2004 but had to use the list of
schools designated as in need of improvement in 2002-2003 because the
District did not issue the 2003-2004 school list before the end of the
school year. Accordingly, in the figure above, both bars in each year
represent students from schools that were designated as in need of
improvement in the previous August.
[End of figure]
Of the 2,845 students who were offered scholarships during the first 3
years of the program, most were African-American, but some were
Hispanic, Asian, and white students,[Footnote 13] and almost all met
the income eligibility requirements.[Footnote 14] About 77 percent had
attended District public schools, with the remainder split among
students who had attended private schools, students who had not
previously attended school, and students from public and private
schools outside the District. Most students--about 80 percent--were
enrolled in elementary and middle schools,[Footnote 15] while about 17
percent were enrolled in grades 9-12.
Although most students used their scholarships when they received them,
many did not or did not continue to use them after the first year,
potentially freeing up scholarship money for some additional
scholarship offers, but students who had been offered scholarships
could decide to use the scholarships at any time during the program's
operation by enrolling or re-enrolling in a participating private
school. About 75 percent of all students who received scholarships
during the first 2 years used them at some point during the time they
were available to them. Among students offered scholarships in the
first year the program operated (cohort 1), 68 percent initially used
their scholarship, and of those who used them initially, 69 percent
used their scholarships all 3 years that the scholarships were
available to them and for which data had been collected. Among students
who were first offered scholarships in the program's second year of
operation (cohort 2), about 71 percent initially used their
scholarship, and of those who used them initially, 78 percent continued
to use their scholarships for the next 2 school years. One hundred
eleven students in cohort 1 and 5 students in cohort 2 did not use
their scholarships when first available, but did so eventually.
WSF had little data that described reasons students did not use their
scholarships when they first received them or continue to use them for
subsequent school years, but anecdotal information was available. The
reasons reported by some parents related to family issues, such as
personal problems, moving, and special education needs, while others
were more logistical, such as lack of transportation, before-or after-
care, or a convenient school. Parents also chose to enroll their
children in District charter schools, neighborhood schools, or other
private schools that did not participate in the program. According to
the Parent and Student Voices May 2007 report,[Footnote 16] some
parents also reported several factors that hindered their children's
ability to use the scholarship, such as losing their scholarship
because the family's income exceeded the previous 200-percent poverty
threshold for renewing scholarships,[Footnote 17] a dearth of openings
at the high school level, and receipt of a scholarship for only one
child in a sibling group.
Scholarship usage rates also varied according to the type of school the
student previously attended and students' needs for special educational
services. The program evaluation found that fewer students from schools
in need of improvement and from high schools in general used their
scholarships than other students. In addition, fewer students that
reported having a learning or physical disability used scholarships
than other students.
Participating Schools Ranged Widely in the Number of Openings Available
to Scholarship Recipients, and Few Openings Were Available at the
Secondary Level:
The 70 District of Columbia private schools that have participated in
the program varied in terms of their characteristics and the number of
OSP students they enrolled.[Footnote 18] Students who received
scholarships ranged from none to 67 percent of the enrollment of
participating private schools. For example, in 2006-2007, more than
half the scholarship students were concentrated in 19 schools. Sixteen
schools had accepted between 0 and 5 scholarship students. Four schools
accepted only students who they had previously accepted.
Some students who received scholarships had limited choices,
particularly students in the upper grades and those who wanted to
attend a secular school. For example, according to WSF data for school
year 2005-2006, only about 70 openings were available at the high
school level (compared to about 650 for students in kindergarten
through grade 5 and about 200 for students in grades 6-8). The majority
of scholarship students attending high schools went to one religious
school, and WSF raised private funding to pay for the tuition above the
scholarship cap. In addition, students who desired a secular school had
a limited number to choose from, since most of the participating
private schools were Catholic or Protestant, and these schools offered
most of the openings. The remaining schools included some that were
Afro-centric or Muslim, or offered only early childhood education.
About 88 percent of all scholarship users attended schools with
tuitions below the $7,500 cap. Although tuition rates varied, only 3
percent attended the most expensive schools that charged $20,000 or
more, perhaps because these schools or private donations could not make
up the difference between the tuition and the $7,500 cap or because
these schools had financial aid programs of their own on which they
could draw that were not subject to OSP financial and residential
eligibility requirements. In general, schools offering high school
placements had higher tuitions than other schools. (See table 5.)
Table 5: Summary of Tuition Rates by Grade Level for Participating
Private Schools, School Year 2006-07:
Tuition: $2,100-5,000;
Elementary: Number of schools: 4;
Elementary: Number of OSP students: 76;
Elementary and middle: Number of schools: 18;
Elementary and middle: Number of OSP students: 999;
Elementary through high school: Number of schools: 1;
Elementary through high school: Number of OSP students: 67.
Tuition: $5,001-7,500;
Elementary: Number of schools: 4;
Elementary: Number of OSP students: 95;
Elementary and middle: Number of schools: 7;
Elementary and middle: Number of OSP students: 190;
Elementary through high school: Number of schools: 2;
Elementary through high school: Number of OSP students: 97.
Tuition: $7,501-10,000;
Elementary: Number of schools: ;
Elementary: Number of OSP students: ;
Elementary and middle: Number of schools: 2;
Elementary and middle: Number of OSP students: 15;
Elementary through high school: Number of schools: ;
Elementary through high school: Number of OSP students: .
Tuition: $10,001-15,000;
Elementary: Number of schools: ;
Elementary: Number of OSP students: ;
Elementary and middle: Number of schools: 1;
Elementary and middle: Number of OSP students: 5;
Elementary through high school: Number of schools: ;
Elementary through high school: Number of OSP students: .
Tuition: $15,001-20,000;
Elementary: Number of schools: 2;
Elementary: Number of OSP students: 4;
Elementary and middle: Number of schools: 1;
Elementary and middle: Number of OSP students: 25;
Elementary through high school: Number of schools: ;
Elementary through high school: Number of OSP students: .
Tuition: $20,000 +;
Elementary: Number of schools: 1;
Elementary: Number of OSP students: 1;
Elementary and middle: Number of schools: 1;
Elementary and middle: Number of OSP students: 3;
Elementary through high school: Number of schools: 3;
Elementary through high school: Number of OSP students: 38.
Total;
Elementary: Number of schools: 11;
Elementary: Number of OSP students: 176;
Elementary and middle: Number of schools: 30;
Elementary and middle: Number of OSP students: 1,237;
Elementary through high school: Number of schools: 6;
Elementary through high school: Number of OSP students: 202.
Middle or junior high: Number of schools: [Empty];
Middle or junior high: Number of OSP students: [Empty];
Middle and high school or high school only: Number of schools: [Empty];
Middle and high school or high school only: Number of OSP students:
[Empty];
All schools: Number of schools: 23;
All schools: Number of OSP students: 1,142.
Middle or junior high: Number of schools: [Empty];
Middle or junior high: Number of OSP students: [Empty];
Middle and high school or high school only: Number of schools: [Empty];
Middle and high school or high school only: [Empty];
All schools: Number of schools: 13;
All schools: Number of OSP students: 382.
Middle or junior high: Number of schools: 1;
Middle or junior high: Number of OSP students: 15;
Middle and high school or high school only: Number of schools: 1;
Middle and high school or high school only: Number of OSP students: 72;
All schools: Number of schools: 4;
All schools: Number of OSP students: 102.
Middle or junior high: Number of schools: 1;
Middle or junior high: Number of OSP students: 12;
Middle and high school or high school only: Number of schools: 2;
Middle and high school or high school only: Number of OSP students: 14;
All schools: Number of schools: 4;
All schools: Number of OSP students: 31.
Middle or junior high: Number of schools: 1;
Middle or junior high: Number of OSP students: 3;
Middle and high school or high school only: Number of schools: 4;
Middle and high school or high school only: Number of OSP students: 5;
All schools: Number of schools: 8;
All schools: Number of OSP students: 37.
Middle or junior high: Number of schools: [Empty];
Middle or junior high: Number of OSP students: [Empty];
Middle and high school or high school only: Number of schools: 1;
Middle and high school or high school only: Number of OSP students: 2;
All schools: Number of schools: 6;
All schools: Number of OSP students: 44.
Middle or junior high: Number of schools: 3;
Middle or junior high: Number of OSP students: 30;
Middle and high school or high school only: Number of schools: 8;
Middle and high school or high school only: Number of OSP students: 93;
All schools: Number of schools: 48;
All schools: Number of OSP students: 1,738.
Source: GAO analysis of WSF data.
Note: In addition, three students attended preschools with tuition
between $7,501 and $10,000, and two students attended a preschool with
tuition between $5,001 and $7,500.
[End of table]
While the Act requires WSF, as the grantee, to ensure that
participating schools are financially responsible and are using the
received funds effectively, WSF did not fulfill its oversight role of
ensuring that participating schools are financially stable. WSF
designed letters of agreement that detailed the requirements for
participating schools, including the need to show evidence of financial
stability on an annual basis. However, for all 25 schools attended by
the 50 students in our random sample, we did not find certain documents
that should have been submitted by the schools showing financial
stability or that a financial audit had been completed.
Proof of financial stability is critical for providing assurance that a
participating school can maintain operations as an ongoing entity and
that OSP students can continue enrollment in those schools. During
school year 2006-2007, one school that enrolled 25 scholarship users
announced closure due to bankruptcy. Two schools that were members of
the Center City Consortium, a group of Catholic schools run by the
Catholic Archdiocese of Washington, closed in school year 2007-2008 due
to financial problems, but the 134 scholarship users affected by the
closures were offered placements in other consortium schools.
Presently, the archdiocese is exploring the possibility of converting 8
of the 12 schools in the consortium to charter schools by school year
2008-2009.
The extent to which private schools reported characteristics associated
with high-quality educational programs varied. We found that, as
reported in the School Directory, D.C. Scholarship Program: 2004-
05,[Footnote 19] at least 3 of 52 schools that participated that year
indicated that at least half of their teachers did not have at least a
bachelor's degree, and 6 schools indicated that about 10 to 20 percent
of their teachers lacked at least a bachelor's degree. Further, many of
the schools were not accredited, and there is no evidence that they
submitted evidence of educational soundness acceptable to the Board.
The MOU and the Act limit participation to private schools operating
lawfully in the District. Yet, neither Education, the District of
Columbia, nor WSF ensure that all participating private schools meet
basic requirements for operating legally in the District of Columbia,
and some schools have not met these requirements. WSF relies on school
officials to self-certify that they are operating lawfully, and the
District had not reviewed any schools to determine whether they had met
the District's requirements. We selected a non-probability sample of 18
schools using various criteria, including whether or not schools had
registered with the District's Department of Consumer and Regulatory
Affairs and whether or not schools were accredited by an agency
recognized by the District and asked the District to supply copies of
their occupancy certificates to us. District officials provided
documentation indicating that 3 of 18 schools we selected for review
for certificates of occupancy lacked them; 6 had permits that did not
specify use as a private school, child development center, or before
and after school care center; and 7 of the 18 appeared to have
occupancy permits that designated use as child development centers with
before and after school care for school-age children, but did not
reflect the operation of a private school. One school where OSP
students constituted over 60 percent of enrollment applied for an
occupancy permit for operating as a private school in March 2007 after
it had participated in OSP for 3 years. Because District officials
could not find any reports on file, they concluded several schools may
not have submitted required annual reports of operation providing basic
information on their curriculum, teachers' education, accreditation,
and school facilities. These officials also reported that the District
government had not previously played an active role in overseeing the
operations of WSF. Figure 5 shows examples of schools participating in
OSP that District officials reported did not have certificates of
occupancy specifying use as an educational facility on file with the
Department of Consumer and Regulatory Affairs.
Figure 5: Examples of Schools That Operated without a Certificate of
Occupancy Specifying Use as a Private Day School:
This figure is a combination of two photographs showing schools that
operated without a certificate of occupancy specifying use as a private
day school.
[See PDF for image]
Source: District of Columbia government.
[End of figure]
According to WSF officials, they had conducted school visits at 42 of
the 58 schools with OSP students. However, we found only one written
report documenting a visit to one of these schools. Selection of
schools to visit was based partly on certain triggers, such as parent
complaints. WSF officials told us that they had several versions of a
school review form used to document visits and provided us a copy of
the form used at the beginning of the 2005-2006 school year. However,
they informed us that the use of this form had been discontinued
shortly afterward. Site visits such as WSF's school visits are an
important control, especially given that schools self-certify that they
meet all regulations established for private schools under District
law. Self-certification without review to verify that the
certifications are factual increases the risk that federal funds
intended to allow children from low-income families to attend private
schools will result in some students attending schools that are not in
compliance with District law.
Information WSF Provided to Help Parents Select Private Schools Was
Incomplete and Contained Inaccuracies:
The Act required the grantee to provide parents with information that
would help them make informed decisions about selecting a school for
their child. The MOU between Education and the District of Columbia
provided that the grantee was to provide certain specific information
about the school to parents, including the qualifications of the
school's teachers, the achievement of its students, and its safety and
environment. The provision of information is important since many
students who were offered OSP scholarships had little experience with
private schools. Parents also said they would like to have information
that would help them evaluate private schools.[Footnote 20]
Although WSF compiled an annual directory to help parents during the
selection process, it did not collect or omitted or incorrectly
reported some information that would have helped parents evaluate the
quality of participating schools.[Footnote 21] WSF did not provide
information about the achievement of each school's students in order to
help parents select appropriate schools for their children. However, it
did require schools to certify that they had given parents of enrolled
OSP students information on their children's academic progress and
overall school safety.[Footnote 22]
WSF also requested some information from the schools that it did not
provide that could have helped parents during their decision-making
process, most notably information on a school's accreditation status.
In addition, some information WSF did provide to parents may have been
misleading. Some schools reported to WSF that they had such facilities
as a library or gymnasium, although the information they reported to
the District indicated no such facilities.
Finally, WSF incorrectly reported information on some schools that
could have significantly affected parents' choice of schools, primarily
the percentage of teachers who had at least a bachelor's degree and
tuition rates. For example, the 2007-2008 school directory reported
that less than half of teachers in five schools and between 51 and 75
percent of teachers in another five schools had at least a bachelor's
degree, although the 2004-2005 directory reported that 100 percent of
teachers in these schools had at least a bachelor' degree. In addition,
the school directory for 2007-2008 reported that the tuition for at
least three private schools was between $10,000 and $15,000. However,
in verifying tuitions listed in the directory by comparing them to
tuitions cited in other sources, we learned these schools do not charge
non-OSP students tuition and typically charge only a modest fee, such
as $25 per month.[Footnote 23],[Footnote 24] The tuitions listed for
these schools may have motivated parents to obtain scholarships in the
belief that they could not otherwise afford them.
Evaluation Contractor Adopted a Strong Methodology, but Program
Implementation and Other Factors Limited the Usefulness and
Generalizability of Findings:
The evaluation contractor adopted an evaluation design that used random
assignment to determine which applicants were offered scholarships and
which were not, but program implementation decisions and other factors
limited the usefulness and generalizability of the evaluation's
findings regarding the effects of scholarship offers on the achievement
of low-income, District public school students. The contractor adopted
a random assignment evaluation design that enabled comparisons between
students from District public schools who were offered scholarships
(the treatment group) and those that were not (the control group) and
was designed to measure the effect of scholarship offers on applicants
who had attended public schools. The contractor used statistical
techniques to estimate the effects of using such a scholarship. The
methodology adopted was a strong experimental methodology that could
have produced an evaluation that reflected statutory requirements.
Many factors have limited the evaluator's ability to make the
achievement comparisons described in the statute and have complicated
interpretation of the evaluation's findings or their usefulness and
generalizability. For example, changes in the District's achievement
testing program and the scarcity of placements offered by participating
high schools limited the evaluators' ability to make comparisons that
were of interest to Congress between students offered scholarships and
their peers in District schools, including comparisons with regard to
graduation rates, dropout rates, and college placement. In addition,
although the Act directed the evaluator to determine the effects of the
program on District regular public schools, the opportunities for
choice within the public school sector grew over the same time frame,
making it very difficult, if not impossible, to disentangle the effects
of this program on regular public schools from the effects of charter
schools, special schools, and magnet and special programs that serve
far more students. Along with changes in the District's testing
program, limited number of scholarship placements at the high school
level, and simultaneous growth of other choice options, other factors
may have limited the evaluator's ability to make requested comparisons
and to detect statistically significantly differences and generalizable
results. These factors include the inability to include some
scholarship recipients in the evaluation, crossover between treatment
and control groups, concentration of scholarship students in a subset
of schools, and the combination of academic and social supports with
scholarship offers. Given these differences, some comparisons relied on
limited data.
Changes in the District's testing program. Changes in the District's
testing program made it impractical for the contractor to perform some
comparisons that Congress had requested, greatly increased the cost of
testing, and will likely continue to decrease the quantity and
sufficiency of the follow-up achievement test data. At Education's
urging, the District changed its standardized achievement testing
program in spring 2006 from a norm-referenced test to a criterion-
referenced test to better comply with the requirements of the No Child
Left Behind Act, but this had major effects on the evaluation. The norm-
referenced test, like all norm-referenced tests, had compared students'
knowledge with respect to the knowledge of other students and sorted
and ranked students according to how much they know in comparison to
other students. Because students are ranked against each other, and
normalized on a bell curve, norm-referenced tests are constructed so
that scores at different grade levels are vertically equivalent and
scores of children at various grade levels can be combined to draw
comparisons. For example, a second and an eleventh grader who received
a percentile score of 75 would have equivalent scores because both
surpassed the scores of 75 percent of their grade peers. In contrast,
the new criterion-referenced tests, like all tests of this type,
measure whether students have mastered certain skills and concepts
and/or what they have learned in certain grades or classes. Scores on
criterion referenced tests do not yield equivalent scores that can be
readily combined to draw comparisons across grade levels.
Although the Act directed the evaluator to compare the achievement of
students offered scholarships with students of the same grade in public
schools and the achievement of scholarship users with students of the
same grade in public schools, the evaluation contractor could not
reasonably or cost-effectively make these comparisons. The District's
new testing program meant a common test that yielded comparable scores
for all grades, from the year eligible applicants entered the program
through the duration of the program, was no longer available, and
administering two testing programs to all students in the District
would not be economically or educationally feasible.
In addition, the change in the District's testing program made it
necessary for the evaluation contractor to expand the number of
students it planned on testing from only those in the treatment group
to also include those in the control group in order to make comparisons
between these groups because students in the control group would no
longer be participating in the norm-referenced testing as part of their
public school experience. According to Education officials, this
increased the cost of testing and outreach efforts associated with the
testing to a total of $800,000 to $1 million per year.
The change in tests will also make it difficult to make certain other
potentially useful achievement comparisons, particularly with respect
to students who do not use their scholarships and students in the
control group. Although threats to withhold scholarship funds have been
generally effective in securing the participation of scholarship users
in testing sessions, no punitive measures are linked to a failure to
participate for the other groups. The rewards offered--cash gifts, free
tax preparation, and an opportunity to enter a lottery to receive a
second chance for a scholarship--have not been sufficient to motivate
many nonusers to attend Saturday testing sessions or to engage in
appropriate test-taking behavior such as completing subtests and
remaining for the duration of the test. Missing test data, as well as
the number of students who fail to participate in testing, will likely
increase over time, reducing the comparability of the treatment and
control group. Although statistical techniques, such as weighting and
imputation, can be used to adjust for missing scores, these techniques
cannot guarantee findings would not be biased when the unweighted
response rate--that is, the number of students for whom scores are
available--falls to very low levels. (See table 6.)
Table 6: Unweighted Percentage of Students with Missing Achievement
Test Scores at Baseline and after 1 Year of Program Participation:
Rates in percent.
Cohort 1;
Reading: Treatment: Missing achievement scores at baseline: 23.3;
Reading: Control: Missing achievement scores at baseline: 14.4;
Math: Treatment: Missing achievement scores at baseline: 13.0;
Math: Control: Missing achievement scores at baseline: 23.8;
Both reading and math: Treatment: Missing achievement scores at
baseline: 13.0;
Both reading and math: Control: Missing achievement scores at baseline:
23.3.
Cohort 2;
Reading: Treatment: Missing achievement scores at baseline: 25.1;
Reading: Control: Missing achievement scores at baseline: 31.0;
Math: Treatment: Missing achievement scores at baseline: 13.6;
Math: Control: Missing achievement scores at baseline: 22.7;
Both reading and math: Treatment: Missing achievement scores at
baseline: 12.8;
Both reading and math: Control: Missing achievement scores at baseline:
22.4.
Cohort 1;
Reading: Treatment: Missing achievement scores after 1 year of program
participation: 21.7;
Reading: Control: Missing achievement scores after 1 year of program
participation: 42.0;
Math: Treatment: Missing achievement scores after 1 year of program
participation: 21.7;
Math: Control: Missing achievement scores after 1 year of program
participation: 42.0;
Both reading and math: Treatment: Missing achievement scores after 1
year of program participation: 21.7;
Both reading and math: Control: Missing achievement scores after 1 year
of program participation: 42.0.
Cohort 2;
Reading: Treatment: Missing achievement scores after 1 year of program
participation: 23.6;
Reading: Control: Missing achievement scores after 1 year of program
participation: 35.2;
Math: Treatment: Missing achievement scores after 1 year of program
participation: 20.8;
Math: Control: Missing achievement scores after 1 year of program
participation: 30.4;
Both reading and math: Treatment: Missing achievement scores after 1
year of program participation: 20.3;
Both reading and math: Control: Missing achievement scores after 1 year
of program participation: 29.5.
Source: GAO analysis of Westat data.
[End of table]
Scarcity of high school placements. The high schools participating in
the program offered very few placements, thus limiting the number of
high school students and graduates in the evaluation and hampering the
ability of the evaluator to make statistically meaningful comparisons
of dropout, graduation, and college admission rates of scholarship
recipients and nonrecipients. (See table 7.) For example, according to
data provided by WSF, the program graduated only 7 students from high
school in its first 2 years of operation, too small a number to make
sound comparisons. In addition, the low numbers of secondary openings
could indirectly exacerbate attrition from the evaluation if, as
scholarship users advance through the grade levels, they find
placements in higher grades are not available and become less motivated
to participate in data collection necessary for the evaluation.
Table 7: Number of Total Private School Placements for OSP Students:
Year: 2004-2005;
Kindergarten: 296;
Grades 1-5: 977;
Grades 6-8: 397;
Grades 9-12: 68;
Total: 1,738.
Year: 2005-2006;
Kindergarten: 169;
Grades 1-5: 474;
Grades 6-8: 192;
Grades 9-12: 72;
Total: 907.
Source: GAO analysis of Washington Scholarship Fund data.
[End of table]
Simultaneous Growth of Other Choice Options. The Act directs the
evaluator to determine the effects of the program on District public
schools, but the opportunities for choice within the public school
sector, such as charter schools and magnet programs, are far more
numerous than those offered by OSP and have grown over the same time
period, making the determination of OSP's independent effect on public
schools, if any, very difficult, if not impossible, to assess. The
evaluation design was based largely on an assumption that the award of
an opportunity scholarship would affect student achievement and public
schools by expanding school choice to include higher performing
schools. This assumption, however, does not account for the large
number of public charter schools, magnet schools, and other specialized
programs available through the District of Columbia as well as
scholarships offered by individual private schools or other private
organizations. Because these opportunities are far more numerous than
those offered by OSP and expanded in a similar time frame, it would be
difficult to attribute changes in the behavior of traditional schools
uniquely to OSP. Nor were participating private schools necessarily
limited to those that might be higher performing. Moreover, the large
portion of some schools' enrollment composed of OSP students suggests
that, in some instances, program effects, if any, may be more profound
for the participating schools that the scholarship users attend than
for the public schools from which the students departed.
Inability to include some scholarship recipients in the evaluation. In
the first year of the program, Education and WSF decided to process
applications in May and hold the placement lottery in August. At that
time, however, the program was undersubscribed at the kindergarten
through grade 5 level; that is, the number of eligible applicants from
public schools who attended kindergarten through grade 5 was less than
the number of private school openings available. Consequently,
Education and WSF gave scholarships to approximately all 800 eligible
public school applicants in these grades, thus eliminating them from
the evaluation. In addition, only 55 eligible students from schools in
need of improvement at the middle and high school levels applied that
year, and Education and WSF decided to offer these students
scholarships on a non-random basis, thereby eliminating these 55
students from the evaluation. Education and WSF further reduced the
number of students in the cohort 1 evaluation pool by awarding
scholarships to about 200 students who were already attending private
schools. Although the Act did not specifically prohibit scholarship
offers to students in private schools, the evaluation was designed to
make comparisons among students who had attended only District public
schools.[Footnote 25] Therefore, the contractor excluded them from the
evaluation so as not to reduce the usefulness of the evaluation
findings. As a result of these decisions, of the 2,454 students who
were offered scholarships in the first 2 years of the program, only
1,387--less than 60 percent---could be included in the evaluation. (See
table 8.)
Table 8: Eligible Applicants Offered and Not Offered Scholarships and
Numbers Included in and Excluded from the Evaluation:
Cohort: 1;
Eligible applicants: Offered scholarship: 1,366;
Eligible applicants: Not offered scholarship: 482;
Eligible applicants: Total: 1,848;
Number included in evaluation: Offered scholarship (treatment): 299;
Number included in evaluation: Not offered scholarship (control): 193;
Number included in evaluation: Total: 492;
Number excluded from evaluation: Offered scholarship: 1,067;
Number excluded from evaluation: Not offered scholarship: 289;
Number excluded from evaluation: Total: 1,356.
Cohort: 2[A];
Eligible applicants: Offered scholarship: 1,088;
Eligible applicants: Not offered scholarship: 728;
Eligible applicants: Total: 1,816;
Number included in evaluation: Offered scholarship (treatment): 1,088;
Number included in evaluation: Not offered scholarship (control): 728;
Number included in evaluation: Total: 1,816;
Number excluded from evaluation: Offered scholarship: 0;
Number excluded from evaluation: Not offered scholarship: 0;
Number excluded from evaluation: Total: 0.
Cohort: Total;
Eligible applicants: Offered scholarship: 2,454;
Eligible applicants: Not offered scholarship: 1,210;
Eligible applicants: Total: 3,664;
Number included in evaluation: Offered scholarship (treatment): 1,387;
Number included in evaluation: Not offered scholarship (control): 921;
Number included in evaluation: Total: 2,308;
Number excluded from evaluation: Offered scholarship: 1,067;
Number excluded from evaluation: Not offered scholarship: 289;
Number excluded from evaluation: Total: 1,356.
Source: GAO analysis of Westat data.
[A] Numbers for cohort 2 exclude eligible private school students
because the evaluation contractor assigned them a zero percent
probability of receiving a scholarship offer.
[End of table]
Because of the limited size of the treatment and control groups, the
evaluation contractor combined the cohort 1 and cohort 2 treatment and
control groups to make comparisons after year one. Future evaluations
will compare outcomes for students in these cohorts after they have
participated in the program for 2 and for 3 years.
Crossover. Crossover between treatment and control groups can reduce
the effectiveness of randomization and can make comparisons between
treatment and control groups difficult to interpret and generalize.
While scholarship offers made to applicants in the treatment group were
not withdrawn even if they failed to find a private school placement,
applicants assigned to the control group sometimes found other means to
attend private school. In fact, 15 percent of students in the control
group who provided math scores were enrolled in private schools. The
evaluation contractor used statistical adjustments to bound its
estimates of treatment effects for both control group students who
attended private schools and those who could not or did not use their
scholarships. However, the contractor did not consider potential
selection effects as a result of the inability of students to find an
acceptable placement as opposed to declining the use of a scholarship
for some other reason. While some students in the treatment group did
not use their scholarships or did not do so consistently and some
students in the treatment group who wanted to use their scholarships
could not because no school, or no school of their choice had openings
or would accept them, the number who did not use scholarships
specifically because they were unable to find acceptable placements is
unknown.
Concentration of scholarship students in a subset of schools. Because
private schools--which differ greatly in motivation, capacity, and
ability to accept OSP students--determine the number of OSP students to
accept, large numbers of scholarship users have been clustered within a
small subset of private schools. Additionally, OSP students constituted
60 percent or more of total reported enrollment in three participating
schools in school year 2006-2007. The evaluation contractor did
consider the potential influence of clustering of students within
families and in baseline schools prior to OSP participation on variance
estimates, which are used in determining the effects of scholarship
offers and scholarship use. However, the evaluation does not discuss
the potential impact of clustering large percentages of OSP students
within relatively few schools. Such clustering of evaluation
participants within a small number of schools has the potential to
confound program and school-level effects and may distort differences
between the comparison groups unless appropriate statistical
methodologies are employed to disentangle these effects when analyzing
the effect of receiving a treatment.
Combining scholarship offers with other supports. In addition to
offering scholarships, the program provides scholarship students with
additional individual supports, which may include case management and
parent empowerment services from the grantee, summer school, tutoring,
remedial classes, before-and-after school care, and mentoring.
Throughout their participation in the program, OSP students receive
different types and combinations of these services that were not
readily available to students in the control group, bringing into
question the comparability of students in the treatment and control
groups and raising problems in generalizing the study's findings to
circumstances in which a scholarship is the only treatment offered.
Some Required Comparisons Relied on Limited Data:
The Act required the contractor to compare the safety of schools
attended by participants in the program and the schools attended by
students who do not participate in the program and to evaluate the
success of the program in expanding choice options for parents. In its
evaluation of effects after year one, the contractor relied exclusively
on self-reported data to compare the safety of schools attended by the
two groups, using survey responses to compare perceptions of school
safety among parents of students offered scholarships and students
offered scholarships with the perceptions of those who were not and
their parents. While perceptions may be a valid measure, they do not
necessarily reflect the safety risks students face at a given school,
and parent and student perceptions sometimes diverged. While not
specifically required by the Act, the contractor also used surveys to
compare school satisfaction of (1) parents of students offered
scholarships and (2) students offered scholarships, with (3) school
satisfaction of parents of students not offered scholarships and (4)
students not offered scholarships. In the evaluation report, the
contractor cast satisfaction as "an indicator of the success of the
program in expanding options for parents."
Conclusions:
The scholarship program administered by WSF has provided low-income
families in the District an additional option to enroll their children
in private schools. However, to maintain a program capable of using
public funds for their intended purpose--that of providing increased
opportunities to low-income parents to send their children,
particularly those attending schools designated as in need of
improvement, to private schools--the agency or organization with
responsibility for operating a school choice program such as OSP needs
a strong financial accountability infrastructure that incorporates a
system of internal control.
During the period of our review, through year three of the program, WSF
struggled to build an accountability infrastructure that could assure
Congress, families of participating students, and the public that its
funds were used effectively and in compliance with laws and
regulations. Strong accountability and internal controls over a
scholarship program, such as OSP, are critical for ensuring that funds
are used for their intended purposes, and to maintain fiscal viability
and credibility. Overall financial management policies and procedures
and the underlying systems need to provide assurance that federal funds
are being used for the purposes intended and that funds are safeguarded
against loss from error, abuse, and fraud.
Without complete and accurate information about schools' performance
and other quality indicators, parents cannot make well-informed choices
among schools for their child. The grantee did not provide parents
information about the achievement levels of all students in
participating private schools and other indicators of school quality
and, for some schools, provided inaccurate information about teacher
qualifications and tuition levels. As a result, parents might have used
opportunity scholarships to place their children in private schools
that were less successful in raising achievement levels than the public
schools their children previously attended. In other cases, parents
might have rejected some private schools because they were given
inaccurate information about the schools or, in the case of the few
schools that do not customarily charge tuition, applied for a
scholarship in the belief that their children needed scholarships to
attend these schools.
In the absence of an accountability mechanism to ensure that
participating schools are operating lawfully in the District, students
were placed at risk of attending facilities that did not meet basic
health and safety requirements. Because OSP is a federal program that
had the support of the District of Columbia's government, parents may
have incorrectly assumed that Education and the city were overseeing
the program and making sure participating schools met such standards.
As a consequence, parents may have been less inclined to investigate
the status of these schools.
Finally, implementation of educational evaluations involving random
assignment often requires substantial oversight and structure. Without
such oversight and structure, the ability to draw valid conclusions
about the program to improve educational policy can be compromised by
the zeal to ensure program participants' success or to maximize access
to a treatment that program operators and applicants may believe is
beneficial even in the absence of empirical evidence. Indeed,
throughout the short history of OSP, decisions made to advance program
goals, such as maximizing the number of scholarships awarded in the
initial year and providing a wide range of academic and social supports
to scholarship users, have undermined the goals of the evaluation to
produce meaningful findings and the ability of Congress to use these
findings for decisions about other programs.
Recommendations for Executive Action:
Given the importance of using funds appropriately to further program
objectives and help ensure that schools are safe and provide sound
educational experiences, parents are given accurate information about
schools their children may attend, and program oversight is sufficient,
we are making the following recommendations. Specifically, we recommend
that the Secretary of Education direct WSF, the grantee, to:
* establish and implement detailed policies and procedures to improve
financial controls over OSP grant funds, including specific
requirements for the process of approving scholarship payments and
documentation of the process;
* establish compensating controls, such as supervisory review, to
reduce the risk of fraud in situations where segregation of duties is
not possible due to the size limitations of OSP's staff so that no one
employee can authorize, process, review, and have access to the funds
relating to OSP;
* continue its efforts to implement an integrated financial management
system to facilitate processing and recording of scholarship payments
and overall financial reporting;
* develop and implement procedures for conducting site visits,
including that site visit reports be prepared and contain information
on the overall financial stability of the school; and:
* develop procedures to ensure that accurate information is provided to
parents before a school is chosen about the summary achievement data of
students, teacher qualifications, and tuition levels, and that schools
make such information available on an annual basis to parents of
enrolled students.
We also recommend that the Secretary of Education collaborate with the
Mayor of the District of Columbia to ensure participating schools are
in compliance with all relevant District of Columbia education and
safety requirements, including school accreditation and health, safety,
and fire code requirements, and receive approval to operate in the
District.
Further, we recommend that the Mayor direct the Office of the State
Superintendent of Schools to collaborate with the Board of Education to
develop and implement procedures for ensuring that private schools in
the District meet applicable District requirements and to actively
participate in the oversight of OSP. Finally, we recommend that, in
planning for future programs for which Congress has required an
evaluation, the Secretary of Education should take steps to make
certain the program to be evaluated is overseen to ensure it is
implemented in a manner consistent with the evaluation design.
Agency Comments and Our Evaluation:
At our invitation, the Office of the Mayor of the District of Columbia
(the District), Education, and WSF provided written comments on a draft
of this report, which are reproduced in appendixes III, IV, and V,
respectively, and summarized below. WSF also provided technical
comments, which we incorporated when appropriate. In general, all three
entities concurred with our recommendations, but Education and WSF
disagreed with many of our findings and interpretations.
The District agreed with our findings and recommendations. It said the
report's findings have been extremely helpful to Mayor Fenty's
administration as it developed its plan for moving forward to fulfill
its responsibilities under the MOU with Education, particularly on the
steps the District should take to ensure that parents receive accurate
and complete information and that schools receiving funds under the
program meet certain basic health, safety, and instructional
requirements. The District's comments include specific ongoing or
planned steps to ensure that WSF complies with the requirements of
federal law and applicable local statutes and regulations. In addition,
the Deputy Mayor for Education has directed the District's Department
of Consumer and Regulatory Affairs to conduct inspections at the
schools identified as having missing or inapplicable certificates of
occupancy and is developing an accelerated schedule for reviewing the
regulatory status of all schools receiving funds under OSP. In its
comments, WSF supplied additional information about one of the several
schools DCRA identified as lacking a certificate of occupancy
reflecting operation of a private school. After verifying this
information with DCRA, we changed the draft to reflect it.
With one exception, Education generally agreed with our recommendations
and stated that it would use the information in our report to continue
to improve its oversight of the program and its coordination with the
District. With respect to our recommendation that, in planning for
future programs for which Congress has required an evaluation,
Education ensure that the program is implemented in a manner consistent
with the evaluation design, Education did not express agreement or
disagreement. Rather, Education's comments focus on the OSP evaluation,
indicating that we overemphasized the evaluation's challenges resulting
from program implementation decisions and other factors and therefore
underestimated the evaluation's utility. Specifically, Education stated
that the inability to compare OSP students to all students does not
undermine the more important analysis of program impact. While
Education adopted a strong methodology for its analysis of impact, as
required by statute, Education's comment is not responsive to our point
that its evaluation was unable to provide a comparison between students
offered scholarships and those in the same grades in District schools,
as the statute mandated. Also, Education wrote that the inability to
include some early scholarship recipients in the evaluation is not a
major issue but also noted that the Department rejected the notion of
excluding these early recipients altogether, "so that there would be
sufficient samples of students to allow impacts to be estimated for
subgroups of students." We maintain excluding about half of students
offered scholarships from the evaluation will affect the ability of the
evaluation to make strong comparisons. Education also wrote that other
supports provided by WSF were not organized and were available to both
the treatment and control groups. We disagree. According to WSF, it
offered case management services, conducted parent empowerment
sessions, found and worked with community organizations that provide
mentoring and tutoring, and, on a case by case basis, funded private
schools to provide enrichment and remedial services to support OSP
students. We continue to believe that the challenges we discuss have
complicated the interpretation of evaluation findings, limit
generalizability, and merit close examination when planning for the
evaluation of future programs.
Education's comments also expressed disagreement with many of our
findings. Education believes we mistakenly concluded that WSF's
practice of paying for tuition scholarships to schools that normally
receive donations to cover tuition violates the Act and WSF's policies.
We disagree. As discussed in the report and as expressly confirmed in
WSF's comments on the draft report, children who attend the three
scholarship-only schools pay no tuition. Section 307(a) (1) of the D.C.
School Choice Incentive Act states that, to be reimbursed under OSP,
the tuition charged by schools may not "exceed the amount of tuition or
fees that the school customarily charges to students who do not
participate in the program." (emphasis added) As section 307(a) (1)
makes clear, the amount of allowable tuition reimbursements depends
upon the amount of tuition the schools charge to students, not what
costs schools incur or what other funding mechanisms, such as donated
scholarships, they use to cover those costs.
In addition, Education stated its belief that the report does not
present a complete and balanced picture with respect to (1) the extent
to which OSP students previously attended schools in need of
improvement; (2) the establishment of OSP in time for the 2004-2005
school year; (3) responses from parents and demand for scholarships;
and (4) participating schools' legal independence in areas such as
hiring and establishing teacher qualifications. With respect to the
extent to which OSP students previously attended schools in need of
improvement, Education wrote that it would have been more accurate to
focus on the percentage of students from schools that received an
improvement designation during the student's first year in OSP. We
disagree. Scholarship lottery organizers could not be expected to give
priority to students from schools that had not yet been designated as
in need of improvement.[Footnote 26] Moreover, regardless of which year
is considered, the percentage of students from schools in need of
improvement was consistently smaller among scholarship recipients than
among the general population of District students from whom scholarship
recipients would have been recruited. With respect to the establishment
of OSP in time for the 2004-2005 school year, we included a timeline
delineating the key events in implementing this program and discussed
the short time available to implement the program in the draft we
provided Education to review and in this report. Education indicates
that GAO should have included information about positive responses from
parents of scholarship students enrolled in private schools. A balanced
picture of parental satisfaction would also include views of parents
whose children no longer participate or comparisons to school
satisfaction among parents who elected not to participate, but
collecting such data would have been tangential to our objectives
regarding mechanisms to account for program funds, results of efforts
to meet recruiting targets, and review of the evaluation. With respect
to demand for scholarships, as discussed in appendix I, we attempted to
work with WSF's data on program applicants and found these data
unreliable. Finally, Education commented that participating schools
were allowed to use their traditional independence and that the intent
of the law was to maximize participation of private schools in the
program. We agree but, as noted above, maintain that participating
schools are required to be in compliance with District education and
safety requirements. We believe we appropriately addressed each of
these areas and, therefore, have not made any changes based on
Education's comments.
Education also expressed dissatisfaction with our exit briefing and
asserts that GAO refused to brief the Department on its proposed
findings and recommendations prior to sending the Department the draft
report. However, at a July 10, 2007, exit conference, the meeting that
Education mentions in its comments, we presented our findings. The
findings we present at exit conferences are preliminary in that they
could change on the basis of information we receive at the exit
conference as well as information contained in official agency
comments. We also held exit conferences with WSF and the Office of the
Mayor. We considered all information provided by Education, WSF, and
the Office of the Mayor at and subsequent to the exit conferences.
Education also noted its concern about the inappropriate disclosure to
the news media of the report draft we had provided to Education, WSF,
and the Office of the Mayor for comment. We share this concern. While
we have policies and procedures designed to help prevent premature
disclosure, we cannot ensure that drafts will not be prematurely
released once they leave GAO's control.
Education provided further comments:
* Education wrote that the concentration of scholarship recipients in a
subset of participating private schools does not interfere with
accurate estimation of the program impact. GAO believes it is essential
to investigate whether OSP's effects are generalizable or limited to a
small subset of participating schools, and we note that the evaluation
contractor's report did not explicitly discuss the potential effects of
clustering in private schools.
* Education requested that GAO investigate the improper disclosure of
the report immediately. GAO is examining the circumstances related to
this specific instance of premature release. In this regard, we would
welcome Education's assistance by requesting that its Inspector General
review the Department's policies and procedures for handling draft GAO
reports in its control.
* Education noted that District requirements do not require teaching
staff to hold District teaching certificates. We agree and have not
asserted District teaching certificates were required.
* Education wrote that WSF has made significant progress in
establishing formal policies and procedures for improved internal
controls and integrating its financial management systems. GAO had
included this information in the draft report. However, as noted by GAO
and WSF, efforts are still underway to integrate the scholarship
payment processing with the financial system, which was scheduled to be
launched on November 1, 2007. WSF expects to finalize its policies and
procedures in January 2008.
* Education stated in its comment letter that since WSF's receipt of
the federal grant funds, WSF's annual independent financial A-133
audits have identified no reportable conditions or material weaknesses.
However, WSF's auditors did report an instance of noncompliance with
OSP requirements based on the finding that WSF had awarded scholarships
to four, out of 40 students tested, from households that did not meet
the income eligibility requirements. The auditor added that the effect
of this condition was that WSF expended federal funds on ineligible
recipients. In response to this finding, WSF stated that it terminated
the contract with the outside vendor and brought the eligibility
verification process in-house and also developed numerous additional
controls.
In WSF's comments, WSF acknowledged that some of GAO's recommendations,
findings, and observations were valid and useful and had taken or will
take action to address some of them but disagreed with many of our
observations and findings. As we noted earlier in our response to
Education, we believe the findings and interpretations are accurate as
stated in the draft, and therefore we have not made changes based on
WSF's comments. WSF disagreed with our interpretation of the D.C.
School Choice Incentive Act with respect to making payments to schools
that do not customarily charge tuition to their students, WSF's success
in meeting recruiting priorities, and the extent to which it had
provided complete information to parents. WSF also stated that it
"vigorously disagrees with our conclusion that there must be formal
'academic support activities' in order for before-and-after school
programs to contribute to student success." We disagree that our draft
report makes this conclusion. We do, however, conclude that we could
not determine based on the documentation provided by WSF that WSF
actually verifies that before-and-after school care programs are tied
to the student's academic program and part of customary fees charged by
the school, requirements that are Education and WSF's criteria for
determining whether a fee is allowable.
WSF also indicated that GAO erred in analyzing the participation of
students from schools in need of improvement, skewing the result toward
finding underrepresentation, by not assessing students by grade level.
GAO's analysis covered participation by students from schools in need
of improvement across all the grades the program was intended to serve.
The draft report we previously provided to them discussed the role of
grade level in an applicant's effective probability of receiving a
scholarship.
WSF officials also stated that our report focused primarily on matters
bearing little on the financial viability and effectiveness of the OSP;
however, evaluation of the financial viability of the OSP was not
within the scope of this engagement. As agreed with the requesters, GAO
focused on the identification and assessment of accountability
mechanisms over appropriated funding to implement the D.C. School
Choice Incentive Act. Implementation of effective internal control is
key to achieving accountability over grant programs, such as the OSP.
We reiterate that while WSF has taken actions to improve
accountability, such as integrating its student database with its
online billing system, as of the end of our field work, its operations
were still hampered by the lack of integration between its scholarship
payment processing and its financial accounting system and the related
impact on accountability over cash. We note that the preparation of
bank reconciliations is a key cash management control. We are pleased
that WSF has begun a process to redo all bank reconciliations starting
with 2004. We note further that efforts to implement a new financial
system began during the time of our review, and according to WSF
officials, they now expect the new system to be launched in November
2007 and to result in an increased level of efficiency and
accountability.
While WSF believes that the deficiencies noted in this report are minor
in nature, we stress that they are indicative of potential problems and
if not addressed could possibly have a material, detrimental effect on
WSF's accountability over federal funds, especially when combined with
the accounting systems and cash reconciliation weaknesses that existed.
WSF hypothetically submits that the incidence of error is inflated by
GAO counting a missing signature on one fee form as 20 errors if the
fee were charged to 20 students' scholarships. Our test results show
that the 39 students (78% of the sample) referenced in our report
attended 19 different schools participating in the program and that the
respective fee forms for those schools were missing an authorizing
signature.
In its comments, referring to its oversight visits to the schools in
the program, WSF also stated that "GAO puzzlingly reported that WSF did
not provide evidence of those visits." While WSF provided us a list of
the 42 schools that were visited, WSF was only able to provide one
completed report documenting a visit to one of these schools.
WSF commented, with reference to what it understood the rationale to be
for GAO's audit, that GAO's draft report cites absolutely no evidence
that federal OSP funds have been spent for anything other than genuine
educational purposes at any time during the three-plus years of OSP's
operation. As we noted in the report, GAO identified instances in which
payments were made to some OSP schools for before-and-after care
services, and it was not clear, based on the documentation provided by
WSF, whether these services were tied to educational activities.
In addition to comments that were also made by Education, WSF wrote
that GAO cited no evidence that OSP families have not received the
educational services that they sought through participation in the OSP.
The direct investigation of the extent OSP families received the
services that they sought was outside the scope of our work. We did
determine, however, that some families declined the use of scholarships
and that others did not remain in the program. For example, in year one
of the program, 32 percent of students offered scholarships did not use
them, and of the 68 percent that did, 31 percent did not use them in
all the years the scholarships were available to them.
We are sending copies of this report to the Secretary of Education, the
Mayor of the District of Columbia, and the President of the Washington
Scholarship Fund, appropriate congressional committees, and others who
are interested. We will also make copies available to others upon
request. In addition, the report will be available at no charge on
GAO's Web site at [hyperlink, http://www.gao.gov].
If you or your staff have any questions or wish to discuss this
material further, please call Cornelia M. Ashby at (202) 512-7215 or
Jeanette M. Franzel at (202) 512-9471.
Signed by:
Cornelia M. Ashby:
Director, Education, Workforce, and Income Security Issues:
and:
Jeanette M. Franzel:
Director, Financial Management and Assurance:
[End of section]
Appendix I: Scope and Methodology:
To address our research objectives, we focused our efforts on assessing
three areas: (1) the accountability mechanisms that Washington
Scholarship Fund (WSF) has put in place to govern the use of funds, (2)
the results of WSF's efforts to meet the program's recruiting
priorities and eligibility requirements and inform parents of their
choices, and (3) the extent to which the evaluation reflects statutory
requirements and the implementation of the program supported the
detection of useful and generalizable findings. In performing our work
to address these objectives, we interviewed officials from WSF, the
Department of Education's Office of Innovation and Improvement and
Institute of Education Sciences, and District government officials. In
doing our work, we requested and received program, demographic,
testing, and funding data from both WSF and Westat. We encountered data
reliability problems that affected the achievement of these objectives.
These problems and the resulting audit work we performed are described
below.
Assessing the Expenditure and Accountability of Grant Funds:
In assessing the expenditure and accountability of grant funds, we met
with grantee and Education officials, reviewed key documents to
identify expenditures and key internal controls, and then tested the
application of those controls for school year 2005-2006 through
sampling. To identify the changes in the grantee's payment processes
from 2004 and key internal controls, we met with the grantee staff and
reviewed their procedures.
Due to limitations in the grantee's financial accounting software, we
used the Excel spreadsheet for school year 2005-2006 as the population
of school scholarship payments from which to draw our sample of 50
students. Before selecting our sample, we were unable to determine that
the student scholarship payments for school year 2005-2006 were
complete and reliable because the total of the payments on the
spreadsheet could not be directly traced to the grantee's financial
accounting software or the total of funds drawn down from Education.
Therefore, we pulled a random sample of students for whom scholarship
payments were made for school year 2005-2006. We tested all OSP
scholarship payments made during school year 2005-2006 for each of the
50 students selected randomly to determine whether key internal
controls were being properly implemented for those transactions.
Assessing Demographic and Program Data:
In the process of assessing data reliability, we found that the data
were sufficiently reliable to address our objectives. However,
reliability problems with the WSF student demographic and program data
system, limited the analyses that we could perform. Data problems
included the following:
* The data were not integrated and different data sets were maintained
on different spreadsheets.
* The student data lacked unique and uniform identification numbers
that would have allowed the efficient linkage of key data sets from
different data and, therefore, made it necessary to link the students'
data by first and last names.
* Many data entry errors including misspellings of names and duplicated
entries.
* Substantial missing data for such important data fields as race,
grade level, and date of birth made the fields insufficiently populated
for analyses; for example, 32 percent of data identifying gender, 34
percent identifying birth date, and 32 percent identifying race were
missing.
As a result, we were limited in our ability to describe characteristics
of students in the program. We were also limited in our ability to link
student demographic and financial eligibility data with payment data.
We reviewed WSF processes for collecting financial eligibility
information and the household income data reported by families of
students offered scholarships but we did not test those data as to
accuracy. Although we determined that students in the program for whom
we had data generally met financial eligibility requirements, we were
unable to positively ensure that all students who were receiving
Opportunity Scholarship Program (OSP) funds had been determined
eligible by WSF due to missing data payment. Using WSF's payment
records, we identified 50 students who WSF indicated had received
scholarships and were attending private schools, but we could not
electronically locate eligibility data for these students. As a result,
we could not verify whether these students were eligible. We
subsequently sent these names to WSF, which provided alternative
spellings for the students' names that enabled us to reduce the 50
nonmatching records to 18. Although these students represent a small
percentage of scholarship users, the absence of their financial
eligibility data represented a considerable weakness in the data
system.
Although we could correct some data errors through manual checking, we
were unable to correct and eliminate apparently duplicate records for
students who had applied for the program. Due to these duplicate
records, we were unable to determine the number of students who applied
for the OSP and, of those, the number and percentage who were
determined to be eligible. We attempted to work with the WSF to
identify which records were duplicates in the applicant data, and which
were valid applicants, but due to data problems we were unable to make
these identifications. Instead, we could only report on the number
entered into the lottery and whether they received scholarship awards
or not.
We used data from WSF to create a database to summarize characteristics
of participating private schools. We used data provided by the District
of Columbia Public Schools to identify schools designated as in need of
improvement and calculate the numbers and percentages of students who
attended these schools by year.
Data limitations prevented us from using computerized methods to match
information on WSF data sets with the evaluator contractor's database.
We attempted to match data from WSF with data collected by the
evaluation contractor as a reliability check. However, because the WSF
and the evaluation contractor did not develop common identifiers for
designating participants in the program, we were unable to match
students in an automated fashion, and therefore had to match records
using student first and last name. Despite intensive efforts, including
manually making spelling changes to over 492 student names, the volume
of data mismatches would not allow us to develop a systematic method to
match the records of WSF and the evaluation contractor.
The evaluation contractor matched its data to WSF records by attempting
to match on a variety of including first and last name and birth date;
developed a crosswalk to link by household number; and by manually
matching records. The contractor also used a computer application that
matched names to other names by phonetic spelling; that is, names were
matched with other names that sounded similar, allowing it to match
most of the data. We determined that this method was not sufficiently
reliable for our purposes because some names had similar phonetic
spelling.
Additionally, we found large discrepancies between the numbers of
applicants and eligible students included on the evaluation
contractor's database and in the WSF files. This further decreased our
confidence in the use of these data for this population.
Assessing Strategies to Recruit Families and Private Schools and Inform
Parents about Their Choices:
To assess how the grantee was meeting the program's recruiting and
eligibility objectives and informing parents about their choices, we
interviewed knowledgeable officials and examined program documentation,
especially relating to recruitment and selection of both students and
private schools. Apart from reviewing publicly available information
about schools, we did not evaluate schools or their performance. To
determine the number of OSP students who had attended schools in need
of improvement, we identified the type and No Child Left Behind status
of schools previously attended by students in cohorts 1, 2, and 3, and
quantified the number of students who had attended District of Columbia
public schools in need of improvement for each year that the program
operated. To determine whether all participating private schools met
the requirements to operate lawfully in the District, we selected a non-
probability sample of 18 schools, using various criteria such as
whether or not schools had registered with the District's Department of
Consumer and Regulatory Affairs and whether or not schools were
accredited by an agency recognized by the District To determine the
validity of information in the directories published by WSF in order to
see if information provided parents was accurate, we compared directory
information with information available from other sources. In reviewing
the directory information, we became aware of discrepancies in
information reported in English and Spanish versions of the directory,
and information provided across different school years. We found the
information across sources, across years, and between the English and
Spanish version was inconsistent.
Assessing the Extent to Which the Evaluation Reflects Statutory
Requirements and the Implementation of the Program Supported the
Detection of Useful and Generalizable Findings:
To assess the extent to which the evaluation reflects the requirements
in the Act and to which the implementation of the program supported the
detection of useful and generalizable findings, we met with Education
officials from the Institute of Education Sciences and examined
documents including the contractor's participation reports for program
year one and program year two, the analysis plan it developed for its
evaluation of impacts after 1 year of program participation, and the
Evaluation of the D.C. Opportunity Scholarship Program: Impacts After
One Year, released in June 2007. We also reviewed contractual
documentation. To assess the usefulness of evaluation findings, we
examined program attrition, the extent of missing test data, and the
statistical methodology the contractor used to analyze comparisons
between students offered scholarships and students not offered
scholarships, and, to a lesser extent, between students in these groups
who used their scholarships and those who remained in public schools.
[End of section]
Appendix II: Laws Authorizing Related Voucher Programs:
Table 9: Analysis of Laws Authorizing Voucher Programs in the District
of Columbia and the State of Ohio:
Analysis of state laws on private school vouchers: Name of program;
District of Columbia: D.C. School Choice Incentive Program;
State of Ohio: Educational Choice Scholarship (EdChoice) Pilot Program.
Analysis of state laws on private school vouchers: Year enacted;
District of Columbia: 2004;
State of Ohio: 2006.
Analysis of state laws on private school vouchers: Description and
eligibility requirements;
District of Columbia: The Opportunity Scholarship Program, also known
as the D.C. School Choice Incentive Program, provides scholarships to
students for attendance at private schools in the District of Columbia.
It is federally funded. To be eligible, students must be from families
who reside in the District and whose household income does not exceed
185 percent of the federal poverty level. Scholarship recipients who
received their first scholarship in 2004-2005 or 2005-2006 school year
may retain their scholarships if their household income does not exceed
300 percent of the poverty level. If the number of new scholarships in
any year is less than the number of eligible applicants, selection of
recipients follows a lottery method. Priority is given to students
attending schools identified for improvement, corrective action, or
restructuring under Title I of the No Child Left Behind Act of 2001;
State of Ohio: Students who attend or will be entering Ohio public
schools that have been designated by the state as "Academic Watch" or
"Academic Emergency" for 2 of the last 3 years are eligible to receive
scholarships to attend the participating private school of their
choice. Students currently enrolled in charter schools but who would
otherwise be assigned to schools in these categories are also eligible.
Students in the Cleveland Municipal School District are not eligible to
participate, as the state offers a separate scholarship program for
these students. Scholarships are not available to students currently
enrolled in a private non-public school or who are home-schooled.
Eligible students must first be accepted at a participating private
school for the next school year before applying for an EdChoice
scholarship.
Analysis of state laws on private school vouchers: Currently
implemented;
District of Columbia: Yes;
State of Ohio: Yes.
Analysis of state laws on private school vouchers: Participating
students;
District of Columbia: 1,819;
State of Ohio: 2,785.
Analysis of state laws on private school vouchers: Geographic area;
District of Columbia: D.C;
State of Ohio: State of Ohio, except Cleveland.
Analysis of state laws on private school vouchers: Limit on number of
students;
District of Columbia: Limited by available funding;
State of Ohio: Limit of 14,000; if number exceeds 14,000, priority will
be given to students who already received scholarships and with family
incomes at or below 200 percent of the poverty level, and students will
be elected by lot to receive remaining scholarships.
Analysis of state laws on private school vouchers: Grades;
District of Columbia: K-12;
State of Ohio: K-12.
Analysis of state laws on private school vouchers: Average or maximum
amount of assistance;
District of Columbia: $7,500;
State of Ohio: $5,000.
Analysis of state laws on private school vouchers: Criteria for
determining amount of assistance;
District of Columbia: The annual scholarship amount is $7,500 or the
participating private school's tuition, fees, and any transportation
costs, whichever is less;
State of Ohio: The EdChoice scholarship amount is currently set at
$4,250 for elementary school students (grades K-8) and $5,000 for high
school students or the private school's tuition amount, whichever is
lower. The scholarship amount will increase slightly each year.
Analysis of state laws on private school vouchers: Types of schools
allowed to participate;
District of Columbia: Any private elementary or secondary school within
DC, including religious schools;
State of Ohio: Any chartered nonpublic school that meets the state's
requirements.
Analysis of state laws on private school vouchers: Testing or
evaluation criteria;
District of Columbia: As mandated by law, the program is evaluated
rigorously on an annual basis by an independent research organization.
Evaluations address the academic achievement and the retention,
dropout, and college admissions rates of scholarship recipients, in
comparison both to students who remain in D.C. public schools and to
students who applied for but did not receive scholarships. Evaluations
also examine the following: the success of the program in expanding
educational options for parents; the reasons why parents choose to have
their children participate in the program; the impact of the program on
students and public schools in the District; and the safety of the
schools attended by scholarship students, in comparison to other D.C.
schools;
State of Ohio: Private schools are required to administer state
achievement tests to scholarship students.
Analysis of state laws on private school vouchers: Accreditation of
private schools;
District of Columbia: Not specified;
State of Ohio: A nonpublic school must hold a valid state charter and
comply with the operating standards for Ohio's schools and agree to
register with the Ohio Department of Education and comply with the
rules of the program, including administering state achievement tests.
Analysis of state laws on private school vouchers: School admission
requirements;
District of Columbia: Not specified, but does state that students must
abide by rules of the school applicable to all students;
State of Ohio: Not specified, but students must gain admittance to the
eligible private school before applying for the scholarship.
Analysis of state laws on private school vouchers: Discrimination
provision;
District of Columbia: Certain exemptions from nondiscrimination
requirements are given to participating schools with a religious
affiliation. Participating private schools cannot discriminate on the
basis of race, color, national origin, religion, or sex (except for
single-sex schools);
State of Ohio: Not specified.
Analysis of state laws on private school vouchers: Religious "opt out"
close;
District of Columbia: Not specified;
State of Ohio: Not specified.
Analysis of state laws on private school vouchers: Financial audit
requirements;
District of Columbia: Administrative entity is responsible for ensuring
that participating schools are financially responsible;
State of Ohio: Not specified.
Analysis of state laws on private school vouchers: Transportation;
District of Columbia: Funds may be used for transportation expenses;
State of Ohio: Students enrolled in nonpublic schools may be entitled
to pupil transportation services from their public school district of
residence. The nonpublic school must be within a 30-minute bus ride
from the public school during the school day. In certain cases, a
public school district may declare a student "impractical to transport"
and instead provide limited reimbursement payment to the parent.
Analysis of state laws on private school vouchers: Authorizing statute;
District of Columbia: D.C. Code § 38-1851.01 --D.C. Code § 38-1851.11;
State of Ohio: Ohio Revised Statues. Sections 3310.01-3310.17.
Analysis of state laws on private school vouchers: Legislative history;
District of Columbia: On January 23, 2004, President Bush signed the
program into law via the D.C. School Choice Incentive Act of 2003,
which was included in the Consolidated Appropriations Act of 2004. In
2006, legislation passed that raised the household income eligibility
renewal limit from 200 to 300 percent of the federal poverty level for
students who received their first scholarship in 2004-2005 and 2005-
2006 school years. The program was first implemented in the 2004-2005
year and is the first of its kind at the federal level;
State of Ohio: On June 30, 2005, the Educational Choice Scholarship
Pilot Program was signed into law as part of an omnibus education bill.
On March 30, 2006, a new omnibus education bill was passed that
expanded eligibility for scholarships under the program to students in
schools in "academic emergency" or "academic watch" for the 3 previous
years; the previous law limited eligibility to students only in schools
in "academic emergency," the lowest category in the school rating
system. On March 30, 2007, eligibility for scholarships was further
extended to students in schools in these categories for 2 of the
previous 3 years.
Analysis of state laws on private school vouchers: Year enacted;
District of Columbia: [Empty];
State of Ohio: [Empty].
Source: GAO Analysis of State Laws and U.S. Department of Education,
Education Options in the States: State Programs that Provide Financial
Assistance for Attendance at Private Elementary or Secondary Schools
(Washington, D.C.: Aug. 2007).
[End of table]
Table 10: Analysis of Laws Authorizing Voucher Programs in Cleveland,
Ohio, and Milwaukee, Wisconsin:
Analysis of State Laws on Private School Vouchers: Name of program:
Year enacted;
Cleveland, Ohio: Cleveland Scholarship and Tutoring Program: 1995;
Wisconsin--Milwaukee: Milwaukee Parental Choice Program: 1990.
Analysis of State Laws on Private School Vouchers: Name of program:
Description and eligibility requirements;
Cleveland, Ohio: Cleveland Scholarship and Tutoring Program: The
Cleveland Scholarship and Tutoring Program provides scholarships to
students in the Cleveland Municipal School District. The scholarships
are for attendance at a qualified private school within the district or
at a public school in any district surrounding Cleveland. Students in
grades K-8 are eligible to apply for scholarships. Scholarship
recipients may retain their scholarships through grade 12. If the
number of new scholarships in any year is less than the number of
eligible applicants, selection of recipients follows a lottery method.
Students from low-income families have priority in receiving new
scholarships;
Wisconsin--Milwaukee: Milwaukee Parental Choice Program: The Milwaukee
Parental Choice Program (MPCP) provides scholarships to students for
attendance at participating private schools in Milwaukee. To be
eligible, students must be from families who reside in Milwaukee and
whose household income does not exceed 175 percent of the federal
poverty level. A student participating in the program, and whose family
income increases, may remain in the program until the family's income
exceeds 220 percent of the federal poverty level. The maximum
participation is 22,500 students. Applications for scholarships are
submitted directly to participating schools. If a participating school
receives more applications in any year than it has seats available,
selection of recipients follows a lottery method. Siblings of current
scholarship recipients have priority in receiving new scholarships.
Analysis of State Laws on Private School Vouchers: Name of program:
Currently implemented;
Cleveland, Ohio: Cleveland Scholarship and Tutoring Program: Yes;
Wisconsin--Milwaukee: Milwaukee Parental Choice Program: Yes.
Analysis of State Laws on Private School Vouchers: Name of program:
Participating students;
Cleveland, Ohio: Cleveland Scholarship and Tutoring Program: 5,921;
Wisconsin--Milwaukee: Milwaukee Parental Choice Program: 17,410.
Analysis of State Laws on Private School Vouchers: Name of program:
Geographic area;
Cleveland, Ohio: Cleveland Scholarship and Tutoring Program: Cleveland
Municipal School District;
Wisconsin--Milwaukee: Milwaukee Parental Choice Program: City of
Milwaukee.
Analysis of State Laws on Private School Vouchers: Name of program:
Limit on number of students;
Cleveland, Ohio: Cleveland Scholarship and Tutoring Program: Number of
scholarships limited by the amount of state-appropriated funds
available;
Wisconsin--Milwaukee: Milwaukee Parental Choice Program: 22,500.
Analysis of State Laws on Private School Vouchers: Name of program:
Grades;
Cleveland, Ohio: Cleveland Scholarship and Tutoring Program: K- 8 for
new applicants or 9-12 for renewal students;
Wisconsin-- Milwaukee: Milwaukee Parental Choice Program: K-12.
Analysis of State Laws on Private School Vouchers: Name of program:
Average or maximum amount of assistance;
Cleveland, Ohio: Cleveland Scholarship and Tutoring Program: $3,450;
Wisconsin--Milwaukee: Milwaukee Parental Choice Program: $6,501.
Analysis of State Laws on Private School Vouchers: Name of program:
Criteria for determining amount of assistance;
Cleveland, Ohio: Cleveland Scholarship and Tutoring Program: To attend
a participating private school, the annual scholarship amount is based
on the tuition of the school and the family income level of the
recipient. Currently, the scholarship amount may not exceed the
approved private school's tuition or $3,450 for students in grades K-
12, whichever is less. For recipients whose family income level is
below 200 percent of the federal poverty level, the actual scholarship
award is 90 percent of (up to) the maximum amount (i.e., a student from
such a family and currently in grade 8 may receive a scholarship award
of at most $3,105). For recipients whose family income is at or above
200 percent of the poverty level, the scholarship is 75 percent of (up
to) the maximum amount. Remaining tuition costs are to be covered by
parents; however, for recipients in grades K-8 whose family income
level is below 200 percent of poverty, participating schools must not
charge any tuition in excess of the remaining 10 percent of the amount,
whatever that amount may be;
Wisconsin--Milwaukee: Milwaukee Parental Choice Program: For the 2006-
2007 academic year, the scholarship amount is the participating private
school's per pupil expenditure or $6,501, whichever is less.
Participating schools must accept the scholarship amount as full
payment of tuition.
Analysis of State Laws on Private School Vouchers: Name of program:
Types of schools allowed to participate;
Cleveland, Ohio: Cleveland Scholarship and Tutoring Program: Any
nonpublic chartered schools within the boundaries of the Cleveland
Municipal School District that meets all applicable requirements;
Wisconsin--Milwaukee: Milwaukee Parental Choice Program: Any private
school within the city.
Analysis of State Laws on Private School Vouchers: Name of program:
Testing or evaluation criteria;
Cleveland, Ohio: Cleveland Scholarship and Tutoring Program: Not
specified;
Wisconsin--Milwaukee: Milwaukee Parental Choice Program: The law
requires participating private schools to administer nationally normed
standardized tests to scholarship recipients in grades 4, 8, and 10.
Analysis of State Laws on Private School Vouchers: Name of program:
Accreditation of private schools;
Cleveland, Ohio: Cleveland Scholarship and Tutoring Program: The school
must meet all state minimum standards for chartered nonpublic schools
in effect on July 1, 1992, except that the state superintendent at the
superintendent's discretion may register nonchartered nonpublic schools
meeting the other requirements of this division;
Wisconsin--Milwaukee: Milwaukee Parental Choice Program: With respect
to participating private schools, the law requires schools to be
accredited from among a list of accrediting agencies.
Analysis of State Laws on Private School Vouchers: Name of program:
School admission requirements;
Cleveland, Ohio: Cleveland Scholarship and Tutoring Program: Schools
are required to give preference to students previously enrolled and
their siblings, but specifies the number of scholarship students for
grades K-3 equals the number that constituted 20 percent of the total
number of students enrolled in the school during the preceding year in
such grade. Scholarship students are to be randomly selected by
lottery. Schools must admit students in grades who were previously
admitted;
Wisconsin--Milwaukee: Milwaukee Parental Choice Program: Participating
private schools shall accept pupils on a random basis, but may give
preference to siblings of pupils already enrolled or current
scholarship participants. May only consider income and grade level, not
academic achievement.
Analysis of State Laws on Private School Vouchers: Name of program:
Discrimination provision;
Cleveland, Ohio: Cleveland Scholarship and Tutoring Program:
Participating private schools may not discriminate on the basis of
race, religion, or ethnic background, and may not promote unlawful
behavior or teach hatred;
Wisconsin--Milwaukee: Milwaukee Parental Choice Program: A school may
not use an applicant's race, ethnic background, religion, priority test
scores, grades, or membership in the church parish when making
admissions decisions. Participating private schools may not
discriminate on the basis of race, religion, or ethnic background, and
may not promote unlawful behavior or teach hatred.
Analysis of State Laws on Private School Vouchers: Name of program:
Religious "opt out" close;
Cleveland, Ohio: Cleveland Scholarship and Tutoring Program: Not
specified;
Wisconsin--Milwaukee: Milwaukee Parental Choice Program: Students may
be excused from religious activities at a religious school if their
parent or guardian submits a written request to the teacher and
principal of the school. Participating schools may not require a
recipient to participate in religious activities.
Analysis of State Laws on Private School Vouchers: Name of program:
Financial audit requirements;
Cleveland, Ohio: Cleveland Scholarship and Tutoring Program: Not
specified;
Wisconsin--Milwaukee: Milwaukee Parental Choice Program: The Act
requires an annual financial audit and specifies other extensive
accounting and financial requirements.
Analysis of State Laws on Private School Vouchers: Name of program:
Transportation;
Cleveland, Ohio: Cleveland Scholarship and Tutoring Program: Must be
provided by Cleveland Municipal School District;
Wisconsin--Milwaukee: Milwaukee Parental Choice Program: The Milwaukee
School District may provide transportation or pay some of the cost, in
certain cases.
Analysis of State Laws on Private School Vouchers: Name of program:
Authorizing statute;
Cleveland, Ohio: Cleveland Scholarship and Tutoring Program: ORC Ann.
3313.974 - ORC Ann. 3313.99;
Wisconsin-- Milwaukee: Milwaukee Parental Choice Program: Wis. Stat. §
119.23.
Analysis of State Laws on Private School Vouchers: Name of program:
Legislative history;
Cleveland, Ohio: Cleveland Scholarship and Tutoring Program: The
program was enacted in 1995 and first implemented in the 1996-1997
school year. On July 1, 2003, the state legislature amended the program
to allow recipients to retain their scholarships through grade 10, as
of the 2004-2005 academic year, and also raised the maximum scholarship
amount from $2,500 to $3,000. In June 2005, the legislature expanded
the grade range of students to whom new scholarships are available from
grades K -3 to K -8; it also allowed scholarship recipients to retain
their scholarships through grade 12 during the 2006-2007 year and
raised the maximum scholarship amount available to all recipients,
regardless of grade, to $3,450 for the 2006-2007 and subsequent years;
Wisconsin--Milwaukee: Milwaukee Parental Choice Program: The Milwaukee
Parental Choice Program was enacted in 1990. In 1995, it was expanded
to include religious schools. In 2006, the law was changed to increase
the number of students who may participate in the program, by
increasing the income limit from 175 percent to 220 percent over the
federal poverty level and by eliminating the prior year attendance
requirements (allowing students who move into the district to be
eligible).
Analysis of State Laws on Private School Vouchers: Name of program:
Year enacted;
Cleveland, Ohio: Cleveland Scholarship and Tutoring Program: 1995;
Wisconsin--Milwaukee: Milwaukee Parental Choice Program: 1990.
Source: GAO Analysis of State Laws and U.S. Department of Education,
Education Options in the States: State Programs that Provide Financial
Assistance for Attendance at Private Elementary or Secondary Schools
(Washington, D.C.: Aug. 2007).
[End of table]
[End of section]
Appendix III: Comments from the Office of the Mayor of the District of
Columbia:
Government Of The District Of Columbia:
Office of the Deputy Mayor for Education:
1350 Pennsylvania Avenue NW:
Washington, D.C. 20004:
Victor Reinoso:
Deputy Mayor for Education:
October 22, 2007:
Cornelia M. Ashby, Director:
Education, Workforce, and Income Security Division:
Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Ms. Ashby:
Thank you for sharing your draft report on the District's school
voucher program with me and my staff and for providing us the
opportunity to comment on it prior to publication.
As you stated in the report, the District's federally-funded voucher
program was authorized by Congress in January 2004 and has been in
operation since the 2004-2005 school year. As you know, neither I, nor
the Mayor, was in a position during the program's first three years to
exercise oversight of the program as envisioned in the Memorandum of
Understanding ("MOU") between that the U.S. Department of Education and
then-Mayor Anthony Williams.
Having said that, the report's findings, and the discussions we had
while your team was conducting its review, have been extremely helpful
to us as we developed our plan for how the Fenty Administration would
fulfill its responsibilities under the MOU moving forward, and, in
particular, what steps we should take to ensure that parents
considering participating in the program receive accurate and complete
information regarding the schools their child might attend and that
schools that receive funds under the program meet certain basic health,
safety, and instructional requirements.
I would like to take a moment to share with you the steps we already
have taken to improve oversight of this program:
* First, on my recommendation, Mayor Fenty delegated his direct
oversight responsibility under the MOU to Deborah Gist, the District's
State Superintendent of Education. Ms. Gist heads a new, independent
state education agency created as part of the Mayor's education reform
legislation. Her agency is charged with oversight of all federal
education funds received by the District and, as such, I believe it is
uniquely positioned to serve in this role.
* Upon receipt of the delegation letter, Ms. Gist contacted the
Department of Education to request a meeting regarding the problems in
program management by the Washington Scholarship Fund ("WSF") and in
oversight identified by GAO. A meeting was then convened to discuss how
the parties should more closely monitor WSF's compliance with statutory
requirements relating to student recruitment, targeting of students
attending schools "in need of improvement," tuition and fees, and
program evaluation[Footnote 27] in the future.
* Because the original MOU between the Department and the District did
not contain any details regarding coordination or division of their
oversight role, Ms. Gist has since recommended to the Department that
the parties enter into a follow-on agreement that clearly assigns
certain activities to her agency and others to the Department.
* In addition, Ms. Gist and her staff met with WSF to explore ways that
the District can assist that organization in complying with the
requirements of the federal law and with applicable local statutes and
regulations.[Footnote 28]
* Ms. Gist has assigned a member of her staff to act as the liaison to
the Department and WSF and has given her primary responsibility for
ensuring that the District's oversight obligations under the follow-on
agreement are met.
* Finally, Ms. Gist is developing an infrastructure within her office
for tracking the compliance of all non-public schools in the District,
including those that participate in the voucher program, with local
regulations[Footnote 29] requiring them to be accredited by a
recognized independent accreditation body or to have otherwise
demonstrated satisfactory evidence of program quality.
In addition, I have directed the District's Department of Consumer and
Regulatory Affairs ("DCRA") to conduct inspections at all of the
schools identified in your report as having inapplicable or missing
Certificates of Occupancy. As we discussed, these certificates are
critical, because they show that the schools have met certain
construction, health, safety, and fire codes.[Footnote 30] Those
inspections are underway and should be completed by the middle of next
week. At that point, we will determine whether those facilities
currently meet the regulatory requirements applicable to school
buildings and can be issued the proper certificates, whether any
violations that exist can be quickly remedied, or whether a school that
is unable or unwilling to remedy its status will be required to close
or relocate in order to protect the health and safety of its students.
My office also will work with DCRA to develop an accelerated schedule
for reviewing the regulatory status of all of the schools receiving
funding under the voucher program.
I would like to reiterate that the primary concern of the Fenty
Administration is ensuring that the instruction received by children
who participate in this program meets widely-accepted, independent
standards for educational programs, that the children learn in
facilities that do not present threats to their health or safety, and
that their parents receive accurate information regarding the quality
of that instruction. We look forward to working with the Department and
WSF to ensure that that is the case.
Thank you, again, for your help in highlighting key areas on which we
should focus as we developed policies and procedures for effective
oversight of this program and as we move forward to implement them.
Sincerely,
Signed by:
Victor Reinoso:
Deputy Mayor:
[End of section]
Appendix IV: Comments from the Department of Education:
United States Department Of Education:
Office Of Innovation And Improvement:
400 Maryland Ave., S.W.:
Washington, DC 20202:
[hyperlink, http://www.ed.gov]:
October 23, 2007:
Ms. Cornelia M. Ashby:
Director, Education, Workforce and Income Securities Issues:
Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Ms. Ashby:
Thank you for the opportunity to comment on the draft report entitled,
"District of Columbia Opportunity Scholarship Program (OSP): Additional
Policies and Procedures Would Improve Internal Controls, Program
Operations, and Goal Attainment."
In the report, GAO makes recommendations that the Secretary of
Education direct the Washington Scholarship Fund (WSF) to improve
internal controls, integrate financial systems, improve monitoring and
site visit procedures, and provide accurate information to parents. GAO
also makes two additional recommendations that the Secretary and the
Mayor of the District of Columbia (District) take specific steps to
ensure participating schools are in compliance with all relevant
District education and safety requirements and that the Mayor implement
procedures to ensure that private schools meet District requirements.
GAO also recommends that, in planning the implementation of future
programs for which an evaluation is required, the Secretary take
measures to ensure that oversight efforts and other program
implementation matters are handled in a manner consistent with the
evaluation design required by Congress.
The Department's comments include several general comments on three
broad concerns, and then comments on specific recommendations.
General Comments:
As an initial observation, we feel obligated to note that GAO refused
to brief the Department on its proposed findings and recommendations
prior to sending us the draft report. This type of briefing is usually
done by GAO at a substantive "exit conference" which is held after
GAO's audit work is completed and before the draft report is prepared
and issued, to provide an opportunity for the parties to discuss the
draft findings and recommendations and clarify possible ambiguities or
misunderstandings, before those findings and recommendations are
finalized in the draft report. At a July 10 meeting GAO staff orally
presented what they indicated were preliminary observations and
indicated that they could not discuss draft findings or any
recommendations at that time.
Following that meeting, Department staff were sufficiently concerned
about possible misunderstandings by GAO on several key points
(including, but not limited to, the participation of students from
schools in need of improvement, the lawful operation of private schools
in the District and related matters including teacher quality and
accreditation, and the program's evaluation) to request a more
substantive exit conference. We also submitted a 10-page document to
GAO on July 26 with clarifying and additional information in a further
effort to correct possible misunderstandings. Additionally, at the
request of a GAO Deputy Assistant General Counsel, on August 6, 2007,
Department representatives participated in a teleconference to address
GAO questions about the permissibility of paying fees for before and
after school programs under the OSP.
Department representatives continued to request that GAO participate in
a more substantive exit conference meeting with our staff prior to
preparing the draft report to ensure there were no further questions
about the information provided and no further misunderstandings to be
resolved before a complete and accurate report could be presented.
Unfortunately, GAO denied the Department's request for such a meeting
and the resulting draft report demonstrates that certain
misunderstandings continue to exist, especially with regard to the
information provided to GAO by the Department after the meeting held on
July 10.
Additionally, the Department continues to be extremely disturbed that
this draft audit report was improperly disclosed to various local media
outlets, including The Washington Post immediately after the issuance
of the report. As discussed in General Counsel Kent Talbert's letter to
the Comptroller General, dated October 12, 2007, while we do not
believe that any GAO personnel were involved in the improper
disclosure, we are very concerned that GAO may not have adequate
procedures in place to "prevent improper disclosure," and to follow up,
once an improper disclosure is discovered.
Under section 6.45 of the "Government Auditing Standards: July 2007
Revision," it is imperative to have a secure process for distributing
"a draft report with findings for review and comment by responsible
officials of the audited entity and others" to help "the auditors
develop a report that is fair, complete, and objective." When this
process is not followed and information is disclosed improperly, the
fairness, completeness, and objectivity of the process are seriously
threatened. As a result, General Counsel Kent Talbert's letter
requested GAO to investigate the improper disclosure immediately. He
has received no response to his letter from the Comptroller General.
Finally, the draft report does not present a complete and balanced
picture in a number of key areas, and does not accurately reflect what
actually occurred in the program during the period audited. For
example, the report criticizes the extent to which students who
participate in the program previously attended schools in need of
improvement (SINI). As previously explained to GAO, the statutory
requirement that SINI students receive priority under the program was
implemented through "scholarship lotteries." Because the lotteries must
be conducted in the spring before each upcoming school year, several
months before the District reports its SINI designations each August,
the priority group categories in the lottery are based on SINI
designations for the previous year.[Footnote 31] In the first year of
the program, the lottery occurred before there was a meaningful
determination of SINI.
Thus, especially for the first two years of the program, it is more
accurate to consider the designation of the school for the school year
in which a student spends his or her first year in the OSP--August 2004
SINI designations for the spring 2004 applicants and August 2005
designations for the spring 2005 applicants. The distinction is most
clear in the second year evaluation report. At the time of the initial
lotteries, only 5.9 percent of spring 2004 applicants were technically
from schools designated as SINI for the 2002-03 school year, but 37.1
percent of the applicants were from schools designated as SINI in
August 2004 based on those schools' performance in the 2003-04 school
year.
GAO was asked to assess the implementation of the OSP program, and we
believe that with regard to a number of key areas, in addition to the
one mentioned above concerning schools in need of improvement, the
draft report presents incomplete and, therefore, less than balanced
views in a number of important respects, including the following:
1) The success of the Department and WSF in establishing the OSP in
time for the 2004- 2005 school year. As Congress considered the bill
that would authorize the OSP program, the Department took steps to
identify the internal resources that would be needed to effectively
implement the program. Once Congress appropriated funding, the
Department was able to conduct a grant competition and award a grant to
WSF in an extremely expeditious manner”this whole process took only six
weeks, but was done in accordance with all appropriate Department
policies and procedures. WSF very promptly and effectively identified
interested private schools and formulated agreements with those schools
to participate in the program, conducted an extraordinary outreach
effort to identify families who might be eligible to participate,
assisted these families in preparing paperwork necessary to document
eligibility, provided for and conducted a lottery, and awarded
scholarships in time for the beginning of the school year.
2) The very positive responses from parents about their children's
experiences in their new schools. Studies by both the Department and
Georgetown University cite the high levels of parental satisfaction
with their experiences with the program and with their schools of
choice. According to the Department's report, using a scholarship
significantly increased parental satisfaction with their child's school
in every measured area. Seventy-four percent of scholarship parents
gave their children's schools an `A' or a `B' grade. In the Georgetown
University study, the majority of parents reported being very satisfied
with their school choice experiences, citing changes in their
children's attitudes about learning as the main source of their
satisfaction. Approximately 90 percent of the parents interviewed
reported that their children would remain in the program for at least
another year.
3) The increased demand for scholarships in each year of the program's
existence. Most recently, the demand for scholarships rose 5.5 percent
for the 2007-2008 school year over the previous year. There have been
similar increases in demand for the program in each of the previous
years, and there is now a waiting list of 400 families.
4) The extent to which private schools participating in the program are
legally permitted to maintain their traditional independence in such
areas as hiring and establishing qualifications for teachers.[Footnote
32] Consistent with the intent and language of the law, the program was
implemented by WSF in order to encourage maximum participation of
private schools to ensure sufficient capacity and options for
participating families. Private schools participating in the program
were permitted to maintain their traditional and lawful independence in
such areas as hiring and establishing qualifications for teachers,
establishing admissions criteria, and setting standards for grading and
promotion. We also note that with regard to teacher qualifications,
District regulation CDCR 5-2100.2(c) provides:
(c) Qualifications of staff: training and educational requirements for
teaching and supervisory staff must be acceptable to the Board,
although a District of Columbia teaching certificate is not required.
5) GAO's understanding of WSF explanations for certain decisions. GAO
mistakenly concluded that "WSF did not adhere to its own procedures for
making scholarship payments" based on WSF's decision to use scholarship
funds to pay tuition for students attending schools that normally
receive donor contributions to cover tuition. As WSF previously
explained to GAO, it believes that this is "a difference of
interpretation of the statute" since these schools are not "free" but
operate on contributions from private donors. The OSP provided funding
so that eligible families could choose to attend these schools and the
schools could serve these children when they would not have had the
financial resources to do so otherwise.
Comments On Specific Recommendations:
Internal Controls and Financial System:
The Secretary shall direct WSF, the grantee, to establish and implement
detailed policies and procedures to improve financial controls over OSP
grant funds, establishing compensating controls to reduce the risks of
fraud in situations where segregation of duties is not possible, and
continue its efforts to implement an integrated financial management
system.
Since the very early implementation of the program, WSF has made
significant progress in establishing more formal policies and
procedures for improved internal controls and in integrating its
financial management systems to greatly reduce the risk of waste,
fraud, and abuse. The organization's annual independent financial A-133
audits since its receipt of the federal grant have identified no
reportable conditions or material weaknesses. In a separate letter,
WSF's auditor recommended, and the organization subsequently developed,
a financial manual consolidating its policies and procedures. This
manual currently is being updated to include further details
recommended by GAO.
WSF has explained to GAO that its new billing system is fully
integrated with its student information database and has replaced the
manual procedures to which GAO refers in its draft report. A fully
improved financial system is scheduled to be in place by November 1 and
will be updated with all data for the 2007-08 school year. Department
staff will continue to work with WSF to ensure these policies and
procedures are appropriately implemented and updated, as necessary.
School Monitoring and Site Visits:
The Secretary shall direct WSF to develop and implement procedures for
conducting site visits.
Regarding GAO's recommendations for site visits by WSF, the
organization keeps records of all visits. WSF has explained to GAO that
it extensively documents problems or issues that are identified during
site visits, and takes timely corrective action. As a control to check
the accuracy of assurances and directory data, WSF is developing
specific site visit questions and related procedures.
Information to Parents:
The Secretary shall direct WSF to develop procedures to ensure that
accurate information is provided to parents before a school is chosen.
On a related GAO recommendation regarding school information provided
to parents, WSF has consulted with the private schools on an annual
basis to collect and update this information. During the first year of
the program, WSF asked each school to complete a survey and in
subsequent years asked each school to edit a printout of the previous
year's information and provide updated data. As referenced above, WSF
will include data verification questions when it conducts site visits
as a control to determine the accuracy of information that schools have
reported for the directory for parents.
Collaboration with Mayor of the District of Columbia:
The Secretary shall collaborate with the Mayor of the District of
Columbia to ensure participating schools are in compliance with all
relevant District of Columbia education and safety requirements.
GAO also recommends that the Secretary collaborate with the Mayor of
the District of Columbia to ensure that participating schools are in
compliance with all relevant District of Columbia regulations,
including education and safety requirements. The Department's July 26
written submission provided detailed information documenting that,
given the guidance set forth by Congress in the statute and
accompanying legislative history regarding private school participation
in the program, the approach taken by the Department and the District
in the Memorandum of Understanding (MOU) ” which requires participating
schools to provide assurances that they are operating lawfully within
the District ” is lawful and reasonable and fulfills the joint
oversight responsibilities set forth in the federal statute. This
approach of using assurances is commonly utilized in a number of
Department programs and is often required by law in programs
administered by the Department. The approach was designed to reduce
burden and maximize participation by eligible private schools in the
District while ensuring compliance with all applicable legal
requirements. Additionally, we note that WSF does have an established
process to review practices of private schools participating in the
program in order to address instances when concerns about participating
schools were identified.
WSF's implementation of the requirements identified in the federal
statute and MOU through its guidance and forms (including its Letter of
Agreement, Key Data Form, consultation with schools, and ongoing
consultation with the Department about the process of school
consultations and complex questions raised during them) were an
appropriate and effective exercise of grantee authority. The Department
did not have any reason to assume that assurances would not fully
reflect the school's legally operating status in the District. Given
the results of this study, however, the Department is taking steps to
ensure that the participating schools are in compliance with all
District regulations.
Department staff and the new State Superintendent of Schools (now the
District's responsible official for the Opportunity Scholarship Program
under the MOU) already have met to discuss and clarify appropriate
roles for the OSP. An important area of cooperation will be ensuring
that all schools meet the requirements of the District for operating
lawfully. We understand that the Mayor's office has directed the
Department of Regulatory and Consumer Affairs to conduct inspections,
on an expeditious basis, of the schools that GAO identified as not
having occupancy permits. Also, the State Superintendent of Schools
office is assuming responsibility for ensuring that there is
appropriate documentation for the accreditation of private schools,
including those participating in the OSP. The Department and the State
Superintendent of Schools office will revise the current MOU to reflect
these specific activities and any other responsibilities agreed upon by
the two agencies.
Program Implementation and Evaluation Design:
Finally, we address GAO's findings and recommendations on evaluation.
GAO asserts that program implementation and other factors have limited
the usefulness of and ability to generalize its evaluation findings,
and that in the future, the Department should ensure that program
implementation is more consistent with the design of a statutorily
mandated evaluation.
While we agree that some external factors did have an effect on
carrying out the evaluation as planned, GAO appears to have
misunderstood some aspects of the evaluation, leading the agency to
underestimate the evaluation's utility. Below, we have addressed GAO's
concerns by correcting misimpressions about the evaluation design in
general and its priorities.
1) The inability to compare OSP participants to all students in DCPS
schools does not undermine the more important analysis of program
impacts. GAO rightly points out that the change in the testing program
by the DC Public Schools (DCPS) prevents the evaluation from comparing
the academic achievement of scholarship recipients to the achievement
of students in DCPS in general. However, that was only one of the
analyses called for in the legislation, and it does not provide a
reliable indicator of program effects because, as the evaluation's
first year report described, DCPS students who did not apply to the OSP
are different from those who applied and are participating, making any
simple comparison between outcomes from the two groups biased and
unscientific. The main purpose of the evaluation was to estimate the
effectiveness or impacts of the program, comparing the achievement of
OSP applicants who did and did not receive scholarships through the
lotteries, and this purpose is being fulfilled.
2) The inability to include some early scholarship recipients in the
evaluation is not a major problem as GAO implies. Very early applicants
to a program are often different (e.g., more motivated or more needy)
than those who apply in subsequent years. It is true that the impact
evaluation could not include the first year OSP applicants for grades K-
5 because they were all given scholarships and therefore no control
groups were formed for them through lotteries. However, the set of
second year (cohort 2) applicants”all of whom are included in the
evaluation”is likely to provide a more reliable estimate of impacts for
the types of students who would be participating when the program is in
a "steady state" of operations. For that reason, the evaluation team
considered restricting the impact analysis to cohort 2. We included the
cohort 1 randomly assigned middle and high school applicants so that
there would be sufficient samples of students to allow impacts to be
estimated for subgroups of students (e.g., by grade band), but we
statistically allow for differential effects across the cohorts. Even
without the K-5 first year applicants, the number of students included
in the impact evaluation is actually the largest used in any rigorous
study of private school scholarships.
3) "Crossover" is a natural part of any program and randomized control
trial, not a flaw in them. GAO describes the fact that some control
group students (those who applied but did not receive a scholarship
through the lotteries) attended private schools as hindering the
analysis and interpretation of program impacts. In fact, the control
group's behavior is supposed to reflect what would have happened in the
absence of the program; students' ability to obtain access to a
scholarship other than the OSP or to find some other way to attend a
private school without the OSP is an important piece of information in
determining the impact of the offer of this scholarship. The evaluation
then uses commonly accepted statistical approaches to "net out" the
crossovers in its analysis of the relationship between attending a
private school and student outcomes, which is separate from the
analysis of program impacts.
GAO also criticizes the evaluation for failing to distinguish
scholarship recipients who did not use their scholarships because they
could not find a school placement ("forced decliners") from non-users
who declined to use their scholarship for other reasons. Since GAO
officials reviewed the evaluation's analysis plan, they know that the
study intends to address this issue through a variety of non-
experimental approaches. However, the first year there were so few of
these forced decliners that the issue did not warrant additional
analyses; these analyses are being incorporated into the next report
describing impacts after two years.
4) The concentration of scholarship recipients (including those in the
impact analysis) in a subset of private schools does not interfere with
accurate estimation of program impacts. It is neither surprising nor
unique to the DC OSP that participating students are not evenly
distributed among the participating private schools. As a policy
intervention, school scholarships provide resources to parents who
subsequently select private schools for their children. Those
participating private schools will inevitably vary in the number of
spaces available to scholarships students and in their relative
attractiveness to parents. That is, choices made by both private
schools and parents are a part of the OSP "treatment." There is no
"confounding" of program and school effects, as GAO states, because the
school effects and the program effects are one and the same; any
statistical techniques to control for school-level characteristics in
evaluating school choice programs would improperly take away most if
not all of the true treatment effect. In technical terms, the
clustering of OSP scholarship users within private schools only reduces
the efficiency of the impact estimates, and only by 2 percent as
described in the Year 1 impact report, but the substantial number of
students in the impact sample helps to offset that slight reduction in
precision.
5) The other supports the program operator provided were not organized
or used in ways that lessen the usefulness of the evaluation findings.
If the program operator (WSF) systematically provided supplemental
services (beyond the offer of a scholarship), then the evaluation would
be estimating the impacts of that combined treatment and would have a
difficult time disentangling the independent effect of the scholarship.
However, as we understand it, WSF makes additional services available
to some OSP recipients by referring them to existing community-based
organizations, but not all recipients take up the offers. For example,
WSF reported that fewer than 20 of about 80 high school first-year
scholarship users went through the process to be assigned to mentors. A
foundation provided tutoring services at two or three Archdiocese
schools, and some OSP students took advantage of that. But it is
important to note that members of the control group also had access to
and availed themselves of these kinds of programs and services offered
through their public schools or generally in their community. In fact,
random assignment should have ensured that students in both the
treatment and control groups were equally motivated to take advantage
of the supplemental services available to them. For the upcoming second
impact report, the evaluation team intends to estimate the impact of
the program on participation in other services, but the availability of
these services to both groups does not undermine the validity of the
evaluation findings.
Thank you for the opportunity to provide a written response to the
draft report. We strongly encourage GAO to consider the information
provided in our July 26 submission and in this response when preparing
the final report on the District of Columbia Opportunity Scholarship
Program. Additionally, we think it would be very helpful if GAO
provided Department officials the substantive meeting that we requested
earlier. We feel that such a meeting will result in a thorough and
balanced audit report that does not include apparent misunderstandings
that we have noted above. In any event, as indicated, we will use the
information in this report to continue to improve the Department's
oversight of the program and its coordination with the D.C. State
Superintendent's Office.
Sincerely,
Signed by:
Morgan S. Brown:
Assistant Deputy Secretary:
[End of section]
Appendix V: Comments from the Washington Scholarship Fund:
Washington Scholarship Fund:
1100 17th Street, NW:
Suite 330:
Washington, DC 20036:
(phone): 202.222.0535:
(fax): 202.222.0543:
[hyperlink, http://www.washingtonscholarshipfund.org]:
October 23, 2007:
Cornelia M. Ashby:
Director, Education, Workforce, and Income Security Issues:
Government Accountability Office:
441 G Street, N.W.:
Washington, D.C. 20548:
Dear Ms. Ashby:
Thank you for permitting the Washington Scholarship Fund ("WSF") an
opportunity to comment on the draft Government Accountability Office
(the "GAO") report entitled, "District of Columbia Opportunity
Scholarship Program: Additional Policies and Procedures Would Improve
Internal Controls, Program Operations, and Goal Attainment" (the "Draft
Report") – a copy of which was forwarded to WSF, under strict
confidentiality requirements, on October 9, 2007. WSF appreciates the
thoroughness of the Draft Report – and the GAO's express acknowledgment
that the Draft Report addresses what essentially was the "start-up"
phase of the D.C. Opportunity Scholarship Program (the OSP). That is,
the Draft Report provides, by apparent consensus, a "snapshot" of a
program that experienced initial challenges, met them, and then
thrived.
We are concerned, however, that the Draft Report does not present an
accurate picture of the OSP's great successes to date, and – from a
strictly objective perspective – contains several factual errors and
incorrect assumptions and conclusions [based at least in part on what
might be an erroneous interpretation of the OSP's federal authorizing
statute (the "OSP Statute")]. Further, the Draft Report focuses
primarily on matters that, in the vast majority of instances, frankly
bear little on the financial viability and effectiveness of the OSP. In
fact, the Draft Report's purported findings, emphases, and other points
often have little to do with the matters proposed for the GAO audit by
those requesting the audit in the first instance (or, for that matter,
with the key objectives of the OSP Statute).
As has been widely reported – including by the U.S. Department of
Education (the "DOE") in its report released this past June – the
extremely low-income families participating in the OSP over the past
three-plus years have been overwhelmingly satisfied with the program
and, more crucially, with their chosen schools. This evidence of
parental satisfaction with the OSP is corroborated by a report released
in May of this year by Georgetown University, which also found that OSP
students are more engaged in their school work and have increased self-
esteem, and that OSP parents are more involved with their children and
their chosen schools. In fact, in its first three-plus years of
operation, the OSP has fulfilled all of the key objectives of both the
federal authorizing statute[Footnote 33] and the Memorandum of
Understanding (the "MOU") between the DOE and the District of Columbia
(the terms of which largely do or should govern the school oversight
matters on which the GAO so closely focuses in the Draft
Report).[Footnote 34]
Ultimately – with reference to what we understood to be the very
rationale for the GAO's audit - we must emphasize that the Draft Report
cites absolutely no evidence that federal OSP funds have been spent for
anything other than genuine educational purposes at any time during the
OSP's three-plus years of operation, or that OSP families have not
received the educational services they sought through participation in
the OSP. More directly – again, given the GAO's apparent focus – the
GAO presented evidence that the financial or other instability of any
school participating in the program has deprived any OSP family of the
education they sought for their children through an exercise of
meaningful educational choice. (To the contrary, and again, OSP
families overwhelmingly are satisfied with their schools and their
children's progress and growth in their schools of choice.)
WSF acknowledges that some of the GAO's observations and
recommendations in the Draft Report (and even some of its findings) –
particularly those relating to the implementation of the OSP during the
program's challenging initial period of operation – are valid and
useful. In fact, as the GAO is aware (and again, as the GAO has
acknowledged), WSF already has taken many of the steps recommended by
the GAO toward more efficient operation of the OSP. On this point,
again worthy of great emphasis, and again as acknowledged by the GAO
itself: Even before the GAO initiated its year- long audit in September
of 2006, WSF already had implemented many of the sorts of efficient and
refined systems required to meet most of the GAO's recommendations.
Certainly, WSF will continue to improve the OSP in our steadfast effort
to make the program a more than viable option for District families
seeking quality educations for their children. Naturally, toward this
end, WSF looks forward to addressing and implementing the GAO's
remaining findings and recommendations in close cooperation with both
the DOE and the District of Columbia Government – both of which have
been, and have committed to remaining, close and supportive partners of
WSF in our continued successful implementation of the OSP.
We provide below a more detailed response to the GAO's Draft Report. In
our response, we first review the questions posed to the GAO by
Senators Kennedy and Durbin and Delegate Holmes Norton in their initial
audit request to the GAO (Section I). We then provide our responses to
the specific purported factual findings set forth by the GAO in the
Draft Report (Section II). Finally, we set forth our response to the
recommendations made by the GAO in the Draft Report (Section
III).[Footnote 35]
I. Questions Posed to the GAO, and Scope of the Resulting
Audit/Investigation:
As stated in the Draft Report, the GAO was requested to answer/address
three specific questions/matters (Draft Report, p. 2):
To assess the implementation of the program, including the (1)
accountability mechanisms in place governing the use of funds, (2)
results of the grantee's efforts to meet the program 's recruiting and
eligibility requirements and inform parents about their choices, and
(3) extent to which the evaluation reflects statutory requirements and
the implementation of the program supported the detection of useful and
generalizable findings.
The GAO in its Draft Report has not addressed the results of WSF's
efforts ” and successes ” in meeting the OSP Statute's priorities:
1. Give priority to students coming from schools identified for
improvement
In year one of the program, 29% of OSP scholarship recipients had been
identified as having come from schools identified as in need of
improvement ("SINI" schools) in 2003 or 2004. Only 4% were identified
at the time of the initial lottery because the 2004 list of SINI
schools was published after the independent federal evaluators, under
the DOE's Institute of Educational Sciences' ("IES") (the "IES
Evaluators"), held the first lottery on June 17, 2004. (The IES
Evaluators described this situation in their first-year report on the
OSP).
In the current 2007-2008 school year, 83% of students participating in
the OSP would be in schools that have failed to meet adequate yearly
progress if it was not for their OSP scholarships.
In the initial OSP lottery, the IES Evaluators ” who designed and
administered the lottery ” gave priority to students in SINI schools.
In year one of the OSP:
* 100% of eligible OSP applicants from a school needing improvement
received a scholarship;
* 2 out of 3 eligible students from public schools in grades 6 to 8
received a scholarship; and:
* 3 out of 10 eligible students from public schools in grades 9-12
received a scholarship.
2. Target resources to families that lack the financial resources to
take advantage of available educational options
* Nearly 20% of eligible District students have applied for OSP
scholarships in the program's first four years of operation.
3. Provide students with the widest range of educational options
* The GAO itself reports that 78 of the 86 non-public schools in the
District (80%) participate in the OSP.
II. WSF's Responses to the GAO's Purported Factual Findings:
In this section, WSF will state the GAO's purported factual findings
(numbered and in bold italics), and then, seriatim, will set forth our
responses to those purported findings.
1. GAO Purported Finding: WSF's internal controls policies and
procedures and systems did not provide adequate accountability over the
use of grant funds:
WSF acknowledges that during the initial start-up of the OSP, WSF's
internal control policies and procedures, systems, and internal control
activities were not optimal, including:
* Several procedures and electronic systems that relied on manual data
transfers, increasing the risk of human error; and:
* Policies and procedures which were not documented in detail.
As the GAO notes (Draft Report, pp. 12 and 14), "Due to the need to
quickly implement the program, WSF had little time to develop internal
control policies and procedures, systems, and internal control
activities," and "had to rely on the knowledge of key staff to perform
daily operations." The demands placed on WSF and our staff in
attempting to get the OSP up and running – within five months of the
date WSF was appointed the administrator of the OSP and the first day
of the then-upcoming (2004-2005) – were enormous. As the OSP includes
many characteristics unique and innovative to scholarship programs,
WSF's policies, procedures and systems evolved as the program matured.
This learning curve made documenting policies and procedures as they
evolved nearly impossible. In addition, the limited operating funds
provided under the OSP Statute – which provide only about 16% of the
actual cost of operating the OSP and serving our students and families
– did not afford WSF the staff resources necessary to both create and
implement the program, and to document our efforts simultaneously
during the 14-month launch period. Indeed, WSF has been required to
raise substantial private funding in order effectively to operate the
OSP.
WSF is appreciative of the GAO's recognition that WSF has already made
major improvements in these areas, having made significant financial
investments in systems and infrastructure improvements in advance of
the request to GAO for the program review. To date, WSF has nearly
completed the full electronic and real-time integration of its data
(already operational) and financial systems (launching November 1,
2007). The appropriate detailed policies and procedures will be
finalized once tandem financial systems have been in operation for two
months (January 2008).
Further, WSF welcomes the GAO's suggestions for additional operational
improvements as set forth on pages 17-18 of the Draft Report concerning
revisions to its Internal Systems Review, and will adopt those
suggestions.
2. GAO Purported Finding: As a result of the manual and disparate
policies, procedures and systems, some payments were not made in
accordance with the act, while others raise questions about compliance:
While WSF agrees that the deficiencies in internal controls and
accountability mechanisms increased the risk of errors, WSF firmly
represents that no federal monies were spent irresponsibly or
improperly used for purposes other than OSP scholarships. The GAO
specifies three areas where it believes that the lack of internal
control mechanisms resulted in errors in the use of federal
appropriations (pp. 12 & 21):
(a) WSF's payment of scholarships for schools that are "scholarship-
only"
Payment of scholarships for schools which GAO identifies as
"scholarship-only" was a result of considered choices, not a result of
any lack of internal control mechanisms. WSF believes that this choice
is in complete accord with the OSP Statute.
Three participating OSP schools (Washington Middle School for Girls,
San Miguel, and Washington Jesuit Academy) are "scholarship-only"
schools – that is, the educational costs for all of the children who
attend those schools arc paid for by scholarships funded by third-party
donors. GAO contends that allowing OSP scholarships to be used for
these programs violates Section 307(a) of the Act, D.C. Code § 38-
1851.06(a)(1), which provides in part that:
Each grantee shall ensure that the amount of any tuition or fees
charged by a school participating in the grantee's program under this
chapter to an eligible student participating in the program does not
exceed the amount of tuition or fees that the school customarily
charges to students who do not participate in the program.
WSF respectfully submits that the GAO's reading of this provision is
legally incorrect. GAO's premise is that if a student receives a
scholarship to a school and thus does not personally pay tuition, then
allowing any use of OSP funds for that school would not be permissible
because non-scholarship students "pay no tuition." But this argument
cannot be what Congress intended, since virtually all schools that
participate in the OSP provide scholarships to some students. GAO's
reading of Section 307(a)(1) would mean that students who receive
scholarships at other schools are viewed as "paying tuition" for
purposes of Section 307(a)(1). There is no logical or legal basis for
applying a different standard to students attending the three "all-
scholarship" schools receiving non-OSP funding than the standard
applied to students receiving non-OSP assistance attending any other
participating school.
The purpose of Section 307(a)(1) is to prevent a school from
"overcharging" OSP students – a concern that is absent in the "all-
scholarship" situation. In the cases of the above-cited schools, the
simple fact is that tuition is not normally paid by the families, but
by other sources similar to OSP scholarships. In fact, the above-cited
schools list tuition amounts for donors and on their websites and in
other materials readily available to the public. As with most other
participating OSP schools, those published tuitions are below the
actual costs of serving the students in those schools.
While the Draft Report does not specify the reasoning behind the GAO's
objection to allowing OSP scholarships at the "all-scholarship"
schools, the draft report refers to these schools as "free." This
reflects a misunderstanding of the way these schools operate. They are
not, in fact, "free" but require money to operate which they collect
from public sources and private donors. If OSP students did not have
OSP scholarships to fund their educations at these schools, those
student could not attend unless other donors stepped up, diverting
funds from a non-OSP student. In other words, OSP scholarships provide
the resources for these schools to educate additional children (i.e.,
the OSP students). WSF does not believe that it would discharge – and
indeed would defy – its statutory responsibility if it disqualified
these three schools from participation in the OSP simply because all of
these schools' students receive scholarships.
In addition, the GAO states that by listing the published tuitions for
these three schools in the WSF School Directory, parents are misled to
believe that they could not attend/afford those schools if they did not
have an OSP scholarship. As OSP families arc provided a copy of the WSF
School Directory when they have completed their initial application for
an OSP scholarship, however, this could not logically affect a family's
decision to apply.
(b) WSF allowing scholarship funds to be used for after-school programs
without demanding proof of "academic support activities":
Again, WSF notes that this represents a considered choice by WSF in its
implementation of the OSP – not any lack of internal control
mechanisms. WSF believes that this choice was and remains fully in
accordance with the OSP Statute.4 School fees are defined under the OSP
Statute as "fees charged by the school that the student is enrolled
in," and "are charged at the same rate as non-OSP students" – in
contrast to the GAO's own definition, which does not comport with the
governing federal statute.
WSF permits the use of OSP scholarship funds for before- and after-
school programs that are offered to other, non-OSP students at
participating schools. The GAO states (Draft Report, p. 12) that a
"key" element of an allowable fee under the OSP Statute and WSF policy
is that the fee is "tied to the student's academic program." The GAO
further states (Draft Report, p. 21) that unless there is proof that
these programs "included any academic support activities," then "before-
and-after care would not meet the requirement that a fee must
contribute to a student's academic success."
WSF respectfully submits that the GAO appears to have misunderstood
and, therefore, misstated WSF's policy for school fees (as well as the
OSP Statute's mandate). School fees are defined as those that support
the academic success of the student. WSF vigorously disagrees with the
GAO's conclusion that there must be formal "academic support
activities" in order for before- and after-school programs to
contribute to students' success. Many studies have shown that before-
school and after-school programs play an important role in student
academic success by providing a supportive and supervised environment.
For example:[Footnote 36]
Research Findings: Studies on the Effects of After-School Programs on
Students' Academic Success and Social Integration Researchers:
Table: Research Findings: Studies on the Effect of After-School
Programs on Students's Academic Success and Social Integration
Researchers:
Researchers: Dr. Joseph Maloney and Dr. Heather Lord, Yale University;
Findings: "Children characterized by the After-School Program (ASP)
care arrangement showed significantly higher reading achievement at the
end of the school year compared with children in all other patterns of
after-school care [e.g. relatives] and were rated by teachers as
compared with children in the other adult/non-adult care pattern.";
Study: An Ecological Analysis of After-School Program Participation and
the Development of Academic Performance and Motivational Attributes for
Disadvantaged Children.
Researchers: Erica Carryl, New York University;
Findings: "The findings supports a conclusion that among the typical
after-school care arrangements poor children experience ASP's appear
unique in their ability to promote academic-related success;
Study: Published in 2005 by the Society for Research in Development,
Inc.
(c) Certain alleged paperwork errors in the sample of scholarship files
reviewed by the GAO caused OSP funds to be spent that were not in
accordance with the Act
WSF believes that the deficiencies noted are minor in nature, and do
not indicate substantive problems with the disbursement of federal
funds.
The GAO reviewed a sample of 50 student files for the 2005-2006 school
year, and asserts (Draft Report, pp. 19-20) that it found certain
deficiencies in the files: (i) Missing signatures or dates on fee
approval forms; (ii) missing documents, such as school placement
letters; and (iii) missing student identifier numbers. WSF has not been
provided with the specifics of these alleged deficiencies and thus
cannot respond in detail. WSF notes, however, that most of the alleged
deficiencies are non-substantive (e.g., missing dates, missing student
identifier numbers) and do not reflect erroneous distribution of
federal funds.
The GAO does indicate that it found one instance of payment of a fee
that was not listed on the payment detail report, and one instance of a
payment made on the basis of a payment detail report that had not been
certified by a school official. Again, WSF does not know the details of
these two situations, or the dollar amounts involved ” but regardless,
and again, these situations under any objective analysis must be
considered only minor clerical oversights.
WSF also would submit that the incidence of error is inflated by the
GAO's counting a missing signature on one fee form as 20 errors if the
fee was charged to 20 students' scholarships.
3. GAO Purported Finding: While OSP students generally met the
program's eligibility requirements, WSF (a) had limited success in
meeting recruiting priorities and (b) did not provide parents complete
information:
(a) Recruiting priorities:
WSF shares the concerns about outreach to District students in "schools
in need of improvement," and certainly has sought assiduously
throughout the first three-plus years of OSP implementation to
prioritize students coming from failing schools. But given the newness
of the No Child Left Behind Act at the time of the OSP's initiation in
2004, and the timing of the program's implementation with respect to
testing requirements in the District, no school could have been
identified as "in need of improvement" by the time the IES Evaluators
needed to perform the first OSP lottery (it takes two years of a school
failing to meet "adequate yearly progress" for the school to be
identified as "in need of improvement").
The IES Evaluators' report on the OSP for the first year of the OSP
identifies this challenge, and provides data showing that 29% of OSP
applicants for the 2004-2005 school year were enrolled the year before
in SINI schools.
With respect to the analysis in the Draft Report, WSF notes that it
cannot be determined from the data set forth whether students from SINI
schools are in fact under-represented among OSP students. As noted by
the GAO, the great majority of OSP students are at the elementary
school level, and the percentage of D.C. elementary school students who
are enrolled in SINI schools is much lower than the percentage of D.C.
middle school and high school students who are enrolled in SINI schools
(Draft Report p. 26). Yet when the GAO compares the percentage of OSP
students from SINI schools to the overall percentage of students from
SINI schools, it erroneously uses as its benchmark all D.C. students,
rather than performing a grade-by-grade comparison. This "apples versus
oranges" methodology skews the results toward finding an under-
representation of SINI students in the OSP, when a more appropriate
grade-by-grade comparison might show very different results.
WSF also submits that the OSP Statute sets forth no specific student
recruiting requirements. In fact, the only provision in the OSP Statute
relating to such matters is contained in Section 305(b)(1)(G), which
requires the grantee [WSF] to address how it will "seek out private
elementary schools and secondary schools in the District of Columbia."
(b) Alleged failure of WSF to perform its financial oversight role
WSF's limited oversight role is defined by Section 305(b)(l)(H) of the
OSP Statute, which requires WSF to ensure that "participating schools
are financially responsible and will use the funds received under [the
OSP Statute] effectively." Based on an unsubstantiated and vague
assertion that "certain documents" were missing from files reviewed by
GAO staffers, the GAO concludes that WSF "did not fulfill its oversight
role of ensuring that participating schools are financially stable"
(Draft Report, p. 32).[Footnote 37]
WSF first must note that the GAO's unfounded conclusion rests on a
misinterpretation of the OSP Statute. The OSP Statute, as cited above,
requires WSF to determine the "financial responsibility" of schools,
not to "ensure" their "financial stability" (which obviously would be
impossible for WSF to do).
In any event, WSF emphatically disagrees with any conclusion that it
has not performed its statutorily- prescribed oversight function. WSF
requires the submission of financial information from every school;
based on those documents and certain statistical "triggers" (such as
high rates of students choosing to leave a school), WSF undertakes more
thorough financial reviews of individual schools in cases where it
appears that financial problems might exist. As a result of such
financial reviews, WSF has disqualified one school from further
participation in the program, and has placed two other schools on a
"pay-as-you-go" basis, whereby these two schools receive federal funds
only after they have provided educational services. As a result of
these careful financial reviews by WSF, not a single dollar of federal
funds has gone to a school unable to provide the educational services
for which federal scholarship funds were spent.
In addition, despite what the GAO seems to suggest, WSF is unclear as
to the ability of a school's clean audit to ensure that the school will
be an ongoing enterprise beyond a given school year. A financial audit
indicates only that an organization has represented its financial
statements fairly and accurately, and has used industry approved
standards; a clean audit by no means "ensures financial stability."
4. GAO Purported Finding: Alleged failure of WSF to ensure that schools
have "certificates of occupancy" and have supplied information to the
District about their instructional programs:
The GAO asserts in the Draft Report (p. 33) that WSF shares in the
responsibility for ensuring that private schools are "operating
lawfully" in the District of Columbia. WSF disagrees with this
conclusion. WSF does not have the authority to enforce District of
Columbia law; nor, for that matter, does WSF have the resources or the
expertise to duplicate the functions of the DOE and/or the District
Government. WSF does, of course, look forward to continuing the very
productive relationships it has forged with the DOE and the District,
and to coordinating on better ways to communicate regarding the status
of District non-public schools participating in the OSP.
Further, WSF is concerned by the GAO's assertion that the GAO's
inability to locate two schools' certificates of occupancy is an
indication that certain schools researched by the GAO may not be
operating legally (Draft Report, pp.8 and 33-34).
Better perspective suggests that this likely may be a matter of
deficiencies in the recordkeeping concerning certificates of occupancy
and annual reports required by the D.C. compulsory school attendance
regulations. Tellingly, there is no indication in the Draft Report that
the GAO checked directly with the schools to determine whether they had
been issued certificates of occupancy. In fact, The Washington Post
noted on Sunday, October 21, 2007, that Pepco and Verizon will not
start power or phone services for a school until the D.C. Department of
Consumer and Regulatory Affairs has issued a certificate of occupancy.
See David Nicholson, "D.C.'s Learning Curve," Washington Post, page B4
(October 21, 2007).
Indeed ” as concrete support of the notion that many of the issues
broached by the GAO have more to do with recordkeeping than with
legality ” WSF is pleased to report that we have located a certificate
of occupancy for one of the schools (Academy De La Recta Porta,
"ADLRP") referenced in the Draft Report. This certificate clearly
indicates that the space used by ADLRP is approved for educational use.
(The referenced certificate of occupancy is attached hereto as an
Addendum.) It is apparent that the GAO was unable to locate ADLRP's
certificate of occupancy because the certificate was filed under the
name of the church that runs the school (New Dimensions Kingdom
Ministries) rather than under the name of the school itself. WSF
located the ADLRP certificate of occupancy simply by telephoning
ADLRP's principal, who then forwarded the certificate to WSF via
facsimile within ten minutes of the phone call.
5. GAO Purported Finding: Allegation that WSF did not provide parents
with "complete information" about participating schools:
The GAO asserts that WSF provided parents with incomplete and incorrect
information about participating schools (Draft Report, pp. 35-36). WSF
concurs that some typographical errors existed in the WSF School
Directory, and will engage in additional proofing upon the next
publication of the directory. WSF does not, however, regard this
directory as the sole or even primary tool used by parents in reviewing
their educational options. In fact, WSF strongly encourages families to
visit any school to which they intend to apply, and to talk to other
families about their experiences in those schools.
WSF understands that the GAO suggests that participating schools are
obligated to publish information concerning the academic achievements
of their students and that WSF should disseminate this information to
parents. The OSP Statute, under Section 310(c), appropriately assigns
to participating schools the responsibility of reporting to individual
parents about the performance of students and school safety.
As the GAO acknowledges, WSF provides parents with information on
enrolled students' academic achievement, as measured by a comparison
with the aggregate academic achievement of other students at the
student's school in the same grade or level on an annual basis.
Further, WSF follows up with each participating school at the end of
each school year and receives certification that each school provided
the required information.
6. GAO Purported Finding: Evaluation contractors adopted a strong
methodology, but program implementation and other factors limited the
usefulness and generalizability of findings:
For the most part, WSF will not respond to the GAO's findings in this
section of the Draft Report, believing that these statistical issues
are best left to the DOE and the IES Evaluators.
WSF is compelled, however, to comment on the Draft Report's possible
implication that WSF somehow acted improperly by implementing decisions
that were "made to advance program goals," because these steps made
statistical assessment of the program's results more complex (Draft
Report, p. 44). WSF was instructed directly by the IES Evaluators in
any matters that might affect the evaluation, and in fact the IES
Evaluators themselves structured and implemented the lotteries through
which the study has been populated.
Regrettably, the GAO wholly omits from its Draft Report any discussion
of these indicators and other evidence clearly showing the success of
the program.
III. WSF's Responses to the GAO's "Recommendations for Executive
Action":
1. Establish and implement policies and procedures to improve financial
controls over OSP grant funds, including specific requirements for the
process of approving scholarship payments and documentation of the
process:
As acknowledged in the Draft Report, and as discussed in this response,
WSF is well underway in implementing many of the recommended
improvements to the program systems identified by the GAO. WSF will
complete its policies and procedures manual once this implementation is
completed.
WSF has received a clean opinion in all three A-133 audits performed
since the OSP was implemented, and expects once again to receive a
clean opinion in its 2006-2007 audit, currently underway.
2. Establish compensating controls, such as supervisory review, to
reduce the risk of fraud in situations where segregation of duties is
not possible due to the size limitations of OSP's staff so that no one
employee can authorize, process, review, and have access to the funds
relating to OSP:
Again, as acknowledged in the Draft Report, and as discussed in this
response, WSF is well underway in implementing the improvements
recommended by the GAO, and compensating controls and access to
financial management systems.
3. Continue its efforts to implement an integrated financial management
system to facilitate processing and recording of scholarship payments
and overall financial reporting:
As acknowledged in the Draft Report, and as discussed in this response,
WSF is well underway in implementing many of the recommended
improvements to the program systems identified by the GAO. WSF will
finalize its policies and procedures manual once this implementation is
completed.
As noted above, WSF has received a clean opinion in all three A-133
audits performed since the OSP was implemented, and expects once again
to receive a clean opinion in its 2006-2007 audit, currently underway.
4. Develop and implement procedures for conducting site visits,
including that site visit reports be prepared to contain information on
the overall financial stability of the school:
WSF does not have the resources to perform a financial analysis "on the
overall financial stability" of every school participating in the OSP,
nor does WSF have the resources to visit every participating OSP school
every year.
WSF does look forward to continuing our very close and productive
relationships with the DOE and the District Government to find ways to
provide more robust information about schools and options to families.
In addition, WSF will expand its documentation of the school visits
that are performed throughout the year. WSF provided the GAO with a
list of 42 school visits performed in the 2005- 2006 school year,
including the dates of the visits and the names of the WSF staff
attending the visits; however, the GAO puzzlingly reported that WSF did
not provide evidence of those visits.
5. Develop procedures to ensure that accurate information is provided
to parents before a school is chosen about the summary achievement data
of students, teacher qualifications, and tuition levels, and that
schools make such information available on an annual basis to parents
of enrolled students:
WSF will re-evaluate the methodology by which school information is
collected. Schools are required to report summary achievement data to
families of enrolled students so that families can gauge their
children's progress. WSF asserts that this is an appropriate forum
through which to provide families with this information.
WSF, as the administrator of a federal program, has no expertise in
evaluating the results of testing data, or in evaluating such data as
between schools.
In addition, it requires intimate knowledge of each student to identify
a productive school-student match. WSF has found that the best such
matches are made when families, students, and schools work together
directly.
WSF will continue to work with the DOE and the District to provide OSP
families access to the resources that will assist parents in making the
most informed, and best possible, educational choices for their
children.
Again, the Washington Scholarship Fund appreciates this opportunity to
respond to the GAO's Draft Report. Naturally, we welcome any questions
or comments the GAO might have regarding our responsive commentary.
Sincerely,
Signed by:
Gregory M. Cork:
President and CEO:
The Washington Scholarship Fund:
Figure:
This figure is a copy of a certificate of occupancy.
[See PDF for image]
Source: District of Columbia Government.
[End of figure]
[End of section]
Appendix VI: Contact and Staff Acknowledgment:
GAO Contacts:
Cornelia M. Ashby, Director (202) 512-7215 or ashbyc@gao.gov:
Jeannette M. Franzel, Director (202) 512-9471 or framzelj@gao.gov:
Acknowledgments:
Education, Workforce, and Income Security Team: Betty Ward-Zukerman,
Assistant Director; Kathleen D. White, Analyst in Charge; Carolyn S.
Blocker, Analyst, Romonda S. McKinney, Analyst; and Susannah Compton,
Communication Analyst.
Financial Management and Assurance Team: Norma Samuel, Assistant
Director; Sandra Silzer, Analyst in Charge; Nicole McGuire, Analyst;
and Maxine Hattery, Communication Analyst.
In addition, Jean McSween, Karen O'Conor, and Anna Maria Ortiz advised
on methodological and statistical aspects of this report; Cynthia Grant
performed the data analyses; and Jessica Botsford, Abe Dymond, Doreen
Feldman, and Jacqueline Hamilton advised on legal aspects.
[End of section]
Related GAO Products:
D.C. Charter Schools: Strengthening Monitoring and Process When Schools
Close Could Improve Accountability and Ease Student Transitions. GAO-
06-73. Washington, D.C.: November 17, 2005.
Charter Schools: Oversight Practices in the District of Columbia. GAO-
05-490. Washington, D.C.: May 19, 2005.
Public Schools: Comparisons of Achievement Results for Students
Attending Privately Managed and Traditional Schools in Six Cities. GAO-
04-62. Washington, D.C.: October 29, 2003.
School Vouchers: Characteristics of Privately Funded Programs. GAO-02-
752. Washington, D.C.: September 10, 2002.
School Vouchers: Publicly Funded Programs in Cleveland and Milwaukee.
GAO-01-914. Washington, D.C.: August 31, 2001.
[End of section]
Footnotes:
[1] Public Law No. 108-199, title III, 1118 Stat. 3, 126 (Jan. 23,
2004).
[2] The No Child Left Behind Act of 2001 reauthorized the Elementary
and Secondary Education Act.
[3] The participation reports and analysis plan were prepared by Westat
in affiliation with Georgetown University and Chesapeake Research
Associates; the program evaluation was prepared by Westat in
affiliation with the University of Arkansas, Georgetown University, and
Chesapeake Research Associates.
[4] Under the No Child Left Behind Act, if a school that receives Title
I funds fails to meet certain performance goals for 2 years, it is
designated as in need of improvement. Schools that do not meet these
goals in 4 years are subject to corrective action and after 5 years,
are subject to restructuring. This report uses the term "in need of
improvement" to refer to all three categories.
[5] D.C. Municipal Regulations 2100.2 and 2100.3.
[6] Accountability can be defined as an organization's ability to
effectively demonstrate to both internal and external parties that
resources are managed properly, programs are achieving intended goals
and outcomes, and programs are being provided effectively and
efficiently. To achieve accountability, organizations need internal
control activities that provide reasonable assurance that (1)
operations are effective and efficient, (2) financial reporting is
reliable, and (3) compliance with laws and regulations is being
achieved. GAO/AIMD-00-21.3.1.
[7] OSP is the largest of WSF's scholarship programs. WSF currently
provides scholarship funds to about 2,400 students, over 1,800 of which
receive OSP scholarships.
[8] The concept of segregation of duties can be defined as the division
or segregation of key duties and responsibilities associated with
financial operations among different members of the staff to reduce
risk of error or fraud. For example, the responsibilities for
authorizing transactions, processing and recording them, reviewing the
transactions, and handling any related aspects should be performed by
different persons. No one individual should control all key aspects of
a transaction or event.
[9] Schools entered the tuition and fee charges for each student into
WSF's online billing system at the beginning of the school year. Some
of the fees commonly included were for uniforms, field trips, before-
and-after care, cafeteria services, and transportation.
[10] In addition, WSF only allowed students entering grades
kindergarten through grade 5 to apply for scholarships for the 2006-
2007 school year because of the shortage of high school openings.
[11] Each year, WSF has included a brochure on OSP with other
information on school choice that the District of Columbia Public
Schools or State Education Office has mailed to families with children
in District public schools.
[12] During the third year, WSF only allowed students entering grades
kindergarten through 5 to apply because there were few placements
available at the high school level and students in cohorts 1 and 2 were
matriculating into these grades. During the fourth year, WSF limited
applications to students in grades kindergarten through grade 7.
[13] We cannot report specific demographic characteristics of students
offered scholarships because the data we received from both WSF and
Westat contained large amounts of missing data.
[14] During the first year of the program, WSF reported that an audit
had found a small number of students erroneously received scholarship
offers whose household incomes exceeded 185 percent of the poverty
level at the time of application and who were, therefore, not
financially eligible for the program. WSF indicated it subsequently
used funds from its private scholarship program to allow these students
to continue private school participation.
[15] With the approval of Education's General Counsel, WSF awarded some
scholarships to children whose parents planned to place them in a
participating preschool or prekindergarten as long as the children
satisfied the District's age requirements to enter kindergarten and the
preschool was best fitted to meet their needs.
[16] Stephen Q. Cornman, Thomas Stewart, and Patrick Wolf, The
Evolution of School Choice Consumers: Parent and Student Voices on the
Second Year of the D.C. Opportunity Scholarship Program (Georgetown
Public Policy Institute, May 2007).
[17] Under the rules of the program, the household income of applicants
could not exceed 185 percent of the poverty level at the time of
application, but participants could continue in the program as long as
their household income did not exceed 200 percent. In December 2006,
Congress passed legislation that raised the limit on household income
for students who had enrolled during the program's first 2 years from
200 percent of the federal poverty level to 300 percent of the federal
poverty level. This allowed K-12 students who would have "earned out"
of the program to maintain their eligibility.
[18] According to the National Center for Education Statistics, the
District of Columbia had 87 private schools as of the 2003-2004 school
year; 70 of these schools have participated in OSP, but not all schools
have participated in all years. The schools that are not participating
include some that are highly specialized, such as a ballet school, and
some that serve only students with severe disabilities.
[19] We used the School Directory D.C. K-12 Scholarship Program: 2004-
2005 School Year, (Washington, D.C.: Washington Scholarship Fund), for
our analysis because the directories for the 2006-2007 and 2007-2008
school years appear to inaccurately report data on teachers' degrees
for many schools.
[20] Stephen Q. Cornman, Thomas Stewart and Patrick Wolf, The Evolution
of School Choice Consumers: Parent and Student Voices on the Second
Year of the D.C. Opportunity Scholarship Program (Georgetown Public
Policy Institute: May 2007).
[21] Each annual directory described the general characteristics of
participating private schools, such as their enrollment, enrichment
programs, and access to public transportation.
[22] The Act required that schools provide parents with information on
enrolled students' academic achievement, as measured by a comparison
with the aggregate academic achievement of other students at the
student's school in the same grade or level on an annual basis.
[23] These schools were specifically designed to help low-income
children and typically enlist financial contributions from
institutional and individual donors in order to operate. Other than a
modest fee, students are not required to pay any tuition or other
costs. These schools may provide academic support, tutoring, extended
school days, Saturday classes or enrichment activities, summer
programs, uniforms, and free meals.
[24] As noted earlier in the report, we found that WSF's use of OSP
funds to pay tuition for students attending schools that do not
normally charge students tuition is not in accordance with the Act.
[25] Education and WSF considered eligible students who would be old
enough to attend kindergarten when scholarships were available as
public school students. They did not receive any additional priority in
the placement lotteries, regardless of the designation of the public
school they would have normally attended.
[26] We used the lists of schools identified as in need of improvement
at the time the lotteries were held to compare OSP scholarship
recipients to District public and charter school students---the pool
from which they were to have been recruited--in terms of the percentage
of students from schools designated in need of improvement.
[27] See Consolidated Appropriations Act for Fiscal Year 2004 (H.R.
2673)/P.L. 108-199, and House Rept. 108-401, Div. C, Title III.
[28] The MOU stipulates that only schools "operating lawfully in the
District" may participate in the program.
[29] See DCMR 2100.2.
[30] DCMR Title 12, 2003 Building Codes Section 110A and DCMR Title 11,
Section 3203.
[31] For example, SINI students in the cohort 1 (spring 2004) lottery
had to be grouped based on SINI designations made in August 2003, using
performance data from the 2002-03 school year. But 2 months after the
lottery, significantly more schools were designated for SINI for 2004,
and it is this later number that provides a more accurate
representation of which schools were low performing when students in
cohort 1 were applying to the OSP.
[32] We also note that responsibility for many of the other concerns
mentioned throughout the GAO report about decisions made during the
implementation of the OSP by the Department, the District, and WSF were
outside of the control of those implementing the program, e.g., the
number of high school slots available for scholarship recipients.
GAO raised a concern about the number of students receiving
scholarships during the first year of the program that already attended
private schools. These students met the definition of "eligible
student" in the statute, and Congress did not limit participation in
the program to eligible students attending public schools.
[33] More specifically, the OSP has served the District's lowest-income
students; it has prioritized students coming from failing public
schools as determined by federal No Child Left Behind Act criteria; and
it has supported the rigorous evaluation of the OSP mandated by the
federal authorizing statute.
[34] For example, among the primary points on which WSF disagrees with
the GAO is the GAO's assumption that WSF should function as a "Super
School Board" that subjectively determines the educational value of
each school participating in the OSP. Such a role for WSF is permitted
neither by the OSP Statute, nor by D.C. law, nor by any other
applicable statute or regulation ” and in any event, under any
circumstances, would be wholly inappropriate.
[35] In the interests of economy and relevance, WSF will not attempt to
respond to each and every factual, logical, or other error in the Draft
Report. Rather, we focus in our response on those Draft Report
inaccuracies and weaknesses that appear most likely to create a
misleading impression about the overall functioning and success of the
OSP, and that most require correction or attention in the public
interest.
[36] Thus, WSF does not believe that these two legal issues are within
the scope of the inquiry requested by the Members of Congress into
"accountability mechanisms."
[37] It is not clear as to which part of the Congressional audit
request the GAO's discussion of this issue is responsive.
GAO's Mission:
The Government Accountability Office, the audit, evaluation, and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, DC 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Gloria Jarmon, Managing Director, JarmonG@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, DC 20548:
Public Affairs:
Susan Becker, Acting Manager, BeckerS@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, DC 20548:
