Juvenile Justice
Technical Assistance and Better Defined Evaluation Plans Will Help Girls' Delinquency Programs
Gao ID: GAO-10-133T October 20, 2009
This testimony discusses issues related to girls' delinquency--a topic that has attracted the attention of federal, state, and local policymakers for more than a decade as girls have increasingly become involved in the juvenile justice system. For example, from 1995 through 2005, delinquency caseloads for girls in juvenile justice courts nationwide increased 15 percent while boys' caseloads decreased by 12 percent. More recently, in 2007, 29 percent of juvenile arrests--about 641,000 arrests--involved girls, who accounted for 17 percent of juvenile violent crime arrests and 35 percent of juvenile property crime arrests. Further, research on girls has highlighted that delinquent girls have higher rates of mental health problems than delinquent boys, receive fewer special services, and are more likely to abandon treatment programs. The Office of Juvenile Justice and Delinquency Prevention (OJJDP) is the Department of Justice (DOJ) office charged with providing national leadership, coordination, and resources to prevent and respond to juvenile delinquency and victimization. OJJDP supports states and communities in their efforts to develop and implement effective programs to, among other things, prevent delinquency and intervene after a juvenile has offended. For example, from fiscal years 2007 through 2009, Congress provided OJJDP almost $1.1 billion to use for grants to states, localities, and organizations for a variety of juvenile justice programs, including programs for girls. Also, in support of this mission, the office funds research and program evaluations related to a variety of juvenile justice issues. As programs have been developed at the state and local levels in recent years that specifically target preventing girls' delinquency or intervening after girls have become involved in the juvenile justice system, it is important that agencies providing grants and practitioners operating the programs have information about which of these programs are effective. In this way, agencies can help to ensure that limited federal, state, and local funds are well spent. In general, effectiveness is determined through program evaluations, which are systematic studies conducted to assess how well a program is working--that is, whether a program produced its intended effects. To help ensure that grant funds are being used effectively, you asked us to review OJJDP's efforts related to studying and promoting effective girls' delinquency programs. We issued a report on the results of that review on July 24, 2009. This testimony highlights findings from that report and addresses (1) efforts OJJDP has made to assess the effectiveness of girls' delinquency programs, (2) the extent to which these efforts are consistent with generally accepted social science standards and federal standards to communicate with stakeholders, and (3) the findings from OJJDP's efforts and how the office plans to address the findings. This statement is based on our July report and selected updates made in October 2009.
With an overall goal of developing research that communities need to make sound decisions about how best to prevent and reduce girls' delinquency, OJJDP established the Girls Study Group (Study Group) in 2004 under a $2.6 million multiyear cooperative agreement with a research institute. OJJDP's objectives for the group, among others, included identifying effective or promising programs, program elements, and implementation principles (i.e., guidelines for developing programs). Objectives also included developing program models to help inform communities of what works in preventing or reducing girls' delinquency, identifying gaps in girls' delinquency research and developing recommendations for future research, and disseminating findings to the girls' delinquency field about effective or promising programs. OJJDP's effort to assess girls' delinquency programs through the use of a study group and the group's methods for assessing studies were consistent with generally accepted social science research practices and standards. In addition, OJJDP's efforts to involve practitioners in Study Group activities and disseminate findings were also consistent with the internal control standard to communicate with external stakeholders, such as practitioners operating programs. The Study Group found that few girls' delinquency programs had been studied and that the available studies lacked conclusive evidence of effective programs; as a result, OJJDP plans to provide technical assistance to help programs be better prepared for evaluations of their effectiveness. However, OJJDP could better address its girls' delinquency goals by more fully developing plans for supporting such evaluations.
GAO-10-133T, Juvenile Justice: Technical Assistance and Better Defined Evaluation Plans Will Help Girls' Delinquency Programs
This is the accessible text file for GAO report number GAO-10-133T
entitled 'Juvenile Justice: Technical Assistance and Better Defined
Evaluation Plans Will Help Girls' Delinquency Program' which was
released on October 20, 2009.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Testimony:
Before the Subcommittee on Crime, Terrorism, and Homeland Security,
Committee on the Judiciary, House of Representatives:
United States Government Accountability Office:
GAO:
For Release on Delivery Expected at 2:30 p.m. EDT:
Tuesday, October 20, 2009:
Juvenile Justice:
Technical Assistance and Better Defined Evaluation Plans Will Help
Girls' Delinquency Programs:
Statement of Eileen R. Larence, Director:
Homeland Security and Justice:
[End of section]
GAO-10-133T:
Mr. Chairman and Members of the Subcommittee:
I am pleased to be here today as you examine issues related to girls'
delinquency--a topic that has attracted the attention of federal,
state, and local policymakers for more than a decade as girls have
increasingly become involved in the juvenile justice system. For
example, from 1995 through 2005, delinquency caseloads for girls in
juvenile justice courts nationwide increased 15 percent while boys'
caseloads decreased by 12 percent.[Footnote 1] More recently, in 2007,
29 percent of juvenile arrests--about 641,000 arrests--involved girls,
who accounted for 17 percent of juvenile violent crime arrests and 35
percent of juvenile property crime arrests.[Footnote 2] Further,
research on girls has highlighted that delinquent girls have higher
rates of mental health problems than delinquent boys, receive fewer
special services, and are more likely to abandon treatment
programs.[Footnote 3]
The Office of Juvenile Justice and Delinquency Prevention (OJJDP) is
the Department of Justice (DOJ) office charged with providing national
leadership, coordination, and resources to prevent and respond to
juvenile delinquency and victimization. OJJDP supports states and
communities in their efforts to develop and implement effective
programs to, among other things, prevent delinquency and intervene
after a juvenile has offended. For example, from fiscal years 2007
through 2009, Congress provided OJJDP almost $1.1 billion to use for
grants to states, localities, and organizations for a variety of
juvenile justice programs, including programs for girls. Also, in
support of this mission, the office funds research and program
evaluations related to a variety of juvenile justice issues.
As programs have been developed at the state and local levels in recent
years that specifically target preventing girls' delinquency or
intervening after girls have become involved in the juvenile justice
system, it is important that agencies providing grants and
practitioners operating the programs have information about which of
these programs are effective. In this way, agencies can help to ensure
that limited federal, state, and local funds are well spent. In
general, effectiveness is determined through program evaluations, which
are systematic studies conducted to assess how well a program is
working--that is, whether a program produced its intended effects. To
help ensure that grant funds are being used effectively, you asked us
to review OJJDP's efforts related to studying and promoting effective
girls' delinquency programs. We issued a report on the results of that
review on July 24, 2009.[Footnote 4] My statement today, as requested,
highlights findings from that report and addresses (1) efforts OJJDP
has made to assess the effectiveness of girls' delinquency programs,
(2) the extent to which these efforts are consistent with generally
accepted social science standards and federal standards to communicate
with stakeholders, and (3) the findings from OJJDP's efforts and how
the office plans to address the findings.
My statement is based on our July report and selected updates made in
October 2009.[Footnote 5] For our report, we reviewed documentation
about OJJDP's establishment of a study group to assess the
effectiveness of girl's delinquency programs, analyzed the groups'
activities and findings, and interviewed OJJDP research and program
officials and the current and former principal investigators of the
study group. Specifically, we reviewed the criteria the study group
used to assess studies of girls' delinquency programs and whether the
group's application of those criteria was consistent with generally
accepted social science standards for evaluation research.[Footnote 6]
We also compared OJJDP's efforts with criteria in Standards for
Internal Control in the Federal Government, specifically that agency
management should ensure that there are adequate means of obtaining
information from and communicating with external stakeholders who may
have a significant impact on the agency achieving its goals, such as
practitioners operating programs or researchers assessing
programs.[Footnote 7] In addition, we conducted interviews with 18
girls' delinquency subject matter experts, that is, researchers and
practitioners, whom we selected on the basis of their knowledge and
experience with girls' delinquency issues.[Footnote 8] While their
comments cannot be generalized to all girls' delinquency experts, we
nonetheless believe that their views gave us useful insights on issues
related to girls' delinquency and OJJDP's efforts to assess girls'
programs. Our work was performed in accordance with generally accepted
government auditing standards. More detail about our scope and
methodology is included in our July report.[Footnote 9]
OJJDP Established the Girls Study Group to Assess the Effectiveness of
Girls' Delinquency Programs:
With an overall goal of developing research that communities need to
make sound decisions about how best to prevent and reduce girls'
delinquency, OJJDP established the Girls Study Group (Study Group) in
2004 under a $2.6 million multiyear cooperative agreement with a
research institute.[Footnote 10] OJJDP's objectives for the group,
among others, included identifying effective or promising programs,
program elements, and implementation principles (i.e., guidelines for
developing programs). Objectives also included developing program
models to help inform communities of what works in preventing or
reducing girls' delinquency, identifying gaps in girls' delinquency
research and developing recommendations for future research, and
disseminating findings to the girls' delinquency field about effective
or promising programs. To meet OJJDP's objectives, among other
activities, the Study Group identified studies of delinquency programs
that specifically targeted girls by reviewing over 1,000 documents in
relevant research areas. These included criminological and feminist
explanations for girls' delinquency, patterns of delinquency, and the
justice system's response to girls' delinquency. As a result, the group
identified 61 programs that specifically targeted preventing or
responding to girls' delinquency. Then, the group assessed the
methodological quality of the studies of the programs that had been
evaluated using a set of criteria developed by DOJ's Office of Justice
Programs (OJP) called What Works to determine whether the studies
provided credible evidence that the programs were effective at
preventing or responding to girls' delinquency.[Footnote 11] The
results of the group's assessment are discussed in the following
sections.
OJJDP Efforts to Assess Program Effectiveness Were Consistent with
Social Science Practices and Standards, and OJJDP Has Taken Action to
Enhance Communication about the Study Group with External Stakeholders:
OJJDP's effort to assess girls' delinquency programs through the use of
a study group and the group's methods for assessing studies were
consistent with generally accepted social science research practices
and standards. In addition, OJJDP's efforts to involve practitioners in
Study Group activities and disseminate findings were also consistent
with the internal control standard to communicate with external
stakeholders, such as practitioners operating programs.[Footnote 12]
According to OJJDP research and program officials, they formed the
Study Group rather than funding individual studies of programs because
study groups provide a cost-effective method of gaining an overview of
the available research in an issue area. As part of its work, the group
collected, reviewed, and analyzed the methodological quality of
research on girls' delinquency programs. The use of such a group,
including its review, is an acceptable approach for systematically
identifying and reviewing research conducted in a field of study. This
review helped consolidate the research and provide information to OJJDP
for determining evaluation priorities. Further, we reviewed the
criteria the group used to assess the studies and found that they
adhere to generally accepted social science standards for evaluation
research. We also generally concurred with the group's assessments of
the programs based on these criteria. According to the group's former
principal investigator, the Study Group decided to use OJP's What Works
criteria to ensure that its assessment of program effectiveness would
be based on highly rigorous evaluation standards, thus eliminating the
potential that a program that may do harm would be endorsed by the
group. However, 8 of the 18 experts we interviewed said that the
criteria created an unrealistically high standard, which caused the
group to overlook potentially promising programs. OJJDP officials
stated that despite such concerns, they approved the group's use of the
criteria because of the methodological rigor of the framework and their
goal for the group to identify effective programs.
In accordance with the internal control standard to communicate with
external stakeholders, OJJDP sought to ensure a range of stakeholder
perspectives related to girls' delinquency by requiring that Study
Group members possess knowledge and experience with girls' delinquency
and demonstrate expertise in relevant social science disciplines. The
initial Study Group, which was convened by the research institute and
approved by OJJDP, included 12 academic researchers and 1 practitioner;
someone with experience implementing girls' delinquency programs.
However, 11 of the 18 experts we interviewed stated that this
composition was imbalanced in favor of academic researchers. In
addition, 6 of the 11 said that the composition led the group to focus
its efforts on researching theories of girls' delinquency rather than
gathering and disseminating actionable information for
practitioners.[Footnote 13] According to OJJDP research and program
officials, they acted to address this issue by adding a second
practitioner as a member and involving two other practitioners in study
group activities. OJJDP officials stated that they plan to more fully
involve practitioners from the beginning when they organize study
groups in the future and to include practitioners in the remaining
activities of the Study Group, such as presenting successful girls'
delinquency program practices at a national conference. Also, in
accordance with the internal control standard, OJJDP and the Study
Group have disseminated findings to the research community,
practitioners in the girls' delinquency field, and the public through
conference presentations, Web site postings, and published bulletins.
The group plans to issue a final report on all of its activities by
spring 2010.
The Study Group Found No Evidence of Effective Girls' Delinquency
Programs; in Response OJJDP Plans to Assist Programs in Preparing for
Evaluations but Could Strengthen Its Plans for Supporting Such
Evaluations:
The Study Group found that few girls' delinquency programs had been
studied and that the available studies lacked conclusive evidence of
effective programs; as a result, OJJDP plans to provide technical
assistance to help programs be better prepared for evaluations of their
effectiveness. However, OJJDP could better address its girls'
delinquency goals by more fully developing plans for supporting such
evaluations.
In its review, the Study Group found that the majority of the girls'
delinquency programs it identified--44 of the 61--had not been studied
by researchers. For the 17 programs that had been studied, the Study
Group reported that none of the studies provided conclusive evidence
with which to determine whether the programs were effective at
preventing or reducing girls' delinquency. For example, according to
the Study Group, the studies provided insufficient evidence of the
effectiveness of 11 of the 17 programs because, for instance, the
studies involved research designs that could not demonstrate whether
any positive outcomes, such as reduced delinquency, were due to program
participation rather than other factors. Based on the results of this
review, the Study Group reported that among other things, there is a
need for additional, methodologically rigorous evaluations of girls'
delinquency programs; training and technical assistance to help
programs prepare for evaluations; and funding to support girls'
delinquency programs found to be promising.
According to OJJDP officials, in response to the Study Group's finding
about the need to better prepare programs for evaluation, the office
plans to work with the group and use the remaining funding from the
effort--approximately $300,000--to provide a technical assistance
workshop by the end of October 2009. The workshop is intended to help
approximately 10 girls' delinquency programs prepare for evaluation by
providing information about how evaluations are designed and conducted
and how to collect data that will be useful for program evaluators in
assessing outcomes, among other things. In addition, OJJDP officials
stated that as a result of the Study Group's findings, along with
feedback they received from members of the girls' delinquency field,
OJJDP plans to issue a solicitation in fiscal year 2010 for funding to
support evaluations of girls' delinquency programs.
OJJDP has also reported that the Study Group's findings are to provide
a foundation for moving ahead on a comprehensive program related to
girls' delinquency. However, OJJDP has not developed a plan that is
documented, is shared with key stakeholders, and includes specific
funding requirements and commitments and time frames for meeting its
girls' delinquency goals. Standard practices for program and project
management state that specific desired outcomes or results should be
conceptualized, defined, and documented in the planning process as part
of a road map, along with the appropriate projects needed to achieve
those results, supporting resources, and milestones.[Footnote 14] In
addition, government internal control standards call for policies and
procedures that establish adequate communication with stakeholders as
essential for achieving desired program goals.[Footnote 15] According
to OJJDP officials, they have not developed a plan for meeting their
girls' delinquency goals because the office is in transition and is in
the process of developing a plan for its juvenile justice
programs,[Footnote 16] but the office is taking steps to address its
girls' delinquency goals, for example, through the technical assistance
workshop. Developing a plan for girls' delinquency would help OJJDP to
demonstrate leadership to the girls' delinquency field by clearly
articulating the actions it intends to take to meet its goals and would
also help the office to ensure that the goals are met.
In our July report, we recommended that to help ensure that OJJDP meets
its goals to identify effective or promising girls' delinquency
programs and supports the development of program models, the
Administrator of OJJDP develop and document a plan that (1) articulates
how the office intends to respond to the findings of the Study Group,
(2) includes time frames and specific funding requirements and
commitments, and (3) is shared with key stakeholders. OJP agreed with
our recommendation and outlined efforts that OJJDP plans to undertake
in response to these findings. For example, OJJDP stated that it
anticipates publishing its proposed juvenile justice program plan,
which is to include how it plans to address girls' delinquency issues,
in the Federal Register to solicit public feedback and comments, which
will enable the office to publish a final plan in the Federal Register
by the end of the year (December 31, 2009).
Mr. Chairman, this concludes my statement. I would be pleased to
respond to any questions that you or other Members of the Subcommittee
may have.
Contacts and Acknowledgements:
For questions about this statement, please contact Eileen R. Larence at
(202) 512-8777 or larencee@gao.gov. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last
page of this statement. Individuals making key contributions to this
statement include Mary Catherine Hult, Assistant Director; Kevin
Copping; and Katherine Davis. Additionally, key contributors to our
July 2009 report include David Alexander, Elizabeth Blair, and Janet
Temko.
[End of section]
Footnotes:
[1] C. Puzzanchera and W. Kang, Juvenile Court Statistics Databook
(2008), [hyperlink, http://ojjdp.ncjrs.gov/ojstatbb/jcsdb/] (accessed
Oct.15, 2009). This Web site provides the most current data available.
[2] C. Puzzanchera, Juvenile Arrests 2007, (2009)
www.ncjrs.gov/pdffiles1/ ojjdp/225344.pdf (accessed Oct.15, 2009).
[3] Elizabeth Cauffman and others, "Gender Differences in Mental Health
Symptoms among Delinquent and Community Youth," Youth Violence and
Juvenile Justice, vol. 5, no. 3 (2007): 287-307. Elizabeth Caufmann
"Understanding the Female Offender" The Future of Children, vol. 18,
no. 2 (2008): 119-142.
[4] GAO, Juvenile Justice: Technical Assistance and Better Defined
Evaluation Plans Will Help to Improve Girls' Delinquency Programs,
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-09-721R] (Washington,
D.C.: July 24, 2009).
[5] In a September 18, 2009, letter regarding the recommendation we
made in our July report, DOJ clarified actions it was taking to address
our recommendation, which we have included in this statement.
[6] For social science standards for evaluation research, see Donald T.
Campbell and Julian Stanley, Experimental and Quasi-Experimental
Designs for Research (Chicago: Rand McNally, 1963); William R. Shadish,
Thomas D. Cook, and Donald T. Campbell, Experimental and Quasi-
Experimental Designs for Generalized Causal Inference (Boston: Houghton
Mifflin, 2002); Carol H. Weiss, Evaluation: Methods for Studying
Programs and Policies, Second Edition (Englewood Cliffs, N.J.: Prentice-
Hall, Inc., 1998); and GAO, Designing Evaluations, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO/PEMD-10.1.4] (Washington, D.C.:
May 1991).
[7] GAO, Standards for Internal Control in the Federal Government,
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?/GAO/AIMD-00-21.3.1]
(Washington, D.C.: November 1999).
[8] GAO defines an expert as a person who is recognized by others who
work in the same subject matter area as having knowledge that is
greater in scope or depth than that of most people working in that
area. The expert's knowledge can come from education, experience, or
both. We identified researchers who focus on girls' delinquency issues
and practitioners who operate programs that address girls' delinquency.
Specifically, these 18 experts included 11 of the 15 study group
members and 7 experts who were not members of the group. While we
contacted all 15 of the study group members, 4 members either did not
respond to requests for interviews or declined to be interviewed.
[9] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-09-721R].
[10] Cooperative agreements, rather than grant awards, can be used by
federal agencies when substantial involvement is expected between the
agency and the recipient when carrying out the activities described in
the program announcement. OJJDP extended the cooperative agreement with
the research institute through June 2010 to complete all of the Study
Group activities.
[11] The What Works criteria define six levels of effectiveness,
including effective, promising, and ineffective, for use in assessing
and classifying studies on the basis of their evidence of
effectiveness. The criteria for an effective program include a
randomized controlled research design--a design that compares the
outcomes for individuals who are randomly assigned to either the
program being studied or to a nonparticipating control group before the
intervention. While other research designs can produce valid results,
we have previously reported that when it is feasible and ethical to do
so, randomized controlled designs provide researchers with the best
method for assessing a program's effectiveness because they isolate
changes caused by the program from other factors.
[12] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-00-21.3.1].
[13] The other seven experts did not express views regarding the
balance of the study group's composition.
[14] Project Management Institute, The Standard for Program Management.
[15] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-00-21.3.1].
[16] OJJDP is required under the Juvenile Justice and Delinquency
Prevention Act to publish an annual program plan that will, among other
things, lay out goals and criteria for conducting research and
evaluation for its juvenile justice programs. 42 U.S.C. § 5614(b)(5).
This plan is required to be published annually in the Federal Register
for public comment, and is to describe the activities the Administrator
intends to carry out under Parts D and E. Under Part D, OJJDP is
authorized to conduct research, evaluation, and technical assistance,
among other things. 42 U.S.C. §§ 5661-62. Under Part E, OJJDP is
authorized to make grants for developing, testing and demonstrating
promising new initiatives and programs. 42 U.S.C. §§ 5665-66.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
[End of section]