Juvenile Justice
Technical Assistance and Better Defined Evaluation Plans Will Help to Improve Girls' Delinquency Programs Gao ID: GAO-09-721R July 24, 2009Girls' delinquency has attracted the attention of federal, state, and local policymakers for more than a decade as girls have increasingly become involved in the juvenile justice system. For example, from 1995 through 2005, delinquency caseloads for girls in juvenile justice courts nationwide increased 15 percent while boys' caseloads decreased by 12 percent. Also, from 1995 through 2005, the number of girls' cases nationwide involving detention increased 49 percent compared to a 7 percent increase for boys. More recently, in 2007, 29 percent of juvenile arrests--about 641,000 arrests--involved girls, who accounted for 17 percent of juvenile violent crime arrests and 35 percent of juvenile property crime arrests. Further, in a 2007 survey of states conducted by the Federal Advisory Committee on Juvenile Justice, 22 states listed girls' delinquency as an issue affecting their states' juvenile justice systems. State justice officials responding to the survey noted that juvenile female offenses have increased sharply and also noted that juvenile female offenders generally had more serious and wide-ranging service needs than juvenile male offenders, including treatment for substance abuse and mental health conditions. As programs have been developed at the state and local levels in recent years that specifically target preventing girls' delinquency or intervening after girls have become involved in the juvenile justice system, it is important that agencies providing grants and practitioners operating the programs have information about which of these programs are effective. In this way, agencies can help to ensure that limited federal, state, and local funds are well spent. In general, effectiveness is determined through program evaluations, which are systematic studies conducted to assess how well a program is working--that is, whether a program produced its intended effects. To help ensure that grant funds are being used effectively, Congress asked us to review OJJDP's efforts related to studying and promoting effective girls' delinquency programs. This report addresses the following questions: 1. What efforts, if any, has OJJDP made to assess the effectiveness of girls' delinquency programs? 2. To what extent are OJJDP's efforts to assess girls' delinquency programs consistent with generally accepted social science standards and the internal control standard to communicate with external stakeholders? 3. What are the findings from OJJDP's efforts to assess the effectiveness of girls' delinquency programs, and how, if at all, does OJJDP plan to address the findings from these efforts?
To assess the effectiveness of girls' delinquency programs, OJJDP established the Girls Study Group (Study Group). With an overall goal of developing research that communities need to make sound decisions about how best to prevent and reduce girls' delinquency, the Study Group was established in 2004 under a $2.6 million multiyear cooperative agreement with a research institute. OJJDP's objectives for the group, among others, included identifying effective or promising programs, program elements, and implementation principles (i.e., guidelines for developing programs) and developing program models to help inform communities of what works in preventing or reducing girls' delinquency; identifying gaps in girls' delinquency research and developing recommendations for future research; and disseminating findings to the girls' delinquency field about effective or promising programs. To meet OJJDP's objectives, among other things, the Study Group identified studies of delinquency programs that specifically targeted girls. The group then assessed the methodological quality of the studies using a set of criteria developed by DOJ's Office of Justice Programs (OJP) called What Works to determine whether the studies provided credible evidence that the programs were effective at preventing or responding to girls' delinquency. OJJDP's effort to assess girls' delinquency programs through the use of a study group and the group's methods for assessing studies were consistent with generally accepted social science research practices and standards, and OJJDP's efforts to involve practitioners in Study Group activities and disseminate findings were also consistent with the internal control standard to communicate with external stakeholders.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-09-721R, Juvenile Justice: Technical Assistance and Better Defined Evaluation Plans Will Help to Improve Girls' Delinquency Programs
This is the accessible text file for GAO report number GAO-09-721R
entitled 'Juvenile Justice: Technical Assistance and Better Defined
Evaluation Plans Will Help to Improve Girls' Delinquency Programs'
which was released on July 24, 2009.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
GAO-09-721R:
United States Government Accountability Office:
Washington, DC 20548:
July 24, 2009:
The Honorable Robert C. Scott:
Chairman:
Subcommittee on Crime, Terrorism, and Homeland Security:
Committee on the Judiciary:
House of Representatives:
Subject: Juvenile Justice: Technical Assistance and Better Defined
Evaluation Plans Will Help to Improve Girls' Delinquency Programs:
Dear Mr. Chairman:
Girls' delinquency has attracted the attention of federal, state, and
local policymakers for more than a decade as girls have increasingly
become involved in the juvenile justice system. For example, from 1995
through 2005, delinquency caseloads for girls in juvenile justice
courts nationwide increased 15 percent while boys' caseloads decreased
by 12 percent. Also, from 1995 through 2005, the number of girls' cases
nationwide involving detention increased 49 percent compared to a 7
percent increase for boys.[Footnote 1] More recently, in 2007, 29
percent of juvenile arrests--about 641,000 arrests--involved girls, who
accounted for 17 percent of juvenile violent crime arrests and 35
percent of juvenile property crime arrests.[Footnote 2] Further, in a
2007 survey of states conducted by the Federal Advisory Committee on
Juvenile Justice, 22 states listed girls' delinquency as an issue
affecting their states' juvenile justice systems.[Footnote 3] State
justice officials responding to the survey noted that juvenile female
offenses have increased sharply and also noted that juvenile female
offenders generally had more serious and wide-ranging service needs
than juvenile male offenders, including treatment for substance abuse
and mental health conditions.
The Office of Juvenile Justice and Delinquency Prevention (OJJDP) is
the Department of Justice (DOJ) office charged with providing national
leadership, coordination, and resources to prevent and respond to
juvenile delinquency and victimization. OJJDP supports states and
communities in their efforts to develop and implement effective
programs to, among other things, prevent delinquency and intervene
after a juvenile has offended. For example, from fiscal years 2007
through 2009, Congress provided OJJDP almost $1.1 billion for grants to
states, localities, and organizations for a variety of juvenile justice
programs. In support of this mission, the office also funds research
and program evaluations related to a variety of juvenile justice
issues, including girls' delinquency.
As programs have been developed at the state and local levels in recent
years that specifically target preventing girls' delinquency or
intervening after girls have become involved in the juvenile justice
system, it is important that agencies providing grants and
practitioners operating the programs have information about which of
these programs are effective. In this way, agencies can help to ensure
that limited federal, state, and local funds are well spent. In
general, effectiveness is determined through program evaluations, which
are systematic studies conducted to assess how well a program is
working--that is, whether a program produced its intended effects. To
help ensure that grant funds are being used effectively, you asked us
to review OJJDP's efforts related to studying and promoting effective
girls' delinquency programs. This report addresses the following
questions:
1. What efforts, if any, has OJJDP made to assess the effectiveness of
girls' delinquency programs?
2. To what extent are OJJDP's efforts to assess girls' delinquency
programs consistent with generally accepted social science standards
and the internal control standard to communicate with external
stakeholders?
3. What are the findings from OJJDP's efforts to assess the
effectiveness of girls' delinquency programs, and how, if at all, does
OJJDP plan to address the findings from these efforts?
To identify OJJDP's efforts to assess the effectiveness of girls'
delinquency programs, we analyzed relevant laws related to the office's
role in supporting research and evaluations on delinquency programs. We
also analyzed OJJDP budget data for fiscal years 2007 through 2009. We
chose these years because they provide the most recent overview of the
funding the office has had available to support its evaluation
activities. We examined reports from research organizations and
academic journal articles on girls' delinquency issues. In our review,
we focused on OJJDP's efforts related to programs that are specifically
designed for girls, not programs designed for both girls and boys. To
identify OJJDP's efforts, we reviewed a list of its grants to fund
studies of girls' delinquency programs from 1998 to 2008. We chose this
time frame, the past 10 years from the start of our work, because it
provided us with an overview of OJJDP's efforts related to assessing
girls' delinquency programs. We also analyzed documentation about
OJJDP's establishment of a study group on girls' delinquency issues,
including the program announcement and cooperative agreement.[Footnote
4] We interviewed OJJDP officials, including the research coordinator
who managed the study group project, about the office's role in
overseeing the group's research. We also interviewed the current and
former principal investigators of the study group project regarding the
formation of the group, its activities, and its methodologies. To
gather information on OJJDP's efforts, we conducted interviews with 18
girls' delinquency subject matter experts, that is, researchers and
practitioners. We selected these experts based on their knowledge and
experience with girls' delinquency issues, which we determined through
our review of the literature and from suggestions of experts to
interview from study group members and OJJDP.[Footnote 5] These 18
experts included 11 of the 15 study group members and 7 experts who
were not members of the group.[Footnote 6] While their comments cannot
be generalized to all girls' delinquency experts, we nonetheless
believe that their views gave us useful insights on issues related to
girls' delinquency and OJJDP's efforts to assess girls' programs.
To determine the extent to which OJJDP's efforts to assess girls'
delinquency programs were consistent with generally accepted social
science standards, we reviewed the criteria the study group used to
assess studies of girls' delinquency programs and whether the group's
application of those criteria was consistent with standards for
evaluation research.[Footnote 7] To determine the extent to which these
OJJDP efforts were consistent with the internal control standard to
communicate with external stakeholders, we compared the office's
efforts with criteria in Standards for Internal Control in the Federal
Government, specifically that agency management should ensure that
there are adequate means of obtaining information from and
communicating with external stakeholders who may have a significant
impact on the agency achieving its goals.[Footnote 8] We reviewed
documentation about the composition of the study group and the criteria
used to select the group members, such as their relevant fields of
expertise, knowledge, and experience with girls' issues. We also
examined the study group's external communications efforts, including
its Web site, findings bulletins, conference presentations, academic
journal articles, and published book. In addition, we interviewed OJJDP
officials about these dissemination efforts, as well as 18 girls'
delinquency experts regarding their views on the composition of the
study group.
To determine the findings from OJJDP's efforts to assess the
effectiveness of girls' delinquency programs, and to assess how, if at
all, OJJDP plans to address these findings, we analyzed documentation
such as published bulletins and conference presentations about the
study group's findings and recommendations related to program
effectiveness. We also interviewed OJJDP officials knowledgeable about
the office's planning efforts and the current and former study group
principal investigators regarding the group's findings and
recommendations. We compared OJJDP's stated plans with criteria in
standard practices for program management.[Footnote 9]
We conducted this performance audit from July 2008 through July 2009 in
accordance with generally accepted government auditing standards. Those
standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.
Results in Brief:
To assess the effectiveness of girls' delinquency programs, OJJDP
established the Girls Study Group (Study Group). With an overall goal
of developing research that communities need to make sound decisions
about how best to prevent and reduce girls' delinquency, the Study
Group was established in 2004 under a $2.6 million multiyear
cooperative agreement with a research institute. OJJDP's objectives for
the group, among others, included identifying effective or promising
programs, program elements, and implementation principles (i.e.,
guidelines for developing programs) and developing program models to
help inform communities of what works in preventing or reducing girls'
delinquency; identifying gaps in girls' delinquency research and
developing recommendations for future research; and disseminating
findings to the girls' delinquency field about effective or promising
programs. To meet OJJDP's objectives, among other things, the Study
Group identified studies of delinquency programs that specifically
targeted girls. The group then assessed the methodological quality of
the studies using a set of criteria developed by DOJ's Office of
Justice Programs (OJP) called What Works to determine whether the
studies provided credible evidence that the programs were effective at
preventing or responding to girls' delinquency.[Footnote 10]
OJJDP's effort to assess girls' delinquency programs through the use of
a study group and the group's methods for assessing studies were
consistent with generally accepted social science research practices
and standards, and OJJDP's efforts to involve practitioners in Study
Group activities and disseminate findings were also consistent with the
internal control standard to communicate with external stakeholders.
[Footnote 11]
* According to OJJDP officials--including the research coordinator--
they formed the Study Group rather than funding individual studies of
programs because study groups provide a cost-effective method of
gaining an overview of the available research in an issue area. As part
of its work, the group collected, reviewed, and analyzed the
methodological quality of research on girls' delinquency programs. The
use of such a group, including its review, is an acceptable approach
for systematically identifying and reviewing research conducted in a
field of study. This review helped consolidate the research and provide
information to OJJDP for determining evaluation priorities. Further, we
reviewed the criteria the group used to assess the studies and found
that they adhere to generally accepted social science standards for
evaluation research. We also generally concurred with the group's
assessments of the programs based on these criteria. According to the
group's former principal investigator, the Study Group decided to use
the What Works criteria to ensure that its assessment of program
effectiveness would be based on highly rigorous evaluation standards,
thus eliminating the potential that a program that may do harm would be
endorsed by the group. However, 8 of the 18 experts we interviewed said
that the criteria created an unrealistically high standard, which
caused the group to overlook potentially promising programs. OJJDP
officials stated that despite such concerns, they approved the group's
use of the criteria because of the methodological rigor of the
framework and their goal for the group to identify effective programs.
* In accordance with the internal control standard to communicate with
external stakeholders, OJJDP sought to ensure a range of stakeholder
perspectives related to girls' delinquency by requiring that Study
Group members possess knowledge and experience with girls' delinquency
and demonstrate expertise in relevant social science disciplines. The
initial Study Group, which was convened by the research institute and
approved by OJJDP, included 12 academic researchers and 1 practitioner,
a member with experience implementing girls' delinquency programs.
Eleven of the 18 experts we interviewed stated that this composition
was imbalanced in favor of academic researchers, six of whom said that
the composition led the group to focus its efforts on researching
theories of girls' delinquency rather than gathering and disseminating
actionable information for practitioners.[Footnote 12] According to
OJJDP officials, they acted to address this issue by adding a second
practitioner as a member and involving two other practitioners in study
group activities. OJJDP officials stated that they plan to more fully
involve practitioners from the beginning when they organize study
groups in the future and to include practitioners in the remaining
activities of the Study Group, such as presenting findings at a
national conference. Also, in accordance with the internal control
standard, OJJDP and the Study Group have disseminated findings to the
research community, practitioners in the girls' delinquency field, and
the public through conference presentations, Web site postings, and
published bulletins and plan to report on all of the group's activities
by spring 2010.
To address the Study Group findings that few girls' delinquency
programs had been studied and that the available studies lacked
conclusive evidence of program effectiveness, OJJDP plans to provide
technical assistance to help programs be better prepared for
evaluation; however, more fully developing plans for supporting
evaluations could help OJJDP address its girls' delinquency goals. The
Study Group found that the majority of the girls' delinquency programs
it identified--44 of the 61--had not been studied by researchers. For
the 17 programs that had been studied, the Study Group reported that
none of the studies provided conclusive evidence with which to
determine whether the programs were effective at preventing or reducing
girls' delinquency. For example, according to the Study Group, 11 of
the 17 studies lacked evidence of program effectiveness because, for
instance, the studies involved research designs that could not
demonstrate whether any positive outcomes, such as reduced delinquency,
were due to program participation rather than other factors. Based on
the results of this review, the Study Group reported that among other
things, there is a need for additional, methodologically rigorous
evaluations of girls' delinquency programs; training and technical
assistance to help programs prepare for evaluations; and funding to
support girls' delinquency programs found to be promising. According to
OJJDP officials, in response to the Study Group's finding about the
need to better prepare programs for evaluation, the office plans to
work with the group and use the remaining funding from the effort--
approximately $300,000--to provide technical assistance workshop in
October 2009. The workshop is intended to help approximately 10 girls'
delinquency programs prepare for evaluation by providing information
about how evaluations are designed and conducted, how to identify
appropriate performance measures, and how to collect data that will be
useful for program evaluators in assessing outcomes. In addition, OJJDP
officials stated that as a result of the Study Group's findings along
with feedback they received from members of the girls' delinquency
field, OJJDP plans to issue a solicitation in early fiscal year 2010
for researchers to apply for funding to conduct evaluations of two to
five girls' delinquency programs. OJJDP has also reported that the
Study Group's findings are to provide a foundation for moving ahead on
a comprehensive program related to girls' delinquency. However, OJJDP
has not developed a plan that is documented, is shared with key
stakeholders, and includes specific funding requirements and
commitments and time frames for meeting its girls' delinquency goals.
Standard practices for program and project management state that
specific desired outcomes or results should be conceptualized, defined,
and documented in the planning process as part of a road map, along
with the appropriate projects needed to achieve those results,
supporting resources, and milestones.[Footnote 13] In addition,
government internal control standards call for policies and procedures
that establish adequate communication with stakeholders as essential
for achieving desired program goals.[Footnote 14] According to OJJDP
officials, they have not developed such a plan because the office is in
transition and is in the process of initiating efforts to develop an
officewide research plan, but they are taking steps to address their
girls' delinquency goals, for example, through the workshop and planned
evaluations. Developing such a plan would help OJJDP to demonstrate
leadership to the girls' delinquency field by clearly articulating the
actions it intends to take to meet its goals and would also help the
office to ensure that the goals are met.
To help ensure that OJJDP meets its goals to identify effective or
promising girls' delinquency programs and supports the development of
program models, we are recommending that the Administrator of OJJDP
develop and document a plan that (1) articulates how the office intends
to respond to the findings of the Study Group, (2) includes time frames
and specific funding requirements and commitments, and (3) is shared
with key stakeholders. In commenting on a draft of this report, OJP
agreed with our recommendation and outlined efforts that OJJDP plans to
undertake to respond to the findings of the Study Group, which we
describe in the report. OJP comments are reprinted in the enclosure.
Background:
Over the past two decades girls have increasingly become involved in
the juvenile justice system, and while the majority of juvenile arrests
and cases involve boys, research has indicated that girls have more
intensive treatment needs than boys. In 1980, 20 percent of all
juvenile arrests were girls; by the mid-1990s about one quarter of
these arrests were girls; and by 2007, girls accounted for 29 percent
of all juvenile arrests. Additionally, while arrests for some violent
crimes, such as assaults, have decreased for males, they have decreased
less, or in some cases have increased, for females. For example,
between 1998 and 2007 juvenile male arrests for simple assault declined
4 percent, and female arrests increased 10 percent.[Footnote 15]
Further, from 1985 through 2005, the estimated number of girls'
delinquency cases involving detention increased by 92 percent, and
those cases that involved probation increased by 88 percent. Research
on girls has highlighted that delinquent girls have higher rates of
mental health problems than delinquent boys, receive fewer special
services, and are more likely to abandon treatment programs. For
example, one study showed that detained girls have more symptoms of
mental illness than would be predicted on the basis of gender or
setting alone.[Footnote 16] Research has also shown that delinquent
girls have higher mortality rates, dysfunctional and violent
relationships, poor educational achievement, and less stable work
histories than nondelinquent girls. Further, girls' delinquency has
been linked to drug abuse, mental health problems and disorders, poorer
physical health, and victimization by and violence toward partners in
adulthood.
In recent years, programs have been developed that specifically target
preventing girls' delinquency and intervening once girls have become
involved in the juvenile justice system. In general, prevention
programs provide services and programming, such as substance abuse
education, mentoring, and life skills education, to deter girls from
becoming involved in criminal or other antisocial activities.
Intervention programs provide services to girls once they have entered
the juvenile justice system, for example, through programs that are
alternatives to probation or that provide intensive services for girls
who are on probation, to prevent them from returning to the system or
entering the adult criminal justice system. These services could
include visits by probation officers, individual case plans, substance
abuse treatment and therapy, funds for emergency situations, life
skills courses, teen pregnancy services, and therapy sessions.
The Juvenile Justice and Delinquency Prevention Act (the Act)
established OJJDP in 1974.[Footnote 17] As the only federal office
charged exclusively with preventing and responding to juvenile
delinquency and victimization and with helping states improve their
juvenile justice systems, OJJDP supports its mission through a variety
of activities, including: funding research and evaluation efforts,
statistical studies, and demonstration programs; providing training and
technical assistance; producing and distributing publications and other
products containing information about juvenile justice topics; and
administering a wide variety of grants to states, territories,
localities, and public and private organizations through formula,
block, and discretionary grant programs.[Footnote 18] Table 1 shows
OJJDP's enacted appropriations for fiscal years 2007 through 2009.
Table 1: Juvenile Justice Appropriations Fiscal Years 2007 through 2009
(Dollars in thousands):
Line item: Part A - Concentration of Federal Efforts[A];
Funding by fiscal year 2007: $703;
Funding by fiscal year 2008: $658;
Funding by fiscal year 2009: $0.
Line item: Part B - State Formula Grants;
Funding by fiscal year 2007: $78,976;
Funding by fiscal year 2008: $74,260;
Funding by fiscal year 2009: $75,000.
Line item: Part D--Research, Evaluation, Technical Assistance and
Training;
Funding by fiscal year 2007: $0;
Funding by fiscal year 2008: $0;
Funding by fiscal year 2009: $0.
Line item: Part E - Developing, Testing and Demonstrating Promising New
Initiatives and Projects;
Funding by fiscal year 2007: $104,670;
Funding by fiscal year 2008: $93,835;
Funding by fiscal year 2009: $82,000.
Line item: Youth Mentoring Grants;
Funding by fiscal year 2007: $9,872;
Funding by fiscal year 2008: $70,000;
Funding by fiscal year 2009: $80,000.
Line item: Title V - Local Delinquency Prevention Incentive Grants;
Funding by fiscal year 2007: $64,168;
Funding by fiscal year 2008: $61,100;
Funding by fiscal year 2009: $62,000.
Line item: Project Childsafe[B];
Funding by fiscal year 2007: $987;
Funding by fiscal year 2008: $0;
Funding by fiscal year 2009: $0.
Line item: Secure Our Schools;
Funding by fiscal year 2007: $14,808;
Funding by fiscal year 2008: $15,040;
Funding by fiscal year 2009: $0.
Line item: VOCA--Improving Investigation and Prosecution of Child Abuse
Program;
Funding by fiscal year 2007: $14,808;
Funding by fiscal year 2008: $16,920;
Funding by fiscal year 2009: $20,000.
Line item: Juvenile Accountability Block Grant Program[C];
Funding by fiscal year 2007: $49,360;
Funding by fiscal year 2008: $51,700;
Funding by fiscal year 2009: $55,000.
Line item: Total;
Funding by fiscal year 2007: $338,352;
Funding by fiscal year 2008: $383,513;
Funding by fiscal year 2009: $374,000.
Sources: Revised Continuing Appropriations Resolution, 2007, Pub. L.
No. 110-5, 121 Stat. 8, 8-9 (including the across-the-board rescission
of 1.28 percent provided in the continuing resolution); Consolidated
Appropriations Act, 2008, Pub. L. No. 110-161, 121 Stat. 1844, 1911-12
(2007); and Omnibus Appropriations Act, 2009, Pub. L. No. 111-8, 123
Stat. 524, 581-82.
[A] According to OJP's fiscal year 2010 congressional budget
submission, the Concentration of Federal Efforts program promotes
interagency cooperation and coordination among federal agencies with
responsibilities in the area of juvenile justice, as authorized by Part
A of the Act, as amended.
[B] Project Childsafe is a nationwide program to promote safe firearms
handling and storage practices through the distribution of safety
education messages and free gun-locking devices.
[C] Under the Juvenile Accountability Block Grant Program, OJJDP
provides funds to states and units of local government for the purpose
of strengthening the juvenile justice system. These funds can be used
for 17 different purpose areas, including establishing programs to help
the successful reentry of juvenile offenders from state and local
custody in the community or for hiring or training programs for
detention and corrections personnel.
[End of table]
OJJDP, through its various grant programs, has provided funding to
states and organizations to support girls' delinquency programs,
although it is not specifically required by the Act to fund such
programs in particular. For example, to be eligible to receive formula
grants, states are required to submit a plan to OJJDP for providing
gender-specific services for juvenile delinquency prevention and
treatment.[Footnote 19] However, the states generally have the
authority to determine how formula and block grants are allocated and
may use these funds to support a range of program areas, including
programs specifically for delinquent girls. For example, for fiscal
years 2007 and 2008, OJJDP reported that states used approximately $1.9
million in Part B formula grant money for girls' delinquency programs,
representing approximately 1 percent of such funding for those years.
In addition, in fiscal year 2007, OJJDP reported awarding about $1.8
million in discretionary grant awards to prevention and intervention
programs addressing girls' delinquency.[Footnote 20]
The Act requires the OJJDP Administrator to conduct and support
evaluations and studies of the performance and results achieved by
federal juvenile delinquency programs and activities, although the law
does not specifically require OJJDP to fund evaluations of state or
locally funded programs or those specifically focused on girls'
delinquency.[Footnote 21] OJJDP has provided funding for evaluations
using (1) funds appropriated for Part D of the Act--which allows the
Administrator to conduct research and evaluation, information
dissemination, and training and technical assistance,[Footnote 22] or
(2) funds set aside from several of its appropriations accounts for use
for research, evaluation, and statistics activities.[Footnote 23]
Funding has not been appropriated to OJJDP for Part D since fiscal year
2005 when it received $10 million, so OJJDP has allocated funding for
research and evaluation of programs from fiscal years 2006 through 2008
using approximately $40 million in funding from appropriation set
asides.[Footnote 24]
OJJDP has provided funding for several efforts designed to provide
information about girls' delinquency programs to the juvenile justice
field in the past decade. For example, in 1998, the office published an
inventory of best practices that included a list of 16 promising girls'
delinquency programs, which had been compiled by a research
organization as part of a $1.1 million cooperative agreement to provide
training and technical assistance to states and localities about girls'
programs.[Footnote 25] The research organization identified these 16
programs on the basis of programmatic criteria--such as whether the
program used appropriate assessments to determine treatment plans;
provided empowerment strategies, such as skill training and vocational
training; or provided its staff with gender-specific training--rather
than on whether the program's effectiveness had been studied by
researchers. Further, this effort found that more research was needed
to draw conclusions about the effectiveness of girls' delinquency
program models. In addition, during this time OJJDP spent approximately
$1.1 million to fund four studies of girls' delinquency issues. While
these studies assessed issues related to girls' delinquency, they did
not specifically assess the effectiveness of girls' delinquency
programs. For example, in 2000, OJJDP funded one study of women in
gangs, which found, among other things, that the optimum time for
prevention and intervention was the middle teen years and that the
optimum place for intervention was school before girls drop out.
Another study compared three treatment models to determine which was
most effective at reducing the number of institutional placements for
adjudicated female offenders. The study found that girls with the most
serious and frequent crises were more dissatisfied with social services
or were denied access to such services. The study highlighted the
importance of youth assistance programs to provide opportunities for
girls to develop pro-social skills through family, school, and
community connections.
OJJDP Established the Girls Study Group to Assess the Effectiveness of
Girls' Delinquency Programs:
OJJDP initiated the Study Group to assess the effectiveness of girls'
delinquency programs. In response to increases in girls' arrests
through the 1990s and early 2000s and questions about the causes of
these increases and how best to respond to the needs of girls entering
the juvenile justice system, OJJDP issued a program announcement in
2003 for a study group to focus on girls' delinquency issues.[Footnote
26] While OJJDP had funded studies on girls' issues and a technical
assistance effort to assist girls' delinquency programs in their
operations, in forming the Study Group, OJJDP determined that a
comprehensive, research-based foundation was needed to guide state and
local policymakers and practitioners in their efforts to effectively
prevent and reduce girls' delinquency. In its announcement for the
Study Group, OJJDP highlighted the need for more information about
female development and female-specific delinquency risk factors, as
well as the effectiveness of girls' delinquency programs to ensure the
best services and treatment. OJJDP sought applications from public and
private organizations to convene a study group to address these issues
and in 2004 awarded a 2-year cooperative agreement to Research Triangle
Institute (RTI) to do so. OJJDP has since provided RTI with an
extension through June 2010 to complete all of the Study Group's
activities. The total funding awarded for the cooperative agreement was
almost $2.6 million.
OJJDP articulated five broad objectives for the Study Group in its
September 2003 program announcement. Three of these objectives
specifically related to assessing and promoting girls' delinquency
programs: (1) identifying effective or promising programs, program
elements, and implementation principles to help inform communities
about what works in preventing or reducing girls' delinquency and to
support the development of these program models; (2) identifying gaps
in girls' delinquency research and developing recommendations for
future research to fill these gaps; and (3) disseminating findings to
the girls' delinquency field about effective or promising programs. The
other two objectives included understanding the trends and consequences
related to girls' delinquency and developing a comprehensive theory of
girls' delinquency.[Footnote 27]
To meet OJJDP's program assessment objectives, among other activities,
the Study Group conducted a review of the literature on girls'
delinquency that included over 1,000 documents in relevant research
areas, such as criminological and feminist explanations for girls'
delinquency, patterns of delinquency, and the justice system's response
to girls' delinquency. To identify girls' delinquency programs, from
June 2005 through October 2006, the Study Group analyzed the results of
this literature search, conducted Web searches, reviewed juvenile
justice 3-year plans from 2000 to 2004 for all 50 states, reviewed
federal agency and private organization lists of delinquency programs,
and solicited suggestions on its Web site.[Footnote 28] The Study Group
initially set out to identify federally funded girls' delinquency
programs but expanded its search to include state and locally funded
programs after it found few federally funded programs. As a result, the
Study Group identified 61 programs that specifically targeted
preventing or responding to girls' delinquency. The group then
determined which of these programs had been studied for program
effectiveness by conducting Web searches for evaluation materials and
published research, reviewing abstracts from academic journals,
contacting program directors, and reviewing program Web sites.
To identify effective programs, the Study Group reviewed the studies of
girls' delinquency programs that it identified and classified them
based on evidence of their effectiveness. To make this determination,
the Study Group compared the studies' methodologies to criteria
established in the OJP What Works classification framework, which
defines six levels of evidence of effectiveness, which are effective,
effective with reservation, promising, and ineffective, as well as
inconclusive evidence and insufficient evidence, as described in table
2.[Footnote 29]
Table 2: Summary of What Works Criteria Used by the Girls Study Group
to Assess Studies of Girls' Delinquency Programs:
Level of effectiveness: Effective;
Description: Effective programs have studies with a randomized
controlled research design. These are designs that compare the outcomes
for individuals that are randomly assigned either to the program or to
a nonparticipating control group before the intervention in an effort
to control for any systematic difference between the groups that could
account for a difference in their outcomes. Effective programs also
demonstrate a significant and sustained effect--that is, statistically
significant positive outcomes that remain for at least 1 year after
subjects stop participating in a program. The program should have been
replicated at least once externally at another site to confirm results.
Level of effectiveness: Effective with reservation;
Description: These programs have studies with a randomized controlled
research design that demonstrates a significant and sustained effect. A
program should have at least one replication to confirm results.
Reservations occur either because the program has only an internal
replication at the same site or because it has an external replication
with modest results.
Level of effectiveness: Promising;
Description: Promising programs have either studies with (1) a
randomized controlled research design without a replication or (2) a
quasi-experimental research design. These programs have significant and
sustained effects.
Level of effectiveness: Insufficient evidence;
Description: These are studies of programs that have a quasi-
experimental research design that lack sufficient methodological rigor,
or have a pre-post test design that involves tests that analyze
measures before and after individuals participated in the program.
Level of effectiveness: Inconclusive evidence;
Description: These studies of programs may have adequately rigorous
research designs but not sustained effects, or they may have
contradictory findings and not enough evidence demonstrating that the
programs are either effective or ineffective.
Level of effectiveness: Ineffective;
Description: These are studies of programs that have an experimental or
quasi-experimental research design that failed to demonstrate a
significant effect in an initial study or in a replication.
Source: GAO analysis of OJP What Works criteria.
[End of table]
According to the Study Group's principal investigator, as of May 2009,
the group had finalized its program review findings and was in the
process of finishing a bulletin on these findings before providing it
to OJJDP for publication. As of June 2009, OJJDP has issued three
bulletins on several of the group's activities. These bulletins have
provided an overview of the Study Group's activities and the group's
findings on its two objectives related to girls' delinquency risk
factors and patterns of offending. According to OJJDP officials, the
Study Group plans to issue a final report that summarizes all of its
activities and findings to OJJDP by spring 2010.
OJJDP Efforts to Assess Program Effectiveness Were Consistent with
Social Science Practices and Standards, and OJJDP Has Taken Action to
Enhance Its Communication about Study Group Activities and Findings
with External Stakeholders:
OJJDP's efforts to assess program effectiveness through the use of a
study group as well as the group's efforts were consistent with
generally accepted social science practices and standards, although
experts we interviewed presented differing views on the criteria used
to assess programs. OJJDP also took action to include external
stakeholders in study group activities and is disseminating the group's
findings consistent with standards for control in the federal
government.
The Use of a Study Group and the Group's Efforts Were Consistent with
Generally Accepted Social Science Practices and Standards; However,
Experts We Interviewed Presented Differing Views on the Criteria Used
to Assess Programs:
OJJDP's efforts to assess girls' delinquency programs, including its
approach of using a study group and the group's methods of assessing
studies, were consistent with generally accepted social science
standards for evaluation research. According to OJJDP officials,
including the research coordinator, they chose to form a study group
rather than fund individual evaluations of programs because study
groups are a cost-effective method of gaining an overview of the
available research in an issue area. As part of its work, the group
collected, reviewed, and analyzed the methodological quality of
research on girls' delinquency programs. Such an approach of
systematically identifying and reviewing research conducted in a field
of study is an acceptable practice to consolidate the research in an
area and provide information to enable program managers to determine
where they might best commit future evaluation resources.[Footnote 30]
Thirteen of the 18 girls' delinquency experts we interviewed (including
11 Study Group members) stated that the Study Group's efforts were
useful for providing an overview of girls' delinquency issues. However,
6 experts (including 2 Study Group members) also noted that it would
have been beneficial to the girls' delinquency field for the group to
conduct evaluations to determine program outcomes or promising models
rather than reviewing completed studies. OJJDP has funded individual
studies of girls' delinquency programs in the past but, according to
OJJDP officials, was seeking to use the Study Group's research to form
a baseline of the available knowledge about girls' delinquency issues.
The Study Group's effort to review the studies according to the What
Works criteria was consistent with generally accepted social science
standards. Specifically, we reviewed the OJP What Works criteria and
found that they adhere to these standards for evaluation research.
Using the What Works criteria, we also assessed the same studies for
the 17 girls' delinquency programs that the Study Group had reviewed
and generally concurred with the Study Group's ratings of the program
studies. While the Study Group's use of the What Works criteria was in
keeping with social science standards, experts we interviewed expressed
differing views on the group's decision to use these criteria.
According to the Study Group's former principal investigator, the group
decided to use the What Works criteria in 2005 because the criteria
ensured that the group's assessment of the effectiveness of programs in
preventing or reducing girls' delinquency would be based on highly
rigorous evaluation standards to identify effective programs--thus
eliminating the potential that a program that may do harm would be
endorsed by the group. Eight Study Group members we interviewed also
stated that the Study Group's use of the criteria was appropriate
because it ensured that the group would only disseminate information on
programs determined to be effective based on a high level of evidence.
However, 8 other experts, including three Study Group members, said
that the criteria created an unrealistically high standard, which
caused the Study Group to overlook potentially promising programs.
[Footnote 31] Further, 9 of the 18 experts (including five Study Group
members) we interviewed also noted that requiring a randomized
controlled research design--a research design that compares the
outcomes for individuals who are randomly assigned to either the
program being studied or to a nonparticipating control group before the
intervention--to demonstrate effectiveness, as the What Works criteria
does, is a difficult standard to achieve because such a design is
expensive, and programs may be reluctant to divert resources from
programming to pay for evaluations. OJJDP officials stated that they
understood the experts' concerns and the trade-offs in using a
classification framework that requires a randomized controlled research
design to demonstrate effectiveness; however, they approved the group's
use of the criteria because it provided a rigorous framework for
assessing program evaluations. We understand that studies can produce
valid results using other research designs, such as studies using quasi-
experimental designs or studies comparing the outcome results for
groups of girls that are statistically matched. We have also previously
reported that randomized controlled research designs provide
researchers with the best method for assessing a program's
effectiveness--they isolate changes caused by the program from other
factors--when doing so is feasible and ethical.[Footnote 32]
OJJDP Has Taken Actions to Reach Out to External Stakeholders on Study
Group Activities and Findings and Is Disseminating the Findings in
Keeping with Internal Control Standards:
OJJDP has taken action to reach out to external stakeholders to address
concerns about the composition of the Study Group after its initial
formation and, moving forward, plans to continue to incorporate program
practitioners in its planned efforts. Standards for Internal Control in
the Federal Government states that program managers should ensure that
there are adequate means of obtaining information from and
communicating with external stakeholders who may have a significant
impact on the agency achieving its goals.[Footnote 33] Regarding
gaining information from external stakeholders, OJJDP's program
announcement for the Study Group sought to ensure a range of
stakeholder perspectives related to girls' delinquency. The
announcement required that the members of the Study Group possess
knowledge of and experience with female development and delinquent
girls and demonstrate expertise in a variety of relevant social science
disciplines, such as criminology, sociology, and developmental
psychology. In awarding the cooperative agreement to RTI through a peer
review process, OJJDP approved the RTI proposal for the Study Group as
responding to the requirements and expectations of the program
announcement. Consistent with the fields of expertise cited in the
program announcement, RTI convened a group of 13 members, including 12
academic researchers from social science disciplines and one
practitioner, a member directly involved in girls' delinquency
programming.[Footnote 34] However, according to several of the experts
we interviewed, this Study Group composition did not include sufficient
representation and input from a key external stakeholders group--girls'
delinquency program practitioners. For example, 11 of the 18 girls'
delinquency experts we interviewed, including 5 study group members,
said that the Study Group was imbalanced in favor of academic
researchers, 6 of whom (including 2 study group members) said that the
composition led the group to focus its efforts on researching theories
of girls' delinquency rather than gathering and disseminating
actionable information for practitioners.[Footnote 35] According to
OJJDP officials we interviewed, they had received feedback from girls'
delinquency stakeholders in 2006 on this issue. In response, according
to OJJDP program managers, they acted to address the imbalance of the
Study Group by adding a second practitioner as a member and involving 2
other practitioners in group activities, such as presenting successful
girls' delinquency program practices at conferences and reviewing the
group's work products.[Footnote 36] OJJDP officials stated that as a
lesson learned, they plan to more fully involve practitioners from the
beginning when they organize study groups in the future. In addition,
OJJDP officials noted that specific to the Study Group, they plan to
continue to reach out to obtain information from and include
practitioners in the remaining activities of the group, such as
presenting findings at a national juvenile justice conference.
OJJDP and the Study Group have disseminated the group's findings to the
research community, practitioners in the girls' delinquency field, and
the public in a variety of ways, and in doing so have made efforts to
respond to stakeholder concerns. In its 2003 program announcement, in
keeping with the internal control standard for communicating with
stakeholders, OJJDP required that the Study Group disseminate its
findings through publications and products that address the needs of
various practitioner audiences in diverse fields, including juvenile
justice, child welfare, mental health, and substance abuse prevention.
Since 2004, Study Group principal investigators and group members have
presented findings at 24 conferences and posted the presentation slides
to the group's Web site.[Footnote 37] OJJDP has also published three
bulletins on the Study Group's activities and findings. Six girls'
delinquency experts we interviewed (including five Study Group members)
stated that the information disseminated was generally helpful because
it provided a useful overview of girls' delinquency trends and
research. However, 10 of the 18 experts we interviewed (including three
Study Group members) also noted that some of the group's dissemination
efforts created confusion among practitioners because Study Group
members presented findings that did not acknowledge factors that
practitioners believed contribute to girls' delinquency, such as
traumatic life experiences. According to OJJDP officials, in response
to feedback they received from girls' delinquency stakeholders about
such concerns, the office and the Study Group sponsored workshop
sessions at a conference for juvenile justice practitioners where the
group clarified its findings and sought practitioner input on subjects
such as delinquency risk and protective factors and trends in girls'
delinquency. According to OJJDP officials, the office and the Study
Group plan to continue disseminating the group's findings by issuing
four additional bulletins and by presenting the findings at a national
conference on juvenile delinquency.
In Response to Study Group Findings of No Evidence of Effective Girls'
Delinquency Programs, OJJDP Plans Technical Assistance to Help Programs
but Could Strengthen Its Plans for Supporting Evaluations:
The OJJDP-sponsored Study Group found that no programs in its review
had evidence of effectiveness and, among other things, that additional
support for program evaluation is needed. To address these findings,
OJJDP plans to provide technical assistance to help girls' delinquency
programs so that they will be better prepared to be evaluated. However,
by articulating time frames and specific funding requirements and
commitments in its plans to support evaluations, OJJDP could better
address its goals for preventing and reducing girls' delinquency.
The Study Group Found No Evidence of Effective Girls' Delinquency
Programs to Promote as Models and, among Other Things, That Evaluation
Is Needed:
In its review of girls' delinquency programs, the Study Group's
findings showed that the majority of the programs it identified--44 of
61--had not been studied by researchers, while 17 of the programs had
been the subject of published studies. The Study Group determined that
none of the 17 programs that had been studied had conclusive evidence
of their effectiveness. Specifically, the Study Group found that the
studies provided insufficient evidence of the effectiveness of 11 of
these 17 programs. For example, our review of one study that the Study
Group assessed as having insufficient evidence showed that the study
had a quasi-experimental design but lacked methodological rigor in that
the treatment and comparison groups had small sample sizes and did not
appear to be well matched, and any statistical tests reported were only
performed on treatment group participants.[Footnote 38] The Study Group
found that for the remaining 6 programs, the studies provided
inconclusive evidence of effectiveness. For example, our review of one
study that the group assessed as having inconclusive evidence showed
statistically significant results for the program; however, sustained
effects were not indicated for at least a 1-year period beyond the end
of the intervention. Further, it was unclear whether the study
participants were representative of the population of girls that the
program was designed to reach. As a result, there was not enough
evidence to demonstrate that the program was either effective or
ineffective for the intended population of delinquent girls. Among the
other findings that the Study Group reported was that 7 of the 17
programs it assessed were no longer in operation, primarily because the
initial grants that supported their operations were not renewed.
Based on its review of girls' delinquency programs, the Study Group
reported several conclusions and recommendations. Among these
conclusions and recommendations is the need for evaluations and support
of girls' delinquency programs. In particular, the Study Group found
that insufficient funding has been provided for evaluations of girls'
delinquency programs, so definitive conclusions of what works for girls
cannot be made. Further, the Study Group found that additional,
methodologically rigorous evaluations of girls' delinquency programs
are needed in order to identify effective and promising programs and
models that could be replicated at the state and local levels. While
the Study Group did not specifically quantify the funding needed to
support these evaluations, it did note that federal sources for
evaluation funding and partnerships with local colleges and
universities are needed. The Study Group also concluded that programs
need technical assistance to help them prepare for evaluations. Lastly,
the group found that girls' delinquency programs that are based on
evidence of promising techniques should be supported and expanded. In
particular, the Study Group highlighted program sustainability as an
issue, stating that funding needs to be provided to ensure that the
most promising programs continue to operate after their initial funding
period is over so that practitioners and policymakers can continue to
implement them.
OJJDP Plans to Provide Technical Assistance to Help Programs, but Could
More Fully Develop Plans for Supporting Evaluations to Address Its
Goals to Prevent and Reduce Girls' Delinquency:
OJJDP has plans to provide technical assistance to girls' delinquency
programs; however, its plans for supporting evaluations could be more
fully developed to help OJJDP reach its goals for addressing girls'
delinquency issues. OJJDP's goals for addressing girls' delinquency, as
stated in the Study Group program announcement, are to identify
effective and promising programs, program elements, and implementation
principles and support the development of program models to prevent and
reduce girls' delinquency. According to OJJDP officials, in response to
the group's finding about the need to better prepare programs for
evaluation, the office plans to work with the Study Group and using the
remainder of its funding--approximately $300,000--provide a technical
assistance workshop in October 2009 to help about 10 girls' delinquency
programs prepare to be evaluated. In this workshop, OJJDP and the Study
Group plan to provide information to programs about how evaluations are
designed and conducted, how to identify appropriate performance
measures, and how to collect data needed for program evaluators to
assess outcomes. OJJDP officials stated that the programs are to be
selected for participation through an application process and have to
meet minimum criteria, including having experience working with girls
and the capability to collect program outcome data. OJJDP officials
noted that they intend to limit participation in the workshop to about
10 programs to ensure that the programs that are selected receive
technical assistance that is targeted to their specific needs. This
assistance, according to OJJDP officials, will help ensure that when
programs do undergo evaluations--whether funded by OJJDP, another
federal agency, or an independent research organization--the
evaluations will be more likely to lead to conclusive findings on
program effectiveness.
In addition to providing girls' delinquency programs with training and
technical assistance, OJJDP officials also described their plan to fund
evaluations of girls' delinquency programs. OJJDP officials stated that
as a result of the Study Group's findings along with feedback they
received from members of the girls' delinquency field, they recognized
the need for evaluations of girls' delinquency programs. OJJDP
officials stated that they recognized the need for evaluation in fiscal
year 2007 but at the time lacked funding to issue a solicitation for
such evaluations. Further, 14 of the 18 girls' delinquency experts that
we interviewed (including nine Study Group members) emphasized the need
for OJJDP leadership in supporting evaluations of girls' delinquency
programs to identify effective programs. For example, one expert noted
that since the Study Group found that few programs had been studied,
OJJDP would be doing a disservice to the girls' delinquency field if it
did not fund rigorous evaluations and help programs partner with
research organizations. According to the OJJDP officials, the office's
goal is to issue a solicitation in early fiscal year 2010 for
researchers to apply for funding to conduct evaluations of two to five
girls' delinquency programs. These evaluations, according to OJJDP
officials, are to focus on girls' delinquency programs that have been
in operation for a number of years and have data to support
evaluations. The officials also stated that the planned solicitation
would require researchers to conduct studies that involve either
randomized controlled or quasi-experimental research designs.
OJJDP officials stated that they expect to fund evaluations using the
portion of appropriation accounts that has been available for research
and evaluations, and noted that the number of evaluations to be
allocated funding depends, in part, on the number of applications
received, the total available funding, as well as other competing
research needs and goals. While OJJDP has not yet received an
appropriation for fiscal year 2010, OJJDP used approximately $12
million in fiscal year 2007 and $14 million in fiscal year 2008 to
support research and evaluations from accounts eligible to support
research and evaluations of girls' delinquency programs. OJJDP
officials stated that they used this funding because in recent years
they have not received an appropriation for programs and activities
authorized under Part D, which is specifically designated for research
and evaluation, but if they were to receive a Part D appropriation they
could increase the number of evaluations funded. While OJJDP officials
verbally described the planned evaluations and funding, they did not
provide us with written documentation of the planned solicitation
because, as of June 2009, it was in draft and subject to change.
OJJDP officials have described actions they plan to take to respond to
the Study Group's findings, and OJJDP reported that these findings will
provide a foundation for creating a comprehensive program of
information dissemination, training, technical assistance, and
programming to help prevent and reduce girls' delinquency. However, the
office has not developed a plan that is documented, is shared with key
stakeholders, and includes time frames and specific funding
requirements and commitments for meeting its girls' delinquency goals.
According to OJJDP officials, they have not developed such a plan
because the office is in transition and is in the process of initiating
efforts to develop an officewide research plan, but they are taking
steps to address their girls' delinquency goals, for example, through
the workshop and planned evaluations. Standard practices for program
and project management state that specific desired outcomes or results
should be conceptualized, defined, and documented in the planning
process as part of a road map, along with the appropriate projects
needed to achieve those results, supporting resources, and
milestones.[Footnote 39] Standards for Internal Control in the Federal
Government states that program managers should ensure that there are
adequate means of obtaining information from and communicating with
external stakeholders who may have a significant impact on the agency
achieving its goals.[Footnote 40] We have also previously reported that
critical to guiding evaluation and research efforts on a national level
is a strategy that outlines a process for funding and conducting
rigorous evaluations and research, identifies the resources needed to
achieve it, and assigns accountability for accomplishing these actions.
[Footnote 41] In that regard, developing a plan that provides a road
map to meeting its goals would provide additional assurance that
OJJDP's goals for identifying and promoting promising programs and
program models would be met and communicated to state and local
policymakers and practitioners responsible for implementing programs to
prevent and reduce girls' delinquency.
Conclusions:
Preventing and responding to girls' delinquency have been a concern for
federal, state, and local governments as well as private and nonprofit
juvenile justice organizations for over a decade, and the most recent
statistics show that girls' involvement in the juvenile justice system
is not stabilizing or declining. While OJJDP has undertaken a 6-year,
$2.6 million study group effort to learn about effective and promising
girls' delinquency programs, the lack of rigorous studies of such
programs meant that the group was unable to identify and promote
effective programs and to develop program models to be supported at
state and local levels. In response to these findings, OJJDP has taken
steps to provide technical assistance to programs to help prepare them
for evaluations and has described plans for funding evaluations of
girls' delinquency programs. While these steps are consistent with
OJJDP's stated goals, the office lacks a comprehensive documented plan
that includes time frames and specific funding requirements and
commitments for meeting its girls' delinquency goals that it can share
with stakeholders. As the Study Group plans to conclude its efforts in
spring 2010, OJJDP is planning to help ensure the development of
effective girls' delinquency programs and program models by providing
training and technical assistance to help these programs plan for
future evaluations. Moreover, such action better positions OJJDP in
ensuring that funding for such programs is directed to those that are
effective in preventing girls' delinquency and intervening after girls
have entered the juvenile justice system. As states are continuing to
make determinations about how to allocate their formula and block
grants, and OJJDP continues to provide funding to programs through some
of its discretionary grant programs, information about promising or
effective programs and program models could help guide these resource
decisions. Developing a plan with time frames that clearly articulates
the office's approach to its evaluation efforts, including available
resources needed and committed toward implementing that plan, would
help OJJDP ensure that its goals to support the development of
effective programs are met, and sharing that plan with stakeholders
would help demonstrate federal leadership to the girls' delinquency
field.
Recommendation for Executive Action:
To help ensure that OJJDP meets its goals to identify effective or
promising girls' delinquency programs and supports the development of
program models, we recommend that the Administrator of OJJDP develop
and document a plan that (1) articulates how the office intends to
respond to the program findings of the Study Group, (2) includes time
frames and specific funding requirements and commitments, and (3) is
shared with key stakeholders.
Agency Comments and Our Evaluation:
We requested comments on a draft of this report from the Attorney
General. On July 16, 2009, we received written comments from OJP, which
are reprinted in the enclosure.
OJP agreed with our recommendation and stated that OJJDP has always
intended to respond to the findings of the Study Group. OJP described
efforts planned in response to the findings of the Study Group,
including a technical assistance workshop and evaluations, which we
have discussed in our report. OJP also stated that subsequent refined
plans and related funding commitments will be based on the outcome of
these activities and noted that OJJDP, in accordance with the Act, will
publish these program plans in the Federal Register for review and
comment by key stakeholders as well as members of the public by
December 2009.
We recognize that OJJDP's planned activities represent a worthwhile
step in responding to the findings from the Study Group effort, and are
encouraged that OJJDP intends to publish a program plan, to include how
it will address girls' delinquency issues. However, it is important to
note that while OJJDP has been required to publish a program plan
annually according to the Act, it has not done so since 2002.[Footnote
42] Following through on its current pledge to issue such a plan by
December of this year will help provide OJJDP with reasonable assurance
that it has a well-thought-out approach to ensure that its goals for
preventing and reducing girls' delinquency are met. We also continue to
maintain that it will be important for this plan to include more than a
list of activities in response to the Study Group's findings as OJJDP
describes in commenting on this report. Specifically, the plan should
serve as a road map for OJJDP's approach for responding to the Study
Group's findings, establish overall time frames as well as those for
each activity, specify funding requirements and associated commitments,
and integrate the input of key stakeholders, such as girls' delinquency
practitioners. Publishing and implementing such a plan would help OJJDP
ensure that it meets the goal it articulated at the beginning of the 6-
year Study Group effort--to identify effective and promising programs,
program elements, and implementation principles and to support the
development of program models to prevent and reduce girls' delinquency.
We are sending copies of this report to interested congressional
committees, the Attorney General, and other interested parties. In
addition, this report will be available at no charge on GAO's Web site
at [hyperlink, http://www.gao.gov].
If you or your staff have any questions concerning this report, please
contact me at (202) 512-6510 or larencee@gao.gov. Contact points for
our Offices of Congressional Relations and Public Affairs may be found
on the last page of this report. Mary Catherine Hult, Assistant
Director; David Alexander; Elizabeth Blair; Amy Brown; Kevin Copping;
Katherine Davis; Dawn Locke; and Janet Temko made key contributions to
this report.
Sincerely yours,
Signed by:
Eileen Regen Larence:
Director, Homeland Security and Justice Issues:
Enclosure:
[End of section]
Enclosure: Comments from the Department of Justice:
U.S. Department of Justice:
Office of Justice Programs:
Office of Audit, Assessment, and Management:
Washington, DC 20531:
July 16, 2009:
Ms. Eileen R. Larence:
Director, Homeland Security and Justice Issues:
Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Ms. Larence:
Thank you for the opportunity to comment on the draft Government
Accountability Office (GAO) letter report entitled "Juvenile Justice:
Technical Assistance and Better Defined Evaluation Plans Will Help to
Improve Girls' Delinquency Programs" (GAO-09721R). The Office of
Justice Programs agrees with the Recommendation for Executive Action,
which is restated in bold text below and is followed by our response.
To help ensure that OJJDP meets its goals to identify effective or
promising girls' delinquency programs and supports the development of
program models, we recommend that the Administrator of OJJDP develop
and document a plan, that (1) articulates how the agency intends to
respond to the program findings of the Girls Study Group, (2) includes
time frames and specific funding requirements and commitments, and (3)
is shared with key stakeholders.
It has always been the intention of the Office of Juvenile Justice and
Delinquency Prevention (OJJDP) to respond to the findings of the Girls'
Study Group. As discussed with the GAO during the review, OJJDP has
efforts underway, as well as planned initiatives to address the
findings. These planned initiatives, and timeframes for implementation,
are described below. Further planning and funding commitments will be
based on the outcome of these activities.
* Hands-On Evaluation Technical Assistance Workshop for Girls'
Delinquency Programs:
The workshop is scheduled for October 28-30, 2009, in Chapel Hill, NC.
The goal of the workshop is to better equip programs to conduct
rigorous evaluations of their interventions. Unlike general workshops,
the Girls' Study Group Evaluation Technical Assistance Workshop will
tailor instruction specifically to address the needs of participating
programs. The faculty at the workshops will be highly skilled in
evaluation methodology, program development, and strategies on how to
partner with evaluation professionals. Each participant will leave the
workshop with a customized concrete plan for `next steps' and upon
request will receive an additional hour of technical assistance by
phone following the workshop.
Eligibility will be limited to programs that provide gender-responsive
delinquency prevention or interventions for girls, and who have some
level of evaluation experience. Organizations that have more than one
distinct gender responsive program are eligible to submit more than one
application. There will be a two-phase application process. The first
phase will require the submission of general program information and a
description of current evaluation history/experience. Program
applications will be reviewed based on their program type and
evaluation needs; and approximately 15 to 20 programs will be invited
to proceed to the next application phase. Those applicants selected for
the second phase will be asked to submit more detailed information on
the evaluation needs of their programs and reports or findings based on
previous evaluation work. The workshop organizers will use this
information to determine which programs provide the best fit between
evaluation needs and faculty expertise. Approximately 10 programs will
ultimately be invited to participate in the workshop. Selection of
participants for the workshop will be completed by the end of September
2009.
* Enhancement of OJJDP's Current Girls Delinquency Training and
Technical Assistance Curriculum:
Using a panel of experts (including staff from the Girls' Study Group,
among others), OJJDP will update, enhance, and revise the existing
Training and Technical Assistance Curriculum for Girls' Delinquency
Programming. The targeted completion date is December 2009.
* Release of the FY 2010 Evaluation of Girls' Delinquency Programs
Solicitation:
This solicitation will be released, pending availability of funds, for
the purpose of encouraging partnerships between girls' delinquency
programs and evaluators, and providing funding for experimental and
quasi-experimental evaluations of girls' delinquency programs.
In response to the findings of the Girls' Study Group, OJJDP has
planned the above described initiatives. Subsequent refined plans and
related funding commitments will be based on the outcome of these
initiatives. As mandated by the Juvenile Justice and Delinquency
Prevention Act, the OJJDP will publish these program plans in the
Federal Register for review and comment by key stakeholders as well as
members of the public. OJJDP anticipates publishing the program plan in
the Federal Register by December 2009.
If you have any questions regarding this response, you or your staff
may contact Maureen Henneberg, Director, Office of Audit, Assessment,
and Management, on (202) 616-3282.
Sincerely,
Signed by:
Laurie O. Robinson:
Acting Assistant Attorney General:
cc:
Beth McGarry:
Deputy Assistant Attorney for Operations and Management:
Jeffrey Slowikowski:
Acting Director:
Office of Juvenile Justice and Delinquency Prevention:
Maureen Henneberg:
Director:
Office of Audit, Assessment, and Management:
LeToya A. Johnson:
Audit Liaison:
Office of Justice Programs:
Richard P. Theis:
Audit Liaison:
Department of Justice:
[End of section]
Footnotes:
[1] C. Puzzanchera and W. Kang, Juvenile Court Statistics Databook
(2008), [hyperlink, http://ojjdp.ncjrs.gov/ojstatbb/jcsdb/] (accessed
June 30, 2009). Most current data available.
[2] C. Puzzanchera, Juvenile Arrests 2007, (2009) [hyperlink,
www.ncjrs.gov/pdffiles1/ojjdp/225344.pdf] (accessed June 26, 2009).
[3] The Federal Advisory Committee on Juvenile Justice is an advisory
body established by the Juvenile Justice and Delinquency Prevention Act
of 1974, as amended, to advise the President and Congress on state
perspectives regarding the operation of the Office of Juvenile Justice
and Delinquency Prevention and on federal legislation pertaining to
juvenile justice and delinquency prevention, to advise the
Administrator of the Office of Juvenile Justice and Delinquency
Prevention, and to review federal policies regarding juvenile justice
and delinquency prevention. 42 U.S.C. § 5633(f). The Federal Advisory
Committee on Juvenile Justice comprises appointed representatives from
each of the 50 states, the District of Columbia, and the 5 U.S.
territories.
[4] Cooperative agreements, rather than grant awards, can be used by
federal agencies when substantial involvement is expected between the
agency and the recipient when carrying out the activities described in
the program announcement.
[5] GAO defines an expert as a person who is recognized by others who
work in the same subject matter area as having knowledge that is
greater in scope or depth than that of most people working in that
area. The expert's knowledge can come from education, experience, or
both. We specifically identified researchers who focus on girls'
delinquency issues and practitioners who operate programs that address
girls' delinquency.
[6] We contacted all 15 of the study group members. However, 1 member
declined to be interviewed, and 3 study group members did not respond
to requests for interviews.
[7] For social science standards for evaluation research, see Donald T.
Campbell and Julian Stanley, Experimental and Quasi-Experimental
Designs for Research (Chicago: Rand McNally, 1963); William R. Shadish,
Thomas D. Cook, and Donald T. Campbell, Experimental and Quasi-
Experimental Designs for Generalized Causal Inference (Boston: Houghton
Mifflin, 2002); Carol H. Weiss, Evaluation: Methods for Studying
Programs and Policies, Second Edition (Englewood Cliffs, N.J.: Prentice-
Hall, Inc., 1998); and GAO, Designing Evaluations, [hyperlink,
http://www.gao.gov/products/GAO/PEMD-10.1.4] (Washington, D.C.: May
1991).
[8] GAO, Standards for Internal Control in the Federal Government,
[hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]
(Washington, D.C.: November 1999).
[9] Program management standards we reviewed are reflected in the
Project Management Institute's The Standard for Program Management ©
(2006).
[10] The What Works criteria define six levels of effectiveness,
including effective, promising, and ineffective, for use in assessing
and classifying studies on the basis of their evidence of
effectiveness. Additional details on these criteria are discussed later
in this report.
[11] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1].
[12] The other seven experts did not express views regarding the
balance of the study group's composition.
[13] Project Management Institute, The Standard for Program Management.
[14] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1].
[15] The Study Group found that possible reasons for increased arrest
rates for girls include changes in local law enforcement policies that
lowered the threshold for reporting assaults or categorizing assaults
as aggravated, reclassification of domestic dispute offenses as simple
assaults that can result in arrest, and increased referrals to police
resulting from schools' zero tolerance policies for violence.
[16] Elizabeth Cauffman and others, "Gender Differences in Mental
Health Symptoms among Delinquent and Community Youth," Youth Violence
and Juvenile Justice, vol. 5, no. 3 (2007): 287-307.
[17] 42 U.S.C. § 5611.
[18] OJJDP allocates some formula and block grants to states on the
basis of states' juvenile populations, while others may be awarded on
the basis of a fixed level to all states. Discretionary grants are
generally awarded through a competitive process to state and local
governments as well as individual agencies and organizations. Under the
Act, "state" means any of the United States, the District of Columbia,
the Commonwealth of Puerto Rico, the Virgin Islands, Guam, American
Samoa, and the Commonwealth of the Northern Mariana Islands. 42 U.S.C.
§ 5603.
[19] 42 U.S.C. § 5633(a)(7)(B).
[20] OJJDP did not report awarding discretionary grants for girls'
programs in fiscal year 2008, and as of June 2009, OJJDP had not
awarded fiscal year 2009 discretionary grants.
[21] 42 U.S.C. § 5614(b)(3).
[22] 42 U.S.C. §§ 5661-62.
[23] Appropriations statutes for fiscal years 2006 through 2008
provided that OJJDP may use not more than 10 percent of each amount
appropriated for research, evaluation, and statistics activities that
benefit the programs or activities authorized, and not more than 2
percent of each appropriated amount for training and technical
assistance. See, e.g, Consolidated Appropriations Act, 2008, Pub. L.
No. 110-161, 121 Stat. 1844, 1906-07 (2007). This provision applied to
appropriation accounts under Juvenile Justice Programs, but did not
apply to amounts appropriated for demonstration projects, as authorized
by sections 261 and 262 of the Act, 42 U.S.C. §§ 5665-66.
[24] The $40 million comprises set asides eligible to be used for
research and evaluation of girls' delinquency programs. As of July
2009, OJJDP has not determined how it would use its fiscal year 2009
appropriation set asides.
[25] Greene, Peters Associates, Guiding Principles for Promising Female
Programming: An Inventory of Best Practices (Washington, D.C.: Office
of Juvenile Justice and Delinquency Prevention, 1998).
[26] In 2000, OJJDP issued program announcements for two separate
girls' delinquency efforts--a study group and a girls' institute. The
first effort, a girls study group, was awarded to a university in 2001.
However, because it was unable to reach agreement on project management
issues, the university terminated the agreement in 2002. In 2003, OJJDP
reissued the program announcement for a girls study group and revised
the announcement to clearly delineate the level of expected federal
involvement, for example, by explicitly stating that OJJDP planned to
review and approve all project consultants, plans, and products
developed. The second effort, a national girls' institute, was intended
to put the study group's findings into practice by, among other things,
promoting programs for girls; providing training and technical
assistance to the field on girls' delinquency issues; facilitating
coordination among federal, state, and local organizations serving
girls; and disseminating information about the research findings of the
study group. According to OJJDP officials, the 2000 announcement never
received funding, and OJJDP did not reissue it in later years because
of funding constraints. Instead, when OJJDP reissued the program
announcement in 2003 for a study group, it incorporated elements of the
planned institute. For example, the 2003 study group solicitation
included objectives for identifying and promoting programs for girls
and for disseminating information to the practitioner field.
[27] Specifically, the Study Group's objective to understand the trends
and consequences of girls' delinquency involved increasing research-
based knowledge about the risk and protective factors related to girls'
delinquency and determining the patterns and consequences of juvenile
justice decisions on female offenders. The objective on developing a
comprehensive theory of girls' delinquency involved examining the
extent to which theories developed primarily to explain boys'
delinquency applied to girls, as well as exploring whether theories
that had been developed for girls were useful in developing and testing
new prevention and intervention strategies.
[28] Under the Act, states are required to submit 3-year plans to OJJDP
outlining their activities for investing in delinquency prevention and
for coordinating services delivered to at-risk juveniles and their
families, among other things. 42 U.S.C. § 5633.
[29] A multiagency working group led by DOJ's OJP, which included the
Department of Health and Human Services and the Department of
Education, developed the classification framework and criteria from
2004 to 2005 to support a planned What Works repository to assist
communities in selecting and replicating evidence-based programs that
was never implemented. Federal government efforts to develop
repositories of evidence-based programs have continued under Find Youth
Info, the Substance Abuse and Mental Health Services Administration's
National Registry of Evidence-Based Programs and Practices, and the
OJJDP Model Programs Guide. Even though the repository was never
implemented, the criteria within its framework are still valid to use
in assessing evidence of program effectiveness.
[30] The approach used by OJJDP is similar to the evaluation synthesis
methodology described in GAO, The Evaluation Synthesis, GAO/PEMD-10.1.2
(Washington, D.C.: March 1992). This type of approach might also be
termed systematic review.
[31] Two experts we interviewed did not express a view on the group's
approach to evaluating programs.
[32] GAO, Juvenile Justice: OJJDP Reporting Requirements for
Discretionary and Formula Grantees and Concerns About Evaluation
Studies, [hyperlink, http://www.gao.gov/products/GAO-02-23]
(Washington, D.C.: Oct. 30, 2001); Justice Outcome Evaluations: Design
and Implementation of Studies Require More NIJ Attention, [hyperlink,
http://www.gao.gov/products/GAO-03-1091] (Washington, D.C.: Sept. 24,
2003); Adult Drug Courts: Evidence Indicates Recidivism Reductions and
Mixed Results for Other Outcomes, [hyperlink,
http://www.gao.gov/products/GAO-05-219] (Washington, D.C.: Feb. 28,
2005); and Abstinence Education: Assessing the Accuracy and
Effectiveness of Federally Funded Programs, [hyperlink,
http://www.gao.gov/products/GAO-08-664T] (Washington, D.C.: Apr. 23,
2008).
[33] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1].
[34] The Study Group members represented 11 of the 12 disciplines
specified in the program announcement.
[35] The other seven experts did not express views regarding the
balance of the study group's composition.
[36] In addition to the practitioner, OJJDP also added an expert in
program evaluation as a group member after the group had begun its
activities.
[37] The Web site is located at [hyperlink,
http://girlsstudygroup.rti.org/index.cfm?fuseaction=dsp_home].
[38] A quasi-experimental design is a controlled study where study
participants are assigned in a nonrandom manner to a treatment group
(individuals participating in the program being studied) or a
comparison group (individuals closely resembling those in the treatment
group on many demographic variables but not participating in the
program).
[39] Project Management Institute, The Standard for Program Management.
[40] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1].
[41] GAO, South Florida Ecosystem Restoration: A Strategic Plan and a
Process to Resolve Conflicts Are Needed to Keep the Effort on Track,
[hyperlink, http://www.gao.gov/products/GAO/T-RCED-99-170] (Washington,
D.C.: Apr. 29, 1999); South Florida Ecosystem Restoration: Substantial
Progress Made in Developing a Strategic Plan, but Actions Still Needed,
[hyperlink, http://www.gao.gov/products/GAO-01-361] (Washington, D.C.:
Mar. 27, 2001); and Great Lakes: A Coordinated Strategic Plan and
Monitoring System Are Needed to Achieve Restoration Goals, [hyperlink,
http://www.gao.gov/products/GAO-03-999T] (Washington, D.C.: July 16,
2003).
[42] 42 U.S.C. § 5614.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
