Juvenile Justice

Technical Assistance and Better Defined Evaluation Plans Will Help Girls' Delinquency Programs Gao ID: GAO-10-133T October 20, 2009

This testimony discusses issues related to girls' delinquency--a topic that has attracted the attention of federal, state, and local policymakers for more than a decade as girls have increasingly become involved in the juvenile justice system. For example, from 1995 through 2005, delinquency caseloads for girls in juvenile justice courts nationwide increased 15 percent while boys' caseloads decreased by 12 percent. More recently, in 2007, 29 percent of juvenile arrests--about 641,000 arrests--involved girls, who accounted for 17 percent of juvenile violent crime arrests and 35 percent of juvenile property crime arrests. Further, research on girls has highlighted that delinquent girls have higher rates of mental health problems than delinquent boys, receive fewer special services, and are more likely to abandon treatment programs. The Office of Juvenile Justice and Delinquency Prevention (OJJDP) is the Department of Justice (DOJ) office charged with providing national leadership, coordination, and resources to prevent and respond to juvenile delinquency and victimization. OJJDP supports states and communities in their efforts to develop and implement effective programs to, among other things, prevent delinquency and intervene after a juvenile has offended. For example, from fiscal years 2007 through 2009, Congress provided OJJDP almost $1.1 billion to use for grants to states, localities, and organizations for a variety of juvenile justice programs, including programs for girls. Also, in support of this mission, the office funds research and program evaluations related to a variety of juvenile justice issues. As programs have been developed at the state and local levels in recent years that specifically target preventing girls' delinquency or intervening after girls have become involved in the juvenile justice system, it is important that agencies providing grants and practitioners operating the programs have information about which of these programs are effective. In this way, agencies can help to ensure that limited federal, state, and local funds are well spent. In general, effectiveness is determined through program evaluations, which are systematic studies conducted to assess how well a program is working--that is, whether a program produced its intended effects. To help ensure that grant funds are being used effectively, you asked us to review OJJDP's efforts related to studying and promoting effective girls' delinquency programs. We issued a report on the results of that review on July 24, 2009. This testimony highlights findings from that report and addresses (1) efforts OJJDP has made to assess the effectiveness of girls' delinquency programs, (2) the extent to which these efforts are consistent with generally accepted social science standards and federal standards to communicate with stakeholders, and (3) the findings from OJJDP's efforts and how the office plans to address the findings. This statement is based on our July report and selected updates made in October 2009.

With an overall goal of developing research that communities need to make sound decisions about how best to prevent and reduce girls' delinquency, OJJDP established the Girls Study Group (Study Group) in 2004 under a $2.6 million multiyear cooperative agreement with a research institute. OJJDP's objectives for the group, among others, included identifying effective or promising programs, program elements, and implementation principles (i.e., guidelines for developing programs). Objectives also included developing program models to help inform communities of what works in preventing or reducing girls' delinquency, identifying gaps in girls' delinquency research and developing recommendations for future research, and disseminating findings to the girls' delinquency field about effective or promising programs. OJJDP's effort to assess girls' delinquency programs through the use of a study group and the group's methods for assessing studies were consistent with generally accepted social science research practices and standards. In addition, OJJDP's efforts to involve practitioners in Study Group activities and disseminate findings were also consistent with the internal control standard to communicate with external stakeholders, such as practitioners operating programs. The Study Group found that few girls' delinquency programs had been studied and that the available studies lacked conclusive evidence of effective programs; as a result, OJJDP plans to provide technical assistance to help programs be better prepared for evaluations of their effectiveness. However, OJJDP could better address its girls' delinquency goals by more fully developing plans for supporting such evaluations.



GAO-10-133T, Juvenile Justice: Technical Assistance and Better Defined Evaluation Plans Will Help Girls' Delinquency Programs This is the accessible text file for GAO report number GAO-10-133T entitled 'Juvenile Justice: Technical Assistance and Better Defined Evaluation Plans Will Help Girls' Delinquency Program' which was released on October 20, 2009. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Testimony: Before the Subcommittee on Crime, Terrorism, and Homeland Security, Committee on the Judiciary, House of Representatives: United States Government Accountability Office: GAO: For Release on Delivery Expected at 2:30 p.m. EDT: Tuesday, October 20, 2009: Juvenile Justice: Technical Assistance and Better Defined Evaluation Plans Will Help Girls' Delinquency Programs: Statement of Eileen R. Larence, Director: Homeland Security and Justice: [End of section] GAO-10-133T: Mr. Chairman and Members of the Subcommittee: I am pleased to be here today as you examine issues related to girls' delinquency--a topic that has attracted the attention of federal, state, and local policymakers for more than a decade as girls have increasingly become involved in the juvenile justice system. For example, from 1995 through 2005, delinquency caseloads for girls in juvenile justice courts nationwide increased 15 percent while boys' caseloads decreased by 12 percent.[Footnote 1] More recently, in 2007, 29 percent of juvenile arrests--about 641,000 arrests--involved girls, who accounted for 17 percent of juvenile violent crime arrests and 35 percent of juvenile property crime arrests.[Footnote 2] Further, research on girls has highlighted that delinquent girls have higher rates of mental health problems than delinquent boys, receive fewer special services, and are more likely to abandon treatment programs.[Footnote 3] The Office of Juvenile Justice and Delinquency Prevention (OJJDP) is the Department of Justice (DOJ) office charged with providing national leadership, coordination, and resources to prevent and respond to juvenile delinquency and victimization. OJJDP supports states and communities in their efforts to develop and implement effective programs to, among other things, prevent delinquency and intervene after a juvenile has offended. For example, from fiscal years 2007 through 2009, Congress provided OJJDP almost $1.1 billion to use for grants to states, localities, and organizations for a variety of juvenile justice programs, including programs for girls. Also, in support of this mission, the office funds research and program evaluations related to a variety of juvenile justice issues. As programs have been developed at the state and local levels in recent years that specifically target preventing girls' delinquency or intervening after girls have become involved in the juvenile justice system, it is important that agencies providing grants and practitioners operating the programs have information about which of these programs are effective. In this way, agencies can help to ensure that limited federal, state, and local funds are well spent. In general, effectiveness is determined through program evaluations, which are systematic studies conducted to assess how well a program is working--that is, whether a program produced its intended effects. To help ensure that grant funds are being used effectively, you asked us to review OJJDP's efforts related to studying and promoting effective girls' delinquency programs. We issued a report on the results of that review on July 24, 2009.[Footnote 4] My statement today, as requested, highlights findings from that report and addresses (1) efforts OJJDP has made to assess the effectiveness of girls' delinquency programs, (2) the extent to which these efforts are consistent with generally accepted social science standards and federal standards to communicate with stakeholders, and (3) the findings from OJJDP's efforts and how the office plans to address the findings. My statement is based on our July report and selected updates made in October 2009.[Footnote 5] For our report, we reviewed documentation about OJJDP's establishment of a study group to assess the effectiveness of girl's delinquency programs, analyzed the groups' activities and findings, and interviewed OJJDP research and program officials and the current and former principal investigators of the study group. Specifically, we reviewed the criteria the study group used to assess studies of girls' delinquency programs and whether the group's application of those criteria was consistent with generally accepted social science standards for evaluation research.[Footnote 6] We also compared OJJDP's efforts with criteria in Standards for Internal Control in the Federal Government, specifically that agency management should ensure that there are adequate means of obtaining information from and communicating with external stakeholders who may have a significant impact on the agency achieving its goals, such as practitioners operating programs or researchers assessing programs.[Footnote 7] In addition, we conducted interviews with 18 girls' delinquency subject matter experts, that is, researchers and practitioners, whom we selected on the basis of their knowledge and experience with girls' delinquency issues.[Footnote 8] While their comments cannot be generalized to all girls' delinquency experts, we nonetheless believe that their views gave us useful insights on issues related to girls' delinquency and OJJDP's efforts to assess girls' programs. Our work was performed in accordance with generally accepted government auditing standards. More detail about our scope and methodology is included in our July report.[Footnote 9] OJJDP Established the Girls Study Group to Assess the Effectiveness of Girls' Delinquency Programs: With an overall goal of developing research that communities need to make sound decisions about how best to prevent and reduce girls' delinquency, OJJDP established the Girls Study Group (Study Group) in 2004 under a $2.6 million multiyear cooperative agreement with a research institute.[Footnote 10] OJJDP's objectives for the group, among others, included identifying effective or promising programs, program elements, and implementation principles (i.e., guidelines for developing programs). Objectives also included developing program models to help inform communities of what works in preventing or reducing girls' delinquency, identifying gaps in girls' delinquency research and developing recommendations for future research, and disseminating findings to the girls' delinquency field about effective or promising programs. To meet OJJDP's objectives, among other activities, the Study Group identified studies of delinquency programs that specifically targeted girls by reviewing over 1,000 documents in relevant research areas. These included criminological and feminist explanations for girls' delinquency, patterns of delinquency, and the justice system's response to girls' delinquency. As a result, the group identified 61 programs that specifically targeted preventing or responding to girls' delinquency. Then, the group assessed the methodological quality of the studies of the programs that had been evaluated using a set of criteria developed by DOJ's Office of Justice Programs (OJP) called What Works to determine whether the studies provided credible evidence that the programs were effective at preventing or responding to girls' delinquency.[Footnote 11] The results of the group's assessment are discussed in the following sections. OJJDP Efforts to Assess Program Effectiveness Were Consistent with Social Science Practices and Standards, and OJJDP Has Taken Action to Enhance Communication about the Study Group with External Stakeholders: OJJDP's effort to assess girls' delinquency programs through the use of a study group and the group's methods for assessing studies were consistent with generally accepted social science research practices and standards. In addition, OJJDP's efforts to involve practitioners in Study Group activities and disseminate findings were also consistent with the internal control standard to communicate with external stakeholders, such as practitioners operating programs.[Footnote 12] According to OJJDP research and program officials, they formed the Study Group rather than funding individual studies of programs because study groups provide a cost-effective method of gaining an overview of the available research in an issue area. As part of its work, the group collected, reviewed, and analyzed the methodological quality of research on girls' delinquency programs. The use of such a group, including its review, is an acceptable approach for systematically identifying and reviewing research conducted in a field of study. This review helped consolidate the research and provide information to OJJDP for determining evaluation priorities. Further, we reviewed the criteria the group used to assess the studies and found that they adhere to generally accepted social science standards for evaluation research. We also generally concurred with the group's assessments of the programs based on these criteria. According to the group's former principal investigator, the Study Group decided to use OJP's What Works criteria to ensure that its assessment of program effectiveness would be based on highly rigorous evaluation standards, thus eliminating the potential that a program that may do harm would be endorsed by the group. However, 8 of the 18 experts we interviewed said that the criteria created an unrealistically high standard, which caused the group to overlook potentially promising programs. OJJDP officials stated that despite such concerns, they approved the group's use of the criteria because of the methodological rigor of the framework and their goal for the group to identify effective programs. In accordance with the internal control standard to communicate with external stakeholders, OJJDP sought to ensure a range of stakeholder perspectives related to girls' delinquency by requiring that Study Group members possess knowledge and experience with girls' delinquency and demonstrate expertise in relevant social science disciplines. The initial Study Group, which was convened by the research institute and approved by OJJDP, included 12 academic researchers and 1 practitioner; someone with experience implementing girls' delinquency programs. However, 11 of the 18 experts we interviewed stated that this composition was imbalanced in favor of academic researchers. In addition, 6 of the 11 said that the composition led the group to focus its efforts on researching theories of girls' delinquency rather than gathering and disseminating actionable information for practitioners.[Footnote 13] According to OJJDP research and program officials, they acted to address this issue by adding a second practitioner as a member and involving two other practitioners in study group activities. OJJDP officials stated that they plan to more fully involve practitioners from the beginning when they organize study groups in the future and to include practitioners in the remaining activities of the Study Group, such as presenting successful girls' delinquency program practices at a national conference. Also, in accordance with the internal control standard, OJJDP and the Study Group have disseminated findings to the research community, practitioners in the girls' delinquency field, and the public through conference presentations, Web site postings, and published bulletins. The group plans to issue a final report on all of its activities by spring 2010. The Study Group Found No Evidence of Effective Girls' Delinquency Programs; in Response OJJDP Plans to Assist Programs in Preparing for Evaluations but Could Strengthen Its Plans for Supporting Such Evaluations: The Study Group found that few girls' delinquency programs had been studied and that the available studies lacked conclusive evidence of effective programs; as a result, OJJDP plans to provide technical assistance to help programs be better prepared for evaluations of their effectiveness. However, OJJDP could better address its girls' delinquency goals by more fully developing plans for supporting such evaluations. In its review, the Study Group found that the majority of the girls' delinquency programs it identified--44 of the 61--had not been studied by researchers. For the 17 programs that had been studied, the Study Group reported that none of the studies provided conclusive evidence with which to determine whether the programs were effective at preventing or reducing girls' delinquency. For example, according to the Study Group, the studies provided insufficient evidence of the effectiveness of 11 of the 17 programs because, for instance, the studies involved research designs that could not demonstrate whether any positive outcomes, such as reduced delinquency, were due to program participation rather than other factors. Based on the results of this review, the Study Group reported that among other things, there is a need for additional, methodologically rigorous evaluations of girls' delinquency programs; training and technical assistance to help programs prepare for evaluations; and funding to support girls' delinquency programs found to be promising. According to OJJDP officials, in response to the Study Group's finding about the need to better prepare programs for evaluation, the office plans to work with the group and use the remaining funding from the effort--approximately $300,000--to provide a technical assistance workshop by the end of October 2009. The workshop is intended to help approximately 10 girls' delinquency programs prepare for evaluation by providing information about how evaluations are designed and conducted and how to collect data that will be useful for program evaluators in assessing outcomes, among other things. In addition, OJJDP officials stated that as a result of the Study Group's findings, along with feedback they received from members of the girls' delinquency field, OJJDP plans to issue a solicitation in fiscal year 2010 for funding to support evaluations of girls' delinquency programs. OJJDP has also reported that the Study Group's findings are to provide a foundation for moving ahead on a comprehensive program related to girls' delinquency. However, OJJDP has not developed a plan that is documented, is shared with key stakeholders, and includes specific funding requirements and commitments and time frames for meeting its girls' delinquency goals. Standard practices for program and project management state that specific desired outcomes or results should be conceptualized, defined, and documented in the planning process as part of a road map, along with the appropriate projects needed to achieve those results, supporting resources, and milestones.[Footnote 14] In addition, government internal control standards call for policies and procedures that establish adequate communication with stakeholders as essential for achieving desired program goals.[Footnote 15] According to OJJDP officials, they have not developed a plan for meeting their girls' delinquency goals because the office is in transition and is in the process of developing a plan for its juvenile justice programs,[Footnote 16] but the office is taking steps to address its girls' delinquency goals, for example, through the technical assistance workshop. Developing a plan for girls' delinquency would help OJJDP to demonstrate leadership to the girls' delinquency field by clearly articulating the actions it intends to take to meet its goals and would also help the office to ensure that the goals are met. In our July report, we recommended that to help ensure that OJJDP meets its goals to identify effective or promising girls' delinquency programs and supports the development of program models, the Administrator of OJJDP develop and document a plan that (1) articulates how the office intends to respond to the findings of the Study Group, (2) includes time frames and specific funding requirements and commitments, and (3) is shared with key stakeholders. OJP agreed with our recommendation and outlined efforts that OJJDP plans to undertake in response to these findings. For example, OJJDP stated that it anticipates publishing its proposed juvenile justice program plan, which is to include how it plans to address girls' delinquency issues, in the Federal Register to solicit public feedback and comments, which will enable the office to publish a final plan in the Federal Register by the end of the year (December 31, 2009). Mr. Chairman, this concludes my statement. I would be pleased to respond to any questions that you or other Members of the Subcommittee may have. Contacts and Acknowledgements: For questions about this statement, please contact Eileen R. Larence at (202) 512-8777 or larencee@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals making key contributions to this statement include Mary Catherine Hult, Assistant Director; Kevin Copping; and Katherine Davis. Additionally, key contributors to our July 2009 report include David Alexander, Elizabeth Blair, and Janet Temko. [End of section] Footnotes: [1] C. Puzzanchera and W. Kang, Juvenile Court Statistics Databook (2008), [hyperlink, http://ojjdp.ncjrs.gov/ojstatbb/jcsdb/] (accessed Oct.15, 2009). This Web site provides the most current data available. [2] C. Puzzanchera, Juvenile Arrests 2007, (2009) www.ncjrs.gov/pdffiles1/ ojjdp/225344.pdf (accessed Oct.15, 2009). [3] Elizabeth Cauffman and others, "Gender Differences in Mental Health Symptoms among Delinquent and Community Youth," Youth Violence and Juvenile Justice, vol. 5, no. 3 (2007): 287-307. Elizabeth Caufmann "Understanding the Female Offender" The Future of Children, vol. 18, no. 2 (2008): 119-142. [4] GAO, Juvenile Justice: Technical Assistance and Better Defined Evaluation Plans Will Help to Improve Girls' Delinquency Programs, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-09-721R] (Washington, D.C.: July 24, 2009). [5] In a September 18, 2009, letter regarding the recommendation we made in our July report, DOJ clarified actions it was taking to address our recommendation, which we have included in this statement. [6] For social science standards for evaluation research, see Donald T. Campbell and Julian Stanley, Experimental and Quasi-Experimental Designs for Research (Chicago: Rand McNally, 1963); William R. Shadish, Thomas D. Cook, and Donald T. Campbell, Experimental and Quasi- Experimental Designs for Generalized Causal Inference (Boston: Houghton Mifflin, 2002); Carol H. Weiss, Evaluation: Methods for Studying Programs and Policies, Second Edition (Englewood Cliffs, N.J.: Prentice- Hall, Inc., 1998); and GAO, Designing Evaluations, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/PEMD-10.1.4] (Washington, D.C.: May 1991). [7] GAO, Standards for Internal Control in the Federal Government, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?/GAO/AIMD-00-21.3.1] (Washington, D.C.: November 1999). [8] GAO defines an expert as a person who is recognized by others who work in the same subject matter area as having knowledge that is greater in scope or depth than that of most people working in that area. The expert's knowledge can come from education, experience, or both. We identified researchers who focus on girls' delinquency issues and practitioners who operate programs that address girls' delinquency. Specifically, these 18 experts included 11 of the 15 study group members and 7 experts who were not members of the group. While we contacted all 15 of the study group members, 4 members either did not respond to requests for interviews or declined to be interviewed. [9] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-09-721R]. [10] Cooperative agreements, rather than grant awards, can be used by federal agencies when substantial involvement is expected between the agency and the recipient when carrying out the activities described in the program announcement. OJJDP extended the cooperative agreement with the research institute through June 2010 to complete all of the Study Group activities. [11] The What Works criteria define six levels of effectiveness, including effective, promising, and ineffective, for use in assessing and classifying studies on the basis of their evidence of effectiveness. The criteria for an effective program include a randomized controlled research design--a design that compares the outcomes for individuals who are randomly assigned to either the program being studied or to a nonparticipating control group before the intervention. While other research designs can produce valid results, we have previously reported that when it is feasible and ethical to do so, randomized controlled designs provide researchers with the best method for assessing a program's effectiveness because they isolate changes caused by the program from other factors. [12] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-00-21.3.1]. [13] The other seven experts did not express views regarding the balance of the study group's composition. [14] Project Management Institute, The Standard for Program Management. [15] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-00-21.3.1]. [16] OJJDP is required under the Juvenile Justice and Delinquency Prevention Act to publish an annual program plan that will, among other things, lay out goals and criteria for conducting research and evaluation for its juvenile justice programs. 42 U.S.C. § 5614(b)(5). This plan is required to be published annually in the Federal Register for public comment, and is to describe the activities the Administrator intends to carry out under Parts D and E. Under Part D, OJJDP is authorized to conduct research, evaluation, and technical assistance, among other things. 42 U.S.C. §§ 5661-62. Under Part E, OJJDP is authorized to make grants for developing, testing and demonstrating promising new initiatives and programs. 42 U.S.C. §§ 5665-66. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO‘s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO‘s Web site, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: [End of section]

The Justia Government Accountability Office site republishes public reports retrieved from the U.S. GAO These reports should not be considered official, and do not necessarily reflect the views of Justia.