Social Security Disability

Management Controls Needed to Strengthen Demonstration Projects Gao ID: GAO-08-1053 September 26, 2008

Since 1980, Congress has required the Social Security Administration (SSA) to conduct demonstration projects to test the effectiveness of possible changes to its Social Security Disability Insurance (DI) and Supplemental Security Income (SSI) programs that could decrease individuals' dependence on benefits or improve program administration. However, in 2004, GAO reported that SSA had not used its demonstration authority effectively. This follow-up report assesses (1) how SSA has used its demonstration authority to test DI and SSI program changes and what information these efforts have yielded and (2) what steps SSA has taken to improve the planning and management of its demonstration projects. To do this, GAO reviewed documents related to SSA's demonstration project management and the steps it took to implement the recommendations in the 2004 report, as well as the projects' designs, evaluations, and costs. GAO also interviewed officials from SSA, its contractors and project sites, and disability experts.

Over the last decade, SSA has initiated 14 demonstration projects under its authority to test possible DI and SSI policy and program changes; however, these projects have yielded limited information for influencing program and policy decisions. Of the 14 projects, SSA has completed 4, cancelled 5, and had 5 projects in progress as of June 2008. In total, SSA spent about $155 million on its projects as of April 2008, and officials anticipate spending another $220 million in the coming years on those projects currently under way. Yet, these projects have yielded limited information on the impacts of the program and policy changes they were testing. SSA did not conduct impact evaluations for two of its completed projects, and intended to evaluate five other projects, but could not do so because significant challenges led SSA to cancel them. SSA officials believe the five projects currently under way will yield useful information, but it is too early to tell. SSA has taken steps to improve its demonstration projects but continues to lack management controls to ensure that the projects yield reliable information for making disability policy decisions. SSA has used methodological designs that GAO determined were strong or reasonable when assessed against professional research standards for 11 of its 14 projects. SSA has also used external research professionals to work with the agency on the design, implementation, or evaluation of 12 of the projects, and appointed new program management to oversee its demonstration program. However, as of August 2008, SSA had not fully implemented the recommendations GAO made in 2004 and did not have written policies and procedures governing how it should review and operate its demonstration project program. Specifically, SSA does not have written policies and procedures for its managers and project officers to follow as they design, implement, and evaluate its demonstration projects. Absent such protocols, SSA did not always apply standard research practices, such as conducting pilot phases or obtaining sufficient stakeholder input, which led to data limitations and project cancellations.

Recommendations

Our recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.

Director: Team: Phone:


GAO-08-1053, Social Security Disability: Management Controls Needed to Strengthen Demonstration Projects This is the accessible text file for GAO report number GAO-08-1053 entitled 'Social Security Disability: Management Controls Needed to Strengthen Demonstration Projects' which was released on September 26, 2008. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to Congressional Requesters: United States Government Accountability Office: GAO: September 2008: Social Security Disability: Management Controls Needed to Strengthen Demonstration Projects: Social Security Disability: GAO-08-1053: GAO Highlights: Highlights of GAO-08-1053, a report to congressional requesters. Why GAO Did This Study: Since 1980, Congress has required the Social Security Administration (SSA) to conduct demonstration projects to test the effectiveness of possible changes to its Social Security Disability Insurance (DI) and Supplemental Security Income (SSI) programs that could decrease individuals‘ dependence on benefits or improve program administration. However, in 2004, GAO reported that SSA had not used its demonstration authority effectively. This follow-up report assesses (1) how SSA has used its demonstration authority to test DI and SSI program changes and what information these efforts have yielded and (2) what steps SSA has taken to improve the planning and management of its demonstration projects. To do this, GAO reviewed documents related to SSA‘s demonstration project management and the steps it took to implement the recommendations in the 2004 report, as well as the projects‘ designs, evaluations, and costs. GAO also interviewed officials from SSA, its contractors and project sites, and disability experts What GAO Found: Over the last decade, SSA has initiated 14 demonstration projects under its authority to test possible DI and SSI policy and program changes; however, these projects have yielded limited information for influencing program and policy decisions. Of the 14 projects, SSA has completed 4, cancelled 5, and had 5 projects in progress as of June 2008. In total, SSA spent about $155 million on its projects as of April 2008, and officials anticipate spending another $220 million in the coming years on those projects currently under way. Yet, these projects have yielded limited information on the impacts of the program and policy changes they were testing. SSA did not conduct impact evaluations for two of its completed projects, and intended to evaluate five other projects, but could not do so because significant challenges led SSA to cancel them. SSA officials believe the five projects currently under way will yield useful information, but it is too early to tell. Table: Demonstration Project Costs and Information Yielded Since 1998: DI (trust fund); Funds spent (1998 to April 2008): $48.3 million; Projected expenditures (April 2008 forward): $174 million. SSI (appropriations); Funds spent (1998 to April 2008): $107.2 million; Projected expenditures (April 2008 forward): $45.6 million. Total; Funds spent (1998 to April 2008): $155.5 million; Projected expenditures (April 2008 forward): $219.6 million. Information yielded; Funds spent (1998 to April 2008): Impact Information–2 of 14; Other information–3 of 14; No information–6 of 14; Projected expenditures (April 2008 forward): Information not yet available–4 of 14. Source: GAO analysis of SSA deminstration project estimated project cost and evaluation results as of April 2008. Note: Projects sum to greater than 14 because one project yielded impact information and other information. [End of table] SSA has taken steps to improve its demonstration projects but continues to lack management controls to ensure that the projects yield reliable information for making disability policy decisions. SSA has used methodological designs that GAO determined were strong or reasonable when assessed against professional research standards for 11 of its 14 projects. SSA has also used external research professionals to work with the agency on the design, implementation, or evaluation of 12 of the projects, and appointed new program management to oversee its demonstration program. However, as of August 2008, SSA had not fully implemented the recommendations GAO made in 2004 and did not have written policies and procedures governing how it should review and operate its demonstration project program. Specifically, SSA does not have written policies and procedures for its managers and project officers to follow as they design, implement, and evaluate its demonstration projects. Absent such protocols, SSA did not always apply standard research practices, such as conducting pilot phases or obtaining sufficient stakeholder input, which led to data limitations and project cancellations. What GAO Recommends: GAO recommends that SSA establish written policies and procedures for managing and operating its projects consistent with standard research practices and internal control standards in the federal government. In response, SSA generally agreed with GAO‘s recommendation and acknowledged that its guidance is a work in progress. To view the full product, including the scope and methodology, click on [http://www.gao.gov/cgi-bin/getrpt?GAO-08-1053]. For more information, contact Daniel Bertoni at (202) 512-7215 or bertonid@gao.gov. [End of section] Contents: Letter: Results in Brief: Background: SSA Has Used Its Demonstration Authority to Initiate Several Projects Since 1998, but Efforts Have Yielded Limited Information: SSA Has Taken Steps to Improve Its Projects but Continues to Lack Management Controls: Conclusions: Recommendation for Executive Action: Agency Comments and Our Evaluation: Appendix I: Scope and Methodology: Appendix II: Additional Statutory Criteria for the Benefit Offset National Demonstration Project: Appendix III: SSA's Use of Research Contractors and Universities on the Demonstration Projects: Appendix IV: Comments from the Social Security Administration: Appendix V: GAO Contact and Staff Acknowledgments: Related GAO Products: Tables: Table 1: Overview of Demonstration Projects Initiated Since 1998: Table 2: Total Estimated Costs for Demonstration Projects Currently in Progress: Table 3: Assessment of Demonstration Projects' Designs against Professional Research Standards and Statutory Requirements: Table 4: Reasons for Demonstration Projects' Cancellations: Table 5: Additional Statutory Criteria for the Benefit Offset National Demonstration Project: Figure: Figure 1: Projects Experienced Schedule Delays and Cancellations Associated with Organizational Changes: Abbreviations: BOND: Benefit Offset National Demonstration: DI: Social Security Disability Insurance: DISP: Disability and Income Security Program: DOL: Department of Labor: FICA: Federal Insurance Contributions Act: FMFIA: Federal Managers' Financial Integrity Act of 1982: HOPE: Homeless Projects Outreach and Evaluation: OPDR: Office of Program Development and Research: PMU: Pediatric Medical Unit: SSA: Social Security Administration: SSI: Supplemental Security Income: WIPA: Work Incentives Planning and Assistance: United States Government Accountability Office: Washington, DC 20548: September 26, 2008: The Honorable Michael R. McNulty: Chairman: The Honorable Sam Johnson: Ranking Member: Subcommittee on Social Security: Committee on Ways and Means: House of Representatives: As of October 2007, the Social Security Administration (SSA) reported that nearly 8.9 million disabled workers and their dependents were receiving benefits under the Social Security Disability Insurance (DI) program, while approximately 7.4 million individuals received federally administered Supplemental Security Income (SSI) payments. In our prior work, we have raised concerns about significant growth in the disability rolls and challenges in devising return-to-work policies. To address these challenges and encourage individuals capable of working to return to work, Congress has required SSA to conduct demonstration projects to test the effectiveness of possible program changes that could decrease individuals' dependency on benefits. Over the last decade, SSA has spent about $155 million dollars on these projects and anticipates that it will spend another $220 million on those currently under way. To conduct these demonstrations, Congress authorized SSA to waive certain program rules temporarily and use trust fund dollars and appropriated funds to finance their development. Congress also required that DI demonstration projects be of sufficient scope and conducted on a wide enough scale to ensure a thorough evaluation of the program or policy change under consideration. However, in 2004, we reported that SSA had not used its DI demonstration authority effectively and that the demonstration projects conducted in the late 1980s and early 1990s had little effect on SSA's and Congress's consideration of disability policy issues.[Footnote 1] We have also placed federal disability programs on our high-risk list--a list of federal programs that need attention and transformation. We did so, in part because of concerns about significant growth in the disability rolls and challenges in devising return-to-work policies.[Footnote 2] As you requested, we reviewed the adequacy of SSA's current and planned demonstration projects for both DI and SSI programs. Specifically, this report assesses (1) how SSA has used its demonstration authority to test DI and SSI program changes and what information these efforts have yielded and (2) what steps SSA has taken to improve the planning and management of its demonstration projects. To complete our work, we reviewed the demonstration projects that SSA planned, initiated, or terminated between the calendar years 1998 and 2008 in order to identify their key characteristics, including their purpose, current status, and costs. To gain a better understanding of the components of SSA's demonstration projects, we conducted interviews with staff at sites where SSA implemented 9 of the 14 demonstration projects. We selected sites that included ongoing, cancelled, and completed projects, and represented diverse geographic regions throughout the United States. To determine what information these projects have yielded about the impacts of the policies and programs being tested and lessons learned, we collected and reviewed available evaluation information on each project. We collected data from SSA on the funds spent to date, projected future expenditures, and total costs for the demonstration projects, and determined that these data were sufficiently reliable for the purposes of this report. We also reviewed documents related to SSA's planning, implementation, and evaluation of its demonstration projects, including the development of the projects' methodological designs. We reviewed the projects' methodological designs against professional research standards for demonstration projects. These research standards were derived from GAO and recognized academic criteria for conducting evaluation research. Our Office of General Counsel reviewed these standards and found them consistent with the authorizing statutes' methodological requirements.[Footnote 3] Further, we identified relevant federal statutes governing DI and SSI demonstration activities and examined other reviews of SSA demonstration projects that we and SSA's Office of the Inspector General had conducted. For both objectives, we interviewed SSA officials, project officers, and research contractors that worked on the demonstration projects, officials from SSA's Office of Inspector General, and individuals from organizations that have a key role in federal disability policy. To address SSA's planning and management of its demonstration projects, we interviewed SSA management and staff about the agency's policies, guidance and procedures for developing and implementing demonstration projects, and collected supporting documentation where available. We assessed the adequacy of SSA's internal controls using the criteria in GAO's Standards for Internal Control in the Federal Government, which provide the overall framework for establishing and maintaining internal control in the federal government.[Footnote 4] We conducted this performance audit from November 2007 to August 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix I discusses our scope and methodology in further detail. Results in Brief: Over the last decade, SSA has initiated 14 projects under its demonstration authority to test both DI and SSI program changes--6 related to DI, 6 related to SSI, and 2 examining both programs jointly- -however, these projects have yielded limited information to date. Of the 14 projects, SSA has completed 4, cancelled 5, and had 5 projects in progress as of August 2008. In general, SSA has used its demonstration authority to conduct projects to consider changes to program administration and benefits counseling, as well as to test options for providing employment assistance and health benefits for beneficiaries. In total, SSA had spent about $155 million on these demonstration projects as of April 2008, and officials anticipate spending another $220 million in the coming years on those projects currently in progress. Yet, SSA's demonstration projects have yielded limited information about the impacts of the policies and programs being tested. We found that SSA did not conduct impact evaluations-- assessments of a project's effects compared to what would have happened in its absence--for two of its completed projects because the data it planned to use was not available. In addition, SSA intended to evaluate the impacts of five other projects but could not because significant challenges led it to cancel them in the early stages, and thus no data was available to assess those policies and programs being tested. Nonetheless, SSA was able to obtain some preliminary information from one of its cancelled projects. SSA also obtained some information from the two completed projects it evaluated, but the evaluations' findings had limitations. In one case SSA could not separate the effects of its project from the effects of a similar federal project that was also under way. SSA has also obtained some preliminary information from one of the five projects currently under way, and agency officials believe that these projects will ultimately yield useful information. However, it is too early to tell whether the information obtained will be used to inform changes to disability program policy. These projects are expected to continue until 2010 or later. SSA has taken steps to improve its demonstration projects but continues to lack written management controls to ensure that the projects yield reliable information about their impacts. For 11 of the projects, SSA has used experimental or quasi-experimental designs--methods commonly used by research professionals for demonstration projects--that we determined were strong or reasonable when assessed against professional research standards. SSA has also used external research professionals to work with the agency on the design, implementation, or evaluation of 12 of the 14 projects, and appointed new management in 2007 to oversee the demonstration project program. The new management team subsequently conducted an internal review of the 10 demonstration projects that were under way at the time of their appointment and cancelled 5 that they determined were unlikely to yield reliable information because they faced significant limitations or challenges, or would have been duplicative of other ongoing research. SSA's decision to cancel these projects appears to have been data-driven and reasonable. However, as of August 2008, SSA had not fully implemented the recommendations we made in 2004 to help ensure the effectiveness of the demonstration projects. Further, the agency does not have written policies and procedures governing how it should review and operate its demonstration program. Specifically, SSA does not have written policies and procedures for its managers and project officers to follow as they design, implement, and evaluate its demonstration projects. Absent such protocols, SSA sometimes did not apply standard research practices, such as conducting pilot phases or obtaining sufficient stakeholder input, which led to data limitations and project cancellations. We are making a recommendation to SSA that is intended to improve the management of its demonstration project program by establishing written policies, procedures, and mechanisms for managing and operating the demonstration projects. These policies and procedures should be consistent with standard research practices and internal control standards in the federal government. In response to our draft report, SSA generally agreed with our recommendation and the need to develop a guidebook to assist its staff in the design, implementation, and evaluation phases of its demonstration projects. Furthermore, SSA considers its current guidebook a work in progress. SSA also stated that the agency has taken steps in recent years to address our prior recommendations. While we acknowledge SSA's efforts, we maintain that additional steps are needed to fully implement them. SSA's comments are reproduced in appendix IV. Background: Under the Social Security Act of 1935, as amended, SSA administers two federal disability programs--DI and SSI--intended to provide benefits to individuals with disabilities who are unable to work. The DI insurance program provides monthly cash benefits to individuals who have a Social Security work record and the amount of benefits is related to prior earnings.[Footnote 5] The DI program is funded primarily through a payroll tax required by the Federal Insurance Contributions Act (FICA) and is levied on most workers, and DI benefits are based on an individual's career earnings. The SSI program is a means-tested entitlement program that provides monthly benefits to aged, blind, or disabled individuals who have very limited income and assets. The SSI program is funded through general revenues. Unlike the DI benefit, the federal SSI benefit is a flat amount (adjusted for other income the individual may have) and is not related to prior earnings. During the 1970s, as the number of disability awards and costs were increasing significantly for the DI program, Congress enacted legislation providing various work incentives to encourage beneficiaries to return to work and, potentially, leave the benefit rolls. To further these efforts, in 1980, Congress provided SSA with the authority to conduct demonstration projects to evaluate the effectiveness of policy alternatives that could encourage both DI and SSI beneficiaries to re-enter the workforce.[Footnote 6] Under this authority, SSA can temporarily waive DI and SSI program rules, including rules regarding program eligibility and benefit administration, in order to test the effect certain program changes would have on beneficiaries' return-to-work rates and the size of the DI and SSI benefit rolls. Because Congress has historically granted SSA DI demonstration authority on a temporary basis, it is subject to periodic review and renewal. Since first providing this authority in 1980, Congress has renewed it in 1986, 1989, 1994, 1999, and 2004. However, in 2004, Congress only extended SSA's DI demonstration authority through December 2005. As a result, SSA cannot initiate new DI demonstration projects but can continue those projects that were initiated on or before the December 2005 expiration date. However, SSA can continue to initiate demonstration projects under its SSI authority.[Footnote 7] In 2008, SSA requested that Congress reauthorize its DI demonstration authority, and a bill was introduced to do so. SSA's DI demonstration projects--unlike other SSA research activities- -are paid for via the DI trust fund. Therefore, SSA is not required to obtain congressional approval for DI demonstration expenditures, although it is required to obtain approval from the Office of Management and Budget for an annual apportionment of the trust funds for these demonstrations. Unlike the DI projects, SSI demonstration projects are funded from SSA's overall congressional research appropriation. Although SSA's DI and SSI demonstration authorities are separate, the agency's disability demonstration projects are sometimes jointly authorized when they involve both DI and SSI beneficiaries and applicants. When a demonstration project is conducted jointly under the DI and SSI demonstration authorities, funding for the project is split between trust fund (i.e., DI) and appropriated (i.e., SSI) sources. SSA's Office of Program Development and Research (OPDR) provides program analysis in support of DI and SSI programs. As part of their responsibilities, OPDR--sometimes with the assistance of outside research organizations--identifies the requirements for individual disability program demonstration projects, including the basic objectives, scope, and methodological standards for these projects. OPDR project officers are primarily responsible for overseeing the projects to ensure that they meet SSA's technical and programmatic requirements. As we have previously reported, demonstration projects examining the impact of social programs aim to provide evidence of the feasibility or effectiveness of a new approach or practice and are inherently complex and difficult to conduct.[Footnote 8] Measuring outcomes, ensuring the consistency and quality of data collected at various site locations, establishing a causal connection between outcomes and program activities, and separating out the influence of extraneous factors can raise formidable technical and logistical problems. Although the legislation granting SSA its demonstration authority does not require the use of particular methodological approaches, SSA has historically recognized that the law's general requirement for its demonstration projects requires SSA to conduct its projects in a rigorous manner that provides the agency with a reliable basis for making policy recommendations. According to professional research standards, a rigorous study should include a clearly stated research question and methodology, including plans for data collection and evaluation, as well as appropriate controls to determine if a relationship exists between observed outcomes and the program change under examination (see app. I). As part of our prior work related to SSA DI demonstration authority we reviewed two DI demonstration projects that SSA conducted in the late 1980s and early 1990s.[Footnote 9] At that time, we found that SSA had not used its demonstration authority to extensively evaluate a wide range of DI policy areas dealing with return to work and found that the demonstration projects had little impact on SSA's and Congress's consideration of DI policy issues. To facilitate close congressional oversight and provide greater assurance that SSA will make effective use of its authority, we recommended that SSA develop a formal agenda for its demonstration projects, establish an expert panel to guide the design and implementation of its demonstration projects, and establish formal processes to ensure full consideration of demonstration project results. We also identified several matters for Congress to consider, including continuation of DI demonstration authority on a temporary basis, establishment of additional reporting requirements for demonstrations, and clearer specification of the methodological and evaluation requirements of demonstrations. SSA Has Used Its Demonstration Authority to Initiate Several Projects Since 1998, but Efforts Have Yielded Limited Information: Over the last decade, SSA has initiated 14 demonstration projects to test policy and program changes, of which SSA has completed 4, cancelled 5, and had 5 projects in progress as of August 2008. In total, SSA had spent about $155 million on these demonstration projects as of April 2008, and officials anticipate spending another $220 million in the coming years on those projects currently in progress. However, these projects have yielded limited information for influencing program and policy decisions. We found that SSA did not conduct impact evaluations for two of its completed projects and cancelled five projects prior to conducting formal evaluations; thus, limited information is available. Since 1998, SSA Has Initiated 14 DI and SSI Demonstration Projects in an Effort to Test Policy and Administrative Changes: Since 1998, SSA has initiated 14 projects under its demonstration authority to test both DI and SSI program changes--6 related to DI, 6 related to SSI, and 2 examining both programs jointly (see table 1 for an overview of each project). As of April 2008, SSA spent $80.3 million on its completed projects, $7.1 million on cancelled projects, and $68.2 million on those currently in progress. Table 1: Overview of Demonstration Projects Initiated Since 1998: Project: Projects conducted under DI demonstration authority: Accelerated Benefits; Policy issues studied: Projects conducted under DI demonstration authority: Focused on provision of immediate health benefits to newly entitled DI beneficiaries to test whether this results in improved health and better return to work outcomes; Status: Projects conducted under DI demonstration authority: Initiated 2004, in progress; Funds spent (as of April 2008): Projects conducted under DI demonstration authority: $3.6 million from the DI trust fund; Information available: Projects conducted under DI demonstration authority: Not yet available; Completion expected 2011. Project: Projects conducted under DI demonstration authority: Benefit Offset - 4 State Pilot; Policy issues studied: Projects conducted under DI demonstration authority: Focused on assessment of pilot of a benefit offset to inform the design of the Benefit Offset National Demonstration (BOND); Status: Projects conducted under DI demonstration authority: Initiated 2003, in progress; Funds spent (as of April 2008): Projects conducted under DI demonstration authority: $4.5 million from the DI trust fund; Information available: Projects conducted under DI demonstration authority: Yes; preliminary process and outcome information available showing some increase in earnings by pilot participants and identifying operational challenges, such as income reporting, encountered when implementing the benefit offset; Completion expected 2009. Project: Projects conducted under DI demonstration authority: Benefit Offset National Demonstration (BOND); Policy issues studied: Projects conducted under DI demonstration authority: Focused on assessment of a benefit offset that would allow beneficiaries to remain eligible for DI benefits when earning above the Substantial Gainful Activity level while offsetting these benefits by $1 for every $2 in earnings; it would also assess benefit counseling in conjunction with the offset; Status: Projects conducted under DI demonstration authority: Initiated 1999, in progress; Funds spent (as of April 2008): Projects conducted under DI demonstration authority: $6.7 million from the DI trust fund; Information available: Projects conducted under DI demonstration authority: Not yet available; Completion date unknown[A]. Project: Projects conducted under DI demonstration authority: California RISE (CA HIV/AI) Demonstration; Policy issues studied: Projects conducted under DI demonstration authority: Focused on provision of additional health and employment services for DI beneficiaries with Human Immunodeficiency Virus (HIV) or Auto-Immune (AI) deficiencies; Status: Projects conducted under DI demonstration authority: Initiated 2003, cancelled 2007; Funds spent (as of April 2008): Projects conducted under DI demonstration authority: $2.1 million from the DI trust fund; Information available: Projects conducted under DI demonstration authority: No; project was not implemented. Project: Projects conducted under DI demonstration authority: Early Intervention Demonstration; Policy issues studied: Projects conducted under DI demonstration authority: Focused on provision of employment services to DI applicants as an alternative to the DI program; Status: Projects conducted under DI demonstration authority: Initiated 2000, cancelled 2007; Funds spent (as of April 2008): Projects conducted under DI demonstration authority: $2 million from the DI trust fund; Information available: Projects conducted under DI demonstration authority: No; project was not implemented. Project: Projects conducted under DI demonstration authority: Mental Health Treatment Study; Policy issues studied: Projects conducted under DI demonstration authority: Focused on provision of mental health and employment services to DI beneficiaries with a primary impairment of schizophrenia or affective disorder; Status: Projects conducted under DI demonstration authority: Initiated 2003, in progress; Funds spent (as of April 2008): Projects conducted under DI demonstration authority: $17.2 million from the DI trust fund; Information available: Projects conducted under DI demonstration authority: Not yet available; Completion expected 2011. Project: Projects conducted under SSI demonstration authority: Disability Program Navigator,; conducted with the Department of Labor (DOL); Policy issues studied: Projects conducted under DI demonstration authority: Focused on expansion of capacity in workforce investment system to serve people with disabilities; Status: Projects conducted under DI demonstration authority: Initiated 2002, completed 2007; Funds spent (as of April 2008): Projects conducted under DI demonstration authority: $12 million from appropriations; Information available: Projects conducted under DI demonstration authority: No; project was not evaluated; SSA officials told us that DOL is pursuing a separate evaluation. Project: Projects conducted under SSI demonstration authority: Early Identification and Intervention Demonstration; Policy issues studied: Projects conducted under DI demonstration authority: Focused on provision of funding for programs that increase developmental screening and early identification of children with developmental delays and/or disabilities; Status: Projects conducted under DI demonstration authority: Initiated 2004, cancelled 2007; Funds spent (as of April 2008): Projects conducted under DI demonstration authority: $50,000 from appropriations; Information available: Projects conducted under DI demonstration authority: No; project was not implemented. Project: Projects conducted under SSI demonstration authority: Florida Freedom Initiative; Policy issues studied: Projects conducted under DI demonstration authority: Focused on assessment of waivers of SSI program rules to foster greater self-sufficiency among SSI recipients in the state of Florida; Status: Projects conducted under DI demonstration authority: Initiated 2003, completed 2007; Funds spent (as of April 2008): Projects conducted under DI demonstration authority: $100,000 from appropriations; Information available: Projects conducted under DI demonstration authority: No; project was not evaluated because too few participants enrolled to produce enough data for the evaluation. Project: Projects conducted under SSI demonstration authority: Homeless Outreach Projects and Evaluation; Policy issues studied: Projects conducted under DI demonstration authority: Focused on provision of services to people with disabilities who are chronically homeless, including assistance with filing disability benefit applications and accessing mainstream treatment and services; Status: Projects conducted under DI demonstration authority: Initiated 2003, completed 2008; Funds spent (as of April 2008): Projects conducted under DI demonstration authority: $21.4 million from appropriations; Information available: Projects conducted under DI demonstration authority: Yes; impact information available showing that project participants received faster decisions from SSA about whether to allow/deny benefits; however, the contractor identified limitations with its evaluation because a similar program that another federal agency initiated after the SSA project began could have influenced the impact on the Homeless Outreach Projects and Evaluation participants. Project: Projects conducted under SSI demonstration authority: Interim Medical Benefits; Policy issues studied: Projects conducted under DI demonstration authority: Focused on provision of medical benefits to child and adult SSI applicants who are likely to be awarded benefits following the disability determination process; Status: Projects conducted under DI demonstration authority: Initiated 2004, cancelled 2007; Funds spent (as of April 2008): Projects conducted under DI demonstration authority: $0; Information available: Projects conducted under DI demonstration authority: No; project was not implemented. Project: Projects conducted under SSI demonstration authority: Pediatric Medical Unit (PMU); Policy issues studied: Projects conducted under DI demonstration authority: Focused on provision of pediatric expertise to disability adjudicators to assist in making determinations on childhood disability cases; Status: Projects conducted under DI demonstration authority: Initiated 2004, cancelled 2008; Funds spent (as of April 2008): Projects conducted under DI demonstration authority: $2.9 million from appropriations; Information available: Projects conducted under DI demonstration authority: Yes; some information from implementation experiences identified potential problems with childhood disability applications; PMU information could inform future research on interdisciplinary approaches to disability determination. Project: Projects jointly authorized under DI and SSI demonstration authorities: State Partnership Initiative; Policy issues studied: Projects conducted under DI demonstration authority: Focused on assessment of waivers of SSI benefit rules and other employment support interventions for DI and SSI beneficiaries provided in 12 states; Status: Projects conducted under DI demonstration authority: Initiated 1998, completed 2006; Funds spent (as of April 2008): Projects conducted under DI demonstration authority: $46.7 million,; $8.5 million from the DI trust fund and $38.2 million from appropriations; Information available: Projects conducted under DI demonstration authority: Yes; some information from implementation experiences, as well as impact information showing that benefits counseling and employment services sometimes increased employment, but reduced earnings or did not affect them. However, the contractor identified some limitations with its impact evaluation because it is based on preliminary data. Project: Projects jointly authorized under DI and SSI demonstration authorities: Youth Transition Demonstration; Policy issues studied: Projects conducted under DI demonstration authority: Focused on provision of services to help youth SSI beneficiaries and other youth with disabilities make the transition from school to work and/or continue their education; it would also assess various waivers of disability program rules; Status: Projects conducted under DI demonstration authority: Initiated 2003, in progress; Funds spent (as of April 2008): Projects conducted under DI demonstration authority: $36.2 million,; $3.7 million from the DI Trust fund and $32.5 million from appropriations; Information available: Projects conducted under DI demonstration authority: Not yet available; Completion expected 2014. Project: Total; Policy issues studied: Projects conducted under DI demonstration authority: [Empty]; Status: Projects conducted under DI demonstration authority: 4 completed; 5 cancelled; 5 in progress; Funds spent (as of April 2008): Projects conducted under DI demonstration authority: $155.5 million; ($48.3 million from DI trust fund, $107.2 million from appropriations); Information available: Projects conducted under DI demonstration authority: Impact information-2 of 14; Other information-3 of 14; No information-6 of 14; Information not yet available-4 of 14. Source: GAO analysis of SSA's demonstration projects and budget data. Notes: In the column Information Available, projects sum to greater than 14 because 1 project yielded impact information and other information. In some cases, SSA officials provided us with estimated initiation dates because the agency had not maintained complete documentation on all demonstrations projects initiated since 1998. For the State Partnership Initiative, SSA officials were unable to verify the actual budget amounts spent in 1998 and 1999 for this project because SSA no longer maintains this information. [A] As of August 2008, SSA had not determined an expected completion date for the BOND project. [End of table] While SSA initiated 14 projects over the past 10 years, the agency has only completed 4 of them to date. These completed projects generally focused on reducing individuals' dependency on the SSI program by primarily testing program waivers and other changes in program administration, as outlined in its SSI demonstration authority.[Footnote 10] We also found that SSA cancelled five projects during this period, citing significant challenges that would have limited the agency's ability to obtain reliable information from them. SSA had five projects in progress as of August 2008. These projects generally addressed topics outlined in the authorizing legislation for DI demonstrations and included strategies to return individuals to work and reduce the growth of certain subgroups of beneficiaries. For example, the legislation required projects to test various incentives to increase DI beneficiaries work activity.[Footnote 11] In addition, the Ticket to Work and Work Incentives Improvement Act of 1999 provided demonstration authority for a benefit reduction, rather than complete benefit termination, when beneficiaries had earnings that exceeded a certain level.[Footnote 12] To address this provision, SSA initiated the Benefit Offset National Demonstration (BOND) shortly after passage of the statute. Another project in progress, the Mental Health Treatment Study, is focused on identifying strategies for providing mental health treatment and employment supports for certain DI beneficiaries with mental illnesses.[Footnote 13] As of April 2008, officials estimated that the total costs for the five projects currently in progress would be about $288 million--about $220 million more than the $68 million already expended (see table 2). Table 2: Total Estimated Costs for Demonstration Projects Currently in Progress: Project: Accelerated Benefits; Year of initiation: 2004; Completion date: 2011; Total estimated cost: $45 million. Project: Benefit Offset - 4 State Pilot; Year of initiation: 2005; Completion date: 2009; Total estimated cost: $8.5 million. Project: Benefit Offset National Demonstration (BOND); Year of initiation: 1999; Completion date: unknown[A]; Total estimated cost: $95 million. Project: Mental Health Treatment Study; Year of initiation: 2003; Completion date: 2011; Total estimated cost: $52.5 million. Project: Youth Transition Demonstration; Year of initiation: 2003; Completion date: 2014; Total estimated cost: $86.7 million; ($78 million from appropriations and $8.7 million from DI trust fund). Project: Total; Year of initiation: [Empty]; Completion date: [Empty]; Total estimated cost: $287.7 million; ($209.7 million from DI trust fund, $78 million from appropriations). Source: GAO analysis of SSA's demonstration projects and budget data. Notes: Expected total cost figures provided by SSA are estimates. [A] As of August 2008, SSA has not determined an expected completion date for the BOND project. [End of table] Demonstration Projects Have Yielded Limited Information to Influence Program and Policy Changes: Despite using its demonstration authority to examine various issues, SSA's demonstration projects have yielded limited information for influencing program and policy decisions. As required under its demonstration authority, SSA's demonstration projects should be conducted in such a way to permit a thorough evaluation of alternative methods under consideration.[Footnote 14] However, we found that SSA had not conducted impact evaluations--assessments of a project's effects compared to what would have happened in its absence--for two of its completed projects, the Disability Program Navigator and the Florida Freedom Initiative. Thus, no information about the impacts of the program and policy changes being tested was available for making decisions about disability policy. The Disability Program Navigator project, which SSA conducted with the Department of Labor (DOL), was not evaluated because the evaluation contractor could not meet SSA's data security requirements established after the project was already in progress, and thus could not access the necessary data.[Footnote 15] SSA developed a plan to evaluate the Florida Freedom Initiative after they became concerned about the state's evaluation plans. However, SSA did not conduct an evaluation because staff at the state level conducting the project did not enroll enough participants in the project to meet sample size requirements. Thus, there was not enough data available to conduct a reliable evaluation. Furthermore, SSA intended to evaluate the impacts of policies and programs being tested in five other projects but could not do so because the significant challenges those projects faced led SSA to cancel them in the early stages. Specifically, four of these projects were cancelled prior to implementation, and thus no data was available to conduct the evaluations of those policy and programs being tested. The other cancelled project--the Pediatric Medical Unit demonstration- -was partially implemented but not evaluated because the project did not establish the comparison group needed for the analysis. The project also did not it enroll enough participants at some implementation sites to meet the sample size requirements needed to generate data for a reliable evaluation. However, SSA was able to obtain some preliminary information on how the project's strategy appeared to be working at two site locations and is considering how to use it. Although SSA did conduct evaluations for two of the completed projects- -the Homeless Outreach Projects and Evaluation (HOPE) project and the State Partnership Initiative project--we found that these projects also yielded little information about the impacts of the strategies being tested because the reported evaluation results could not reliably demonstrate the projects' effects. For example, an outcome evaluation of the HOPE project showed that although disability program applicants assisted by the project received faster decisions from SSA about whether to allow or deny benefits, another federal agency initiated a similar project even though the HOPE project was under way. Therefore, SSA's evaluation results were weakened, in part because researchers could not separate the effects of the SSA project from the effects of the other federal project. While SSA did not obtain reliable impact evaluation results from this project, agency officials told us that they did obtain a great deal of information about the process of conducting this type of demonstration project. For the State Partnership Initiative, we found that SSA did conduct an impact evaluation when the project ended, but data available at that time were incomplete, and thus information about the impact of the project may not be a reliable indicator of the project's long-term effects. SSA's contractors recommended that a final evaluation be conducted once all the data were collected to assess whether the preliminary results were valid. However, SSA management chose not to pursue further evaluation because the preliminary results indicated that the project was not successful at increasing earnings enough to allow individuals returning to work to exit the rolls and no longer be dependent on disability benefits. Nonetheless, SSA's contractors and agency officials said that lessons learned from implementing the State Partnership Initiative have influenced the agency's subsequent approach to return beneficiaries to work. For example, SSA used the job descriptions of benefits planners, as well as data systems from this project, to design the agency's national Benefits Planning and Outreach program.[Footnote 16] SSA has also begun to obtain some information from one of the five projects currently under way. SSA has used preliminary results of the Benefit Offset - 4 State Pilot to aid in the design of the BOND project. Each of the four states conducting this pilot has provided an interim report to SSA detailing lessons learned from the implementation of this project. Because the pilot and BOND both test a benefit offset in conjunction with other DI program changes, SSA officials and the BOND project contractor believe that states' experiences implementing this pilot will help SSA identify and resolve operational issues before rolling BOND out nationally. In addition, the four states have conducted preliminary impact evaluations for the pilot project and expect to complete final evaluations once the project's implementation and data collection phases are over. SSA also plans to conduct impact evaluations of the other demonstration projects it had in progress as of August 2008. While they have the potential to yield reliable results, it is too early to tell whether they will ultimately be useful for informing DI and SSI policy and program changes. These projects address issues outlined in the demonstration authority statutes and disability programs more broadly, and SSA officials believe they will yield useful information. For example, SSA officials anticipate that the results of the Accelerated Benefits demonstration project could help policymakers determine whether to eliminate the 24 month waiting period for Medicare that DI beneficiaries encounter under current law. SSA officials also anticipate that demonstration projects in progress could yield key information on how to improve outcomes for certain subgroups of beneficiaries. For example, SSA officials said that the Youth Transition Demonstration, which targets young people with disabilities as they transition from school to work, could identify strategies for improving the self-sufficiency of these beneficiaries and thus reduce their dependence on the disability programs. Most of SSA's current demonstration projects are expected to continue until 2010 or later before generating final evaluation results that could inform changes to disability program policy. SSA Has Taken Steps to Improve Its Projects but Continues to Lack Management Controls: SSA has taken steps to improve its demonstration projects, in part by applying more rigorous methodologies than it did for the projects SSA initiated prior to 1998; however, it has not fully implemented GAO's recommendations from 2004 and does not have written policies and procedures in place to ensure that projects are routinely reviewed and effectively managed so that they yield reliable information about their impacts. As a result, some projects faced challenges, such as low participation rates or data collection problems, which were significant enough to hinder the agency's ability to evaluate the projects' impacts as planned. In addition, without comprehensive written policies and procedures governing how SSA manages and operates its demonstration program, the project objectives, designs, and evaluation plans may be impacted during times of organizational change. SSA Has Taken Steps to Improve Its Demonstration Projects: SSA has improved its demonstration projects by applying more rigorous methodologies than it did prior to 1998, contracting with professional researchers and appointing new management for the program. Specifically, SSA is applying more rigorous evaluation methodologies to the projects it has initiated since 1998 than it did to the projects initiated in the late 1980s and early 1990s. At the time of our prior report, SSA officials acknowledged that the limited rigor of those earlier projects reduced their usefulness and indicated that the agency had placed a new emphasis on ensuring that its projects going forward would be more rigorously designed.[Footnote 17] Of the 14 projects that SSA has initiated since 1998, 13 were early enough in the planning or design stages at that time to give SSA an opportunity to make such improvements.[Footnote 18] Since that time, SSA has completed much of the design work for its 14 projects and provided us with detailed design information for 12 of them, enabling us to assess the rigor of these projects' designs for our current review. Our current analysis shows that SSA did use more rigorous methodologies for the projects initiated over the last decade than for its earlier projects. SSA is now using methodologies known as experimental or quasi- experimental designs, which are commonly used by research professionals conducting demonstration projects to estimate the impacts of program or policy changes. On the basis of our assessment, we determined that 11 of the 12 projects' designs were strong or reasonable when assessed against professional research standards (see table 3).[Footnote 19] We compared each project's design against GAO and recognized academic criteria for conducting evaluation research, which were also consistent with statutory requirements that DI projects be generally sufficient in scope and planned in such a way to permit a thorough evaluation of the program or policy changes under consideration. We also determined that the projects currently under way could provide some reliable results if implemented and evaluated as designed. Despite this progress, we found that SSA did not always meet additional DI and SSI statutory requirements regarding the general applicability of the projects' results and the use of expert advice, respectively. The authorizing statute for DI demonstration projects requires that the results derived from the projects will obtain generally in the operation of the disability program.[Footnote 20] While one of the six DI projects, the BOND project, has been designed to yield nationally representative information about the impacts of the project, the statute does not require that the results be applicable to all DI beneficiaries nationwide.[Footnote 21] However, the results should apply to a larger group of beneficiaries than just those that participated in the demonstration project, and SSA may be able to apply the results from three other DI projects--the Accelerated Benefits demonstration project, the Benefit Offset - 4 State Pilot, and the Mental Health Treatment Study--more generally because it plans to implement and evaluate the projects in a consistent manner at multiple sites. In addition, one of the two jointly authorized projects--the State Partnership Initiative--did not yield generally applicable results because the projects were not implemented consistently across each state.[Footnote 22] The authorizing statute for SSI projects requires the Commissioner of SSA to obtain the advice and recommendations of specialists who are competent to evaluate the proposed projects as to the soundness of their design, the possibilities of securing productive results, the adequacy of resources to conduct the proposed research or demonstrations, and their relationship to other similar research or demonstrations already completed or in process before entering into a contract, grant, or cooperative agreement for the project.[Footnote 23] However, SSA obtained advice from experts for only two of the six SSI projects. Finally, SSA generally met other design criteria required by statute for the BOND project (see app. II).[Footnote 24] Table 3: Assessment of Demonstration Projects' Designs against Professional Research Standards and Statutory Requirements: Project: Projects conducted under DI demonstration authority: Accelerated Benefits (In progress); Methodology: Experimental; Assessment against professional research standards[A]: Strong; Assessment against additional DI statutory requirements[B]: Meets to some extent; Assessment against additional SSI statutory requirements[C]: NA. Project: Projects conducted under DI demonstration authority: Benefit Offset - 4 State Pilot[D[(IN] In progress); Methodology: Experimental; Assessment against professional research standards[A]: Reasonable; Assessment against additional DI statutory requirements[B]: Meets to some extent; Assessment against additional SSI statutory requirements[C]: NA. Project: Projects conducted under DI demonstration authority: Benefit Offset National Demonstration (In progress); Methodology: Experimental; Assessment against professional research standards[A]: Strong; Assessment against additional DI statutory requirements[B]: Meets; Assessment against additional SSI statutory requirements[C]: NA. Project: Projects conducted under DI demonstration authority: California RISE (CA HIV/AI) (Cancelled); Methodology: Experimental; Assessment against professional research standards[A]: Reasonable; Assessment against additional DI statutory requirements[B]: Does not meet; Assessment against additional SSI statutory requirements[C]: NA. Project: Projects conducted under DI demonstration authority: Early Intervention Demonstration (Cancelled); Methodology: Experimental; Assessment against professional research standards[A]: Strong; Assessment against additional DI statutory requirements[B]: Does not meet; Assessment against additional SSI statutory requirements[C]: NA. Project: Projects conducted under DI demonstration authority: Mental Health Treatment Study (In progress); Methodology: Experimental; Assessment against professional research standards[A]: Reasonable; Assessment against additional DI statutory requirements[B]: Meets to some extent; Assessment against additional SSI statutory requirements[C]: NA. Project: Projects conducted under SSI demonstration authority: Disability Program Navigator (Completed); Methodology: Unknown; Assessment against professional research standards[A]: Could not assess; Assessment against additional DI statutory requirements[B]: NA; Assessment against additional SSI statutory requirements[C]: Meets to some extent. Project: Projects conducted under SSI demonstration authority: Early Identification and Intervention Demonstration (Cancelled); Methodology: Unknown; Assessment against professional research standards[A]: Could not assess; Assessment against additional DI statutory requirements[B]: NA; Assessment against additional SSI statutory requirements[C]: Does not meet. Project: Projects conducted under SSI demonstration authority: Florida Freedom Initiative (Completed); Methodology: Quasi- experimental; Assessment against professional research standards[A]: Relatively weak; Assessment against additional DI statutory requirements[B]: NA; Assessment against additional SSI statutory requirements[C]: Does not meet. Project: Projects conducted under SSI demonstration authority: Homeless Outreach Projects and Evaluation (Completed); Methodology: Quasi-experimental; Assessment against professional research standards[A]: Reasonable; Assessment against additional DI statutory requirements[B]: NA; Assessment against additional SSI statutory requirements[C]: Does not meet. Project: Projects conducted under SSI demonstration authority: Interim Medical Benefits (Cancelled); Methodology: Quasi- experimental; Assessment against professional research standards[A]: Reasonable; Assessment against additional DI statutory requirements[B]: NA; Assessment against additional SSI statutory requirements[C]: Does not meet. Project: Projects conducted under SSI demonstration authority: Pediatric Medical Units (Cancelled); Methodology: Experimental; Assessment against professional research standards[A]: Reasonable; Assessment against additional DI statutory requirements[B]: NA; Assessment against additional SSI statutory requirements[C]: Meets. Project: Projects jointly authorized under DI and SSI demonstration authorities: State Partnership Initiative[D,E[(completed); Methodology: Mixed-experimental; quasi-experimental; Assessment against professional research standards[A]: Mixed-some strong; some reasonable; Assessment against additional DI statutory requirements[B]: Does not meet; Assessment against additional SSI statutory requirements[C]: Meets to some extent. Project: Projects jointly authorized under DI and SSI demonstration authorities: Youth Transition Demonstration[E[(IN] In progress); Methodology: Experimental; Assessment against professional research standards[A]: Strong; Assessment against additional DI statutory requirements[B]: Meets to some extent; Assessment against additional SSI statutory requirements[C]: Meets. Source: GAO analysis of SSA's demonstration projects' designs. [A] Appendix I discusses professional research standards in further detail. [B] The Social Security Act 234(b) requires that in addition to the demonstration projects being of sufficient scope and carried out on a wide enough scale to permit a thorough evaluation of the alternative methods under consideration, the projects should also give assurance that the results derived from the projects obtain generally in the operation of the disability program. [C] The Social Security Act 1110(a)(1) requires the Commissioner of SSA to obtain the advice and recommendations of specialists who are competent to evaluate the proposed projects as to the soundness of their design, the possibilities of securing productive results, the adequacy of resources to conduct the proposed research or demonstrations, and their relationship to other similar research or demonstrations already completed or in process. [D] Multiple evaluations were conducted for this demonstration project. [E] This project must meet the requirements specified in both the DI and SSI statutes because it is authorized under both of them. [End of table] Evaluation methodologies: Experimental designs involve random assignment of study participants to either a treatment group or a control group. The treatment group is subjected to the new program or policy, and the control group is not. The strength of the experimental design is in its assurance that those who experience the treatment are like those that do not experience the treatment in all important ways, except for the difference of receiving the treatment itself. Thus, after the study is over, researchers can usually attribute observed differences in outcomes between the treatment and control groups to the new policy or program. Research experts generally consider experimental designs to be the "gold standard" for impact evaluations. However, they may not always be feasible or may not necessarily provide the optimal approach for conducting a demonstration because of various implementation or ethical considerations. Quasi-experimental methods-- reasonable but less rigorous alternatives to experimental designs-- include study participants that receive a treatment, but do not involve the use of randomized control groups and instead rely on the nonrandom selection of a comparison group of nonparticipants with characteristics similar to the study participants. Quasi-experimental methods may instead be preferable or may need to be used in conjunction with experimental methods to produce the most rigorous results. For instance, quasi- experimental methods may be preferred when there is a chance that a randomized experimental approach will not remain intact, such as when participants in the control group are likely to receive elements of the policy or program under evaluation. However, because quasi-experiments do not randomly assign participants to treatment and control groups, the assurance that the two groups are actually similar in all critical ways except for receiving the treatment is missing, and they cannot establish cause and effect between the program under study and the observed outcomes. Nonetheless, they can be used to determine whether a relationship exists between the strategy being tested and the observed outcomes. To further improve the demonstration projects' planning and methodological rigor, SSA has used external research professionals to work with the agency on the design, implementation, or evaluation of 12 of the 14 projects. SSA officials have acknowledged the need for additional expertise to design and implement methodologically rigorous demonstration projects.[Footnote 25] Thus, SSA has awarded, or planned to award, contracts and cooperative agreements to research consultants and universities with such expertise to evaluate 12 of its 14 projects (see app. III). In nine cases, these researchers also worked on the design or implementation of the projects. For example, for the Accelerated Benefits demonstration project, an SSA research contractor also designed how and where the project would be conducted and managed its implementation so that the data needed for the evaluation it plans to conduct will be available. We also found that SSA and most of these researchers communicated regularly when collaborating on these projects and researchers submitted monthly or quarterly progress reports to SSA, which included information on expenditures, progress, and areas of concern that needed to be addressed. SSA also appointed new program management in 2007. Since that time, the new management team has conducted an internal review of the 10 demonstration projects that were under way at the time of their appointment. SSA officials told us that all projects underwent a thorough review that was conducted by the Acting Associate Commissioner for Program Development and Research and others with appropriate expertise. Documents we obtained indicate that the review identified the projects' strengths, weaknesses, and whether they were likely to yield reliable, useful results. For example, SSA considered whether a project's sample size and site selection were appropriate, if it had been implemented in accordance with its design, and if it faced any challenges that would prevent researchers from conducting a rigorous evaluation of its results. SSA concluded that five of its projects would continue, expecting that they were likely to yield reliable impact information; these five projects are currently in progress.[Footnote 26] However, the agency did need to make significant changes to strengthen one of the five projects' designs--the BOND project. At an earlier point in the design phase, SSA expanded the BOND project's scope to include multiple components in addition to the benefit offset, such as a health benefits package. The new management team subsequently determined that the cost estimates and program complexity associated with several of those components raised questions about the feasibility of implementing the project and significantly scaled back the scope of the study. For the other five projects the new management was reviewing, SSA determined they faced significant limitations or challenges--such as poorly chosen implementation sites and low participation--that made it highly unlikely for them to obtain reliable results or would have been duplicative of other ongoing research (see table 4). Thus, SSA cancelled those projects in 2007 and 2008. Although the only information SSA has obtained from these projects are some lessons learned from the Pediatric Medical Unit project, the agency projected that it would have spent another $82 million had it not canceled them.[Footnote 27] Based on the information SSA provided us about the challenges facing the projects and the expected future costs of conducting them, its decisions to cancel the five projects appear to have been data-driven and reasonable. Table 4: Reasons for Demonstration Projects' Cancellations: Project: California RISE (CA HIV/AI) Demonstration; Program focus: DI; Phase of project when cancelled: Design; Amount spent prior to cancellation: $2.1 million; Reasons SSA cited for cancellation: Reliable evaluation results were unlikely because similar services were already available in the state selected for implementation, potentially introducing control group contamination and problems recruiting enough participants for the study. Also, nationally representative results were unlikely because of unique conditions in the state; Significant coordination problems also existed between the implementation and evaluation components of the project. Project: Early Identification and Intervention Demonstration; Program focus: SSI; Phase of project when cancelled: Solicitation of proposals; Amount spent prior to cancellation: $50,000; Reasons SSA cited for cancellation: Reliable evaluation results were unlikely because the Request for Proposal for this study did not require the use of a rigorous evaluation methodology; The project also had little connection to SSA's mission, and there were competing needs for SSA's research resources. Project: Early Intervention Demonstration; Program focus: DI; Phase of project when cancelled: Design; Amount spent prior to cancellation: $2 million; Reasons SSA cited for cancellation: This project was initiated as a stand-alone demonstration project but was incorporated into the BOND project in 2005. It added significant complexity to BOND and became very costly to carry it out. Project: Interim Medical Benefits; Program focus: SSI; Phase of project when cancelled: Prior to solicitation of proposals; Amount spent prior to cancellation: $0; Reasons SSA cited for cancellation: This project was very similar to the Accelerated Benefits demonstration project, and information from it would have been duplicative. Project: Pediatric Medical Unit (PMU); Program focus: SSI; Phase of project when cancelled: Implementation; Amount spent prior to cancellation: $2.9 million; Reasons SSA cited for cancellation: While the design for this project was reasonable in that it specified the use of an experimental design and establishment of a comparison group, when SSA implemented the project, the comparison group was not established. This prevented SSA from obtaining the level of rigor it wanted for this project; Insufficient sample size also made reliable evaluation results unlikely because some of the participating Disability Determination Services were not referring enough cases to the PMU for review. In addition, the low referral rates made the cost per participant very high. Project: Total; Program focus: [Empty]; Phase of project when cancelled: [Empty]; Amount spent prior to cancellation: $7.05 million; ($4.1 million from DI trust fund, $2.95 million from appropriations); Reasons SSA cited for cancellation: [Empty]. Source: GAO analysis of SSA's demonstration projects and budget data. [End of table] Further, SSA consolidated its research expertise by merging the Office of Disability and Income Security Programs (DISP) with SSA's Office of Policy in February 2008, creating the Office of Retirement and Disability Policy. As of June 2008, each office's research unit remains intact and no formal organizational changes were made to the demonstration program, but agency officials told us that the merger has facilitated communication and strengthened relationships between researchers within the agency.[Footnote 28] For example, experts from the former Office of Policy's research unit routinely review and provide input on the demonstration projects' designs and evaluations. SSA Continues to Lack Policies, Procedures, and Other Mechanisms to Ensure Projects Yield Reliable Information: While SSA is taking steps to generally improve its demonstration projects' designs and address specific project limitations, it does not have policies, procedures, and mechanisms to ensure that demonstration projects will yield reliable information about the impacts of the programs they are testing. According to internal control standards in the federal government, federal agencies should have policies, procedures, and mechanisms in place to provide reasonable assurance that a program's objectives are being achieved.[Footnote 29] However, we found that, as of August 2008, SSA had not fully implemented the recommendations we made in 2004 to help ensure the effectiveness of the demonstration projects.[Footnote 30] Specifically, SSA continues to lack: * a formal, comprehensive, long-term agenda for conducting demonstration projects; * an expert panel to review and provide regular input on the design and implementation of demonstration projects from the early stages of a project through its final evaluation; and: * a formal process for fully considering the potential policy implications of its demonstration projects' results and fully apprising Congress of the results and their policy implications. We did find that SSA has developed a limited research agenda for its projects that lacks basic details about the projects, including their objectives, schedules, and costs. The agenda was also developed without broadly consulting key internal and external stakeholders to obtain their input. SSA officials told us that they do not plan to update the agenda to reflect that some of the projects have been cancelled. In contrast, other federal agencies that conduct research have published much more detailed research agendas and update them regularly. For example, the Department of Education's National Institute on Disability and Rehabilitation Research publishes a 5-year plan that outlines priorities for rehabilitation research, demonstration projects, training and related activities, explains the basis for such activities, and publishes the plan for public comment before submitting it to Congress. In addition, agency officials said that SSA planned to continue using experts only on an ad hoc basis, citing how potential conflicts of interest could pose challenges to serve on an expert panel as we had recommended.[Footnote 31] SSA established ad hoc panels or consulted with outside experts for 8 of its 14 demonstration projects, including 4 of those that are currently in progress. Many of SSA's project officers and contractors reported that this was a positive experience, and SSA management has told us that they plan to continue using experts in the future. However, under this approach SSA may miss the opportunity to obtain advice more broadly on the demonstration program. For example, the panels and experts that SSA used were brought on board after the agency decided to initiate the demonstration projects; therefore, they were not in position at an early enough point in time to help SSA consider whether a demonstration project or an alternative research approach was, in fact, the best way to meet the agency's needs. Furthermore, SSA did not regularly seek input from the Social Security Advisory Board or the National Council on Disability, which both play key roles in federal disability policy and could be in a position to advise SSA more broadly on the demonstration projects.[Footnote 32] In addition, our prior work found that SSA had not sufficiently provided information on the status and results of its demonstration projects to Congress. In our current review, we found that SSA regularly submits annual reports about the DI demonstration projects to its congressional oversight committees. While SSA meets its statutory requirements by submitting these reports, the information in them is generally limited to descriptions of the projects' objectives and the dates of upcoming milestones. Similarly, the information that SSA reports about its SSI demonstration projects is limited to brief descriptions in the agency's annual congressional budget justifications. Key information that could help Congress monitor the progress of the demonstration projects--including project costs, potential risks and obstacles to their success, or the policy implications of their results--was rarely included in the annual reports or budget justifications. However, SSA officials also told us that they sometimes share additional information with Congress about the demonstration projects. For example, SSA officials told us they met with congressional committees in October and November 2007 to share information about its design plans for the BOND project. In addition to not fully addressing our prior recommendations, SSA does not have written policies and procedures governing how it should review and operate its demonstration program. Accordingly, SSA does not have a written policy requiring SSA management to review its project officer's demonstration projects on a regular basis. Standards for internal control in the federal government state that managers should compare a program's actual performance against expected targets and analyze significant differences.[Footnote 33] Although the new program management team reviewed each of the demonstration projects at the time of their appointment, SSA does not have a written policy requiring such a review process periodically throughout the design, implementation, and evaluation phases of each project. SSA's lack of a policy to systematically review each project on a periodic basis contributed to problems sometimes going undetected after the projects were implemented, and they did not yield the data needed for their evaluations. For example, because SSA was not actively involved in implementing or monitoring the Florida Freedom Initiative demonstration project, it was not in a position to take steps to ensure that the project proceeded as planned, and staff at the state level failed to enroll enough participants to generate data for the evaluation that SSA planned. Therefore, no evaluation was conducted for this project. In addition, we found that SSA does not have written procedures for its project officers to follow as they design, implement, and evaluate its demonstration projects. Such procedures could be used to ensure that standard research practices, such as conducting pilot phases and including internal and external stakeholders, are applied when planning and implementing the demonstration projects. Specifically, SSA does not require staff to regularly use pilots to test projects' underlying assumptions, operational logistics, or feasibility before they are implemented. As a result, SSA planned or conducted pilots or phased implementations for only 8 of its 14 projects, although GAO criteria for evaluation research emphasize the importance of conducting pilots, as they are a critical test of a project's design.[Footnote 34] At least four of the projects that did not include pilot phases experienced the type of logistical challenges that pilots are intended to identify. For example, the Homeless Outreach Projects and Evaluation demonstration project experienced start-up delays because of compatibility problems between the contractor's online data collection system and the computer systems at the 41 sites where the project was implemented. If SSA had conducted a pilot phase for this project, it may have detected these issues at a smaller number of sites and developed a plan to resolve them prior to implementing the full project. In addition, SSA does not have written procedures directing its project officers or contractors to routinely consult with internal and external stakeholders when planning the demonstration projects. We found that at least 11 of the 14 projects experienced challenges or limitations because SSA had not obtained sufficient input from, or coordinated effectively with, internal and external stakeholders. For example, SSA officials told us that they were aware that the Benefit Offset - 4 State Pilot project and the Benefit Offset National Demonstration project required a change to internal SSA processes for calculating DI benefits but did not coordinate with key internal stakeholders early on to determine how to make this change, and no systematic process was put in place. As a result, SSA had to calculate payments by hand for the Benefit Offset - 4 State Pilot, and the BOND project's implementation has been delayed while SSA now works with its internal stakeholders to determine how to make the needed changes. Furthermore, SSA officials, contractors, and representatives from the various demonstration projects' implementation sites told us that there was little input from internal stakeholders and that internal coordination problems existed on at least seven of these projects, including three of those that were cancelled. We also found that lack of coordination or communication with external stakeholders led to challenges in at least seven of these projects, including four of those that were cancelled. For example, coordination problems between the two contractors for the California Rise project resulted in the project's components being designed in isolation from each other, which complicated the evaluation plans and eventually contributed to the project's cancellation. In addition, SSA did not always include the prospective implementation sites in the planning and design of its projects, although they could have provided insight into the feasibility and logistical requirements of the project. While SSA has periodically provided direction informally to its project officers, some project officers told us that more formal guidance would have helped them to better understand what steps were necessary and expected, and we concluded from our discussions with others that such guidance would have been helpful. SSA recognizes that the program's lack of written procedures is a limitation and is drafting a guidebook on standard research practices for staff to follow when planning and designing demonstration projects. Although an SSA official told us that this document is a work in progress, it appears that the handbook will include key procedures for designing a project, such as identifying what data is needed for the evaluation and how it should be obtained. It also directs project officers to assemble a team for project development that includes staff from across the agency who will be able to collaborate and provide the input necessary to address the multiple components of the demonstration project. The draft guidebook also includes provisions for assembling a research panel composed of internal and external experts. This panel would review proposed research projects and identify those that present the most promising opportunities, taking into consideration the extent to which prior research has already addressed the topic. However, in its current form, the guidebook provides little direction for the implementation or evaluation phases of the demonstration projects, and SSA officials had not finalized it as of May 2008. Without comprehensive written policies and procedures governing how SSA manages and operates its demonstration programs, the project objectives, designs, and evaluation plans may be impacted during times of organizational change. Because government operating conditions continually change, agencies should have mechanisms in place to identify and address any special risks arising from such changes, especially those caused by hiring new personnel to occupy key positions in the agency.[Footnote 35] However, because SSA lacks mechanisms such as a standing advisory panel or written policies and procedures to provide continuity for its demonstration program when organizational changes occur, it cannot guarantee that institutional knowledge about the projects is shared or that the impacts of such changes are considered as the projects progress. SSA has experienced several organizational changes since the first of these projects was initiated in 1998, which have included the demonstration program's relocation from the Office of Policy to the Office of Disability and Income Support Programs in 2002, program management's replacement in 2007, and the Office of Policy and Office of Disability and Income Support Program's merger in 2008. At least six of SSA's projects experienced schedule delays and cancellations, in part because newly appointed officials made significant changes to some projects or determined that because others faced significant limitations or potential challenges it was not in the agency's interests to continue them (see fig. 1). While certain management actions may be reasonable, SSA's lack of written policies and procedures governing how such steps are taken leaves current and future projects vulnerable to disruption. For example, we found that the Benefit Offset National Demonstration project is still in the design phase after 9 years, during which time it has gone through numerous revisions by different program managers and was moved from one office to another. As of August 2008, an interagency working group was determining how to implement this administratively complex project. SSA has put the project's implementation and evaluation on hold until this issue has been resolved. Figure 1: Projects Experienced Schedule Delays and Cancellations Associated with Organizational Changes: This figure is a chart showing projects which experienced schedule delays and cancellations associated with organizational changes. [See PDF for image] Source: GAO analysis of SSA data. [A] As of August 2008, SSA had not determined the expected implementation or evaluation date for the BOND project. [End of figure] Conclusions: For over two decades, SSA has had the authority to conduct demonstration projects to test strategies that could address the challenges posed by the low rate of return into the workforce and the growing number of applicants and disabled beneficiaries. However, the agency has missed opportunities to identify ways to modernize DI and SSI programs and policies because it has generally not conducted the demonstration projects effectively. Since 1998 alone, SSA has spent over $150 million on 14 demonstration projects; yet these projects have generated limited information about the impacts of the strategies that were being tested. Although many of these projects were generally well designed, SSA's lack of written polices, procedures, and mechanisms for managing and operating these projects is one of the key reasons the projects SSA has completed and cancelled to date were generally not implemented and evaluated in a way that yielded reliable, data-driven impact information. As a result, Congress, SSA, and other organizations that play a critical role in federal disability policy continue to lack key information about important issues, such as the impact of providing health care or employment supports to DI and SSI beneficiaries as a means to help beneficiaries achieve self sufficiency and leave the rolls. SSA's five demonstration projects currently in progress have the potential to identify solutions to some of SSA's challenges. However, if SSA does not address the limitations in the way it manages and operates demonstration projects, these projects may encounter the same challenges that past projects have faced, and SSA could again have little to show for its efforts. Given that SSA estimates it will spend approximately $220 million over the next several years to complete these projects, it is important that steps be taken to make the projects less vulnerable to the challenges and organizational changes they could encounter in the future. SSA's actions to review its demonstration projects and to begin drafting guidance to help staff better plan and design its projects are encouraging first steps. As SSA officials work toward finalizing this guidance, it is also necessary for the agency to address its lack of written policies and procedures for managing and operating its projects during their implementation and evaluation phases. SSA should also take action to fully implement the recommendations we made in 2004. Implementing those recommendations by fully developing its research agenda, establishing an expert panel to advise it about the projects on a regular basis, and improving its communications with Congress could help improve the effectiveness and transparency of its demonstration program going forward. Recommendation for Executive Action: To improve SSA's management of its demonstration projects we recommend that the Commissioner of Social Security direct the Deputy Commissioner for the Office of Retirement and Disability Policy to: * establish written policies, procedures, and mechanisms for managing and operating its demonstration projects that are consistent with standard research practices and internal control standards in the federal government, including those for coordinating with internal and external stakeholders and sharing information with Congress. Agency Comments and Our Evaluation: We obtained written comments on a draft of this report from SSA, which are reproduced in appendix IV. We incorporated technical comments we received throughout the report, as appropriate. In response to our draft report, SSA generally agreed with our recommendation and acknowledged the need to develop a guidebook to assist its staff in the design, implementation, and evaluation phases of its demonstration projects. SSA further discussed its existing processes and written procedures for managing and reviewing its programs, including the demonstration project program. While noteworthy, we continue to believe SSA needs to establish written procedures that incorporate professional research standards and internal control mechanisms for ensuring that the demonstration projects yield reliable information about their impacts. SSA considers its current guidebook a work in progress. Further, SSA stated that the agency has taken steps in recent years to address our prior recommendations. While we acknowledge SSA's efforts, the agency needs to take additional steps to fully implement them. For example, SSA continues to lack a standing expert panel to review and provide regular input on the demonstration projects overall, even though it has employed subject matter experts for some of its demonstration projects. Although SSA officials have raised concerns about the difficulty of establishing an expert panel because research contractors serving on the panel would be precluded from working on individual projects, we continue to believe that such a panel could be established. As previously recommended, this panel should also include SSA's key research personnel and outside disability experts in addition to researchers. We are sending copies of this report to the Commissioner of SSA and other interested parties. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at [hyperlink, http://www.gao.gov]. If you or your staffs have any questions concerning this report, please contact me at (202) 512-7215. Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. Signed by: Daniel Bertoni: Director, Education, Workforce, and Income Security: [End of section] Appendix I: Scope and Methodology: To determine how SSA has used its current demonstration authority, we reviewed legislation authorizing the Social Security Administration (SSA) to conduct Disability Insurance (DI) and Supplemental Security Income (SSI) demonstration projects, prior GAO report and SSA's Office of the Inspector General (OIG) reports, and reports to Congress on the demonstration authority. We reviewed documents from SSA and from publicly available sources, including the Federal Register and reports by other research organizations. We interviewed current and former SSA officials in the Office of Disability and Income Security Programs,[Footnote 36] specifically, the Office of Program Development and Research (OPDR) and the Office of Research, Evaluation and Statistics who had responsibility for, or involvement in, the demonstration projects. We also interviewed research contractors that worked on the demonstration projects and individuals from organizations that have a key role in federal disability policy. In addition, we interviewed staff from sites where SSA implemented 9 of the 14 demonstration projects. We selected sites that included ongoing, cancelled and completed projects, and represented diverse geographic regions throughout the United States. To better understand SSA's DI and SSI demonstration projects, we reviewed SSA documents describing the purpose, design, and status for all demonstration projects that were in progress, completed, or had been cancelled prior to completion. These documents included requests for proposals, project plans and schedules, interim or final project reports, and reports to Congress from 1998 to 2008. We used these documents to identify key characteristics of the projects, including the policy issues addressed, use of contractors, the authority used to conduct each project, project timelines, and information resulting from each project. We also examined the issues SSA tested or was in the process of testing in its demonstration program. We reviewed the authorizing statutes for the DI and SSI demonstration programs, as well as requirements for specific demonstrations included in the Ticket to Work and Work Incentives Improvement Act of 1999 to determine the extent to which the projects in SSA's demonstration program address statutory requirements. For projects cancelled during this time period, we collected cancellation memos and other documentation to determine SSA's reasons for the cancellations. To describe the costs associated with the program, we collected expenditures data from SSA for each project--including funds spent to date for each project and total anticipated funding for the projects that are currently in progress. We assessed the reliability of the budget data by (1) manually checking the required data elements, (2) reviewing existing information about the data and the system that produced them, and (3) interviewing agency officials knowledgeable about the data. When we found discrepancies, we brought them to the attention of SSA officials and worked with them to correct the discrepancies before conducting our analyses. Based on these efforts, we determined that the data were sufficiently reliable for the purposes of this report. To assess the extent to which SSA's demonstration projects were designed in accordance with professional research standards and statutory criteria, we reviewed the most current information that SSA provided about each project, either an evaluation design or final evaluation report. These design and evaluation methodologies were assessed against professional research standards, consistent with the authorizing statutes' methodological requirements and GAO's and recognized academic criteria for conducting evaluation research.[Footnote 37] Key components of the professional research standards include: * methodological rigor of the project's design and evaluation and their appropriateness given the purpose of the research (e.g. use of an experimental or quasi-experimental design for an impact evaluation); * appropriate handling of any problems encountered when implementing the evaluation's design, such as participant attrition or insufficient sample sizes; * appropriate handling of any problems encountered with the data, such as missing values or variables; * appropriate variables to ensure internal and external validity, given the evaluation's design; * appropriate data analysis and statistical models, such as frequencies or multivariate analysis, given the evaluation's design; and: * overall strength of the evaluation design and analysis. To assess the appropriateness of each study's methodology for answering the research questions, we developed two data collection instruments based on these professional research standards--one for evaluation designs and one for the final evaluation reports. We then examined the strengths and weaknesses of the evaluation designs and final reports, taking into consideration the project's objectives, resource constraints, methodological approach, technical adequacy of plans for data collection and analysis, and when available, the presentation of the findings. A social scientist read and coded each evaluation design or final report. A second social scientist reviewed each completed data collection instrument and the relevant documentation to verify the accuracy of every coded item. For each DI demonstration project, we also reviewed the reports to ascertain whether they met statutory requirements that the project's results be broadly applicable to relevant segments of the DI beneficiary population, not just the project participants. For each SSI demonstration project, we also interviewed agency officials to determine whether SSA met its statutory obligation to obtain the advice and recommendations of specialists who are competent to evaluate the projects as to the soundness of their design, the possibilities of securing productive results, the adequacy of resources to conduct them, and their relationship to other similar research or demonstrations already in progress. We also identified key provisions of the demonstration authority statutes to assess SSA's compliance with congressional reporting requirements. To address SSA's planning and management of its demonstration projects, we interviewed SSA management and staff about the agency's policies, guidance and processes on developing and implementing demonstration projects, and collected supporting documentation where available. We assessed the adequacy of SSA's internal controls using the criteria in GAO's Standards for Internal Control in the Federal Government, GAO/ AIMD 00-21.3.1, dated November 1999. These standards, issued pursuant to the requirements of the Federal Managers' Financial Integrity Act of 1982 (FMFIA), provide the overall framework for establishing and maintaining internal control in the federal government. Also pursuant to FMFIA, the Office of Management and Budget issued Circular A-123, revised December 21, 2004, to provide the specific requirements for assessing the reporting on internal controls. Internal control standards and the definition of internal control in Circular A-123 are based on the GAO Standards for Internal Control in the Federal Government. [End of section] Appendix II: Additional Statutory Criteria for the Benefit Offset National Demonstration Project: In addition to meeting professional research standards and the mandated methodological requirements for the DI demonstration projects, the Benefit Offset National Demonstration (BOND) project is required to address a number of specific issues.[Footnote 38] We found that BOND met all of these requirements except the one to test the project's effect on induced entry to and reduced exit from the DI rolls (table 5). Table 5: Additional Statutory Criteria for the Benefit Offset National Demonstration Project: Statutory requirement: The effects, if any, of induced entry into the project and reduced exit from the project; SSA action: SSA considered including an experimental analysis of whether a benefit offset would induce entry to the DI program. In consultation with their research contractor, agency officials determined that the best possible approach would be cost prohibitive in terms of resources and time, and very complex, and still would be unlikely to produce solid, reliable estimates of what induced demand would occur if a DI benefit offset were implemented nationally. SSA will explore alternative approaches to estimating potential entry effects. Statutory requirement: The extent, if any, to which the project being tested is affected by whether it is in operation in a locality within an area under the administration of the Ticket to Work and Self- Sufficiency Program established under section 1148 of the Social Security Act; SSA action: An SSA official told us the effects of the Ticket to Work (Ticket) program could not be evaluated as intended in the statute because the two programs did not run concurrently, although it was originally expected that they would. The statutory criteria presume that the Ticket program would be operating in some of the same locations as the BOND project, but not all, so that some BOND participants would be eligible to participate in the Ticket program, but others would not be eligible. The Ticket program has now been implemented nationwide, and most SSA beneficiaries are now eligible to participate in it, thus the two groups needed for this evaluation no longer exist. However, the agency official and SSA's research contractor told us that SSA will be able to determine which BOND participants are also participating in the Ticket program, and the influence of the Ticket program on these participants would be considered in the BOND evaluation. Statutory requirement: The savings that accrue to the Federal Old-Age and Survivors Insurance Trust fund, the Federal Disability Insurance Trust fund, and other federal programs under the project being tested; SSA action: The BOND project design includes a benefit-cost analysis that will estimate the net budgetary effects of the benefit offset and counseling provided in each treatment package separately for participants, society, SSA, the federal government, and state and local governments. Agency officials and the research contractor told us that the savings that accrue to the Federal Old-Age and Survivors Insurance Trust fund, the Federal Disability Insurance Trust fund, and other Federal programs will be part of that analysis. Statutory requirement: The annual cost (including net cost) of the project and the annual cost (including net cost) that would have been incurred in the absence of the project; SSA action: The BOND project design includes a benefit-cost analysis that will estimate the net budgetary effects of the benefit offset and counseling provided in each treatment package separately for participants, society, SSA, the federal government, and state and local governments. Agency officials and the research contractor told us that the annual cost (including net cost) of the project and the annual cost (including net cost) that would have been incurred in the absence of the project will be part of that analysis. Statutory requirement: The determinants of return to work, including the characteristics of the beneficiaries who participate in the project; SSA action: The BOND project design includes a participation analysis that will examine the proportion of beneficiaries that take up the offer of the benefit offset and/or counseling, the timing and intensity of their participation, the characteristics of participants and nonparticipants, and the reasons for participation and nonparticipation. Statutory requirement: The employment outcomes, including wages, occupations, benefits, and hours worked, of beneficiaries who return to work as a result of participation in the project; SSA action: The BOND project design includes an impact analysis of the benefit offset and benefits counseling on levels of employment, earnings and benefits, wages, and occupations. Other employment analyses such as effects on regularity of work will also be conducted. Impacts on other dimensions will also be investigated, such as health status, function in activities of daily living, taxes paid, and family income. Source: GAO analysis of statutory requirements and SSA data. [End of table] In addition, while not included as part of the BOND demonstration, SSA plans to conduct research on variations in the amount of the offset as a proportion of earned income to determine the appropriate offset disregard, in accordance with the DI authorizing statute's requirements.[Footnote 39] [End of section] Appendix III: SSA's Use of Research Contractors and Universities on the Demonstration Projects: Project: Projects conducted under DI demonstration authority: Accelerated Benefits; Research consultant or university: MDRC (prime contractor); Mathematica Policy Research (subcontractor); Role on the demonstration project: SSA awarded MDRC a contract in 2006 for the project's design, implementation, and evaluation. MDRC's design for the project and evaluation expanded on preliminary design criteria SSA developed between 2004 and 2005, and published in a Request for Proposal. MDRC finalized the design in collaboration with Mathematica Policy Research and SSA. MDRC and Mathematica began enrolling participants in the project in October 2007. Implementation will continue through 2011. MDRC is conducting the impact evaluation and Mathematica is conducting the process evaluation. SSA expects the evaluation to be completed in 2011. Project: Projects conducted under DI demonstration authority: Benefit Offset - 4 State Pilot; Research consultant or university: None; SSA awarded contracts directly to the states that are implementing the pilot; Role on the demonstration project: NA. Project: Projects conducted under DI demonstration authority: Benefit Offset National Demonstration (BOND); Research consultant or university: Abt Associates; Role on the demonstration project: SSA awarded Abt Associates a contract in 2004 for the project's design and will award Abt a second contract for its implementation and evaluation contingent on successful completion of the design. Abt's design for the project and evaluation is based on preliminary design criteria SSA developed between 1999 and 2004 and published in a Request for Proposal. Abt is finalizing the design and planning for implementation in collaboration with SSA. Project: Projects conducted under DI demonstration authority: California RISE (CA HIV/AI) Demonstration; Research consultant or university: Mathematica Policy Research; SSA also awarded the California Department of Rehabilitation a cooperative agreement in 2006 to design the employment supports for the project, and implement the project; Role on the demonstration project: SSA awarded Mathematica Policy Research a contract in 2006 to design and implement the health services part of the project and to design and conduct the evaluation. Mathematica's design for the heath services component and the evaluation is based on preliminary design criteria SSA developed between 2004 and 2006 and published in a Request for Proposal. Mathematica worked on the design in 2006 and 2007, and SSA reviewed and provided input on it. Mathematica expected to implement the project in 2007 and complete its evaluation in 2011, but SSA cancelled the project in 2007 before implementation began. Project: Projects conducted under DI demonstration authority: Early Intervention Demonstration; Research consultant or university: Disability Research Institute (Rutgers University); Abt Associates; Role on the demonstration project: SSA awarded the Disability Research Institute a contract in 2000 for Rutgers University to design, implement, and evaluate an Early Intervention pilot project. Rutgers proposed a design for the project and evaluation and submitted an evaluation design report for SSA's approval in 2002. Rutgers planned to implement the project in 2003 and begin its evaluation in 2004, but the project remained under design until the contract expired in 2005; SSA modified Abt's contract for the BOND project in 2005 to incorporate Early Intervention into BOND's design, implementation, and evaluation. Abt proposed options to incorporate the Early Intervention project, but SSA later decided to cancel it and eliminated it from Abt's contract in 2007. Project: Projects conducted under DI demonstration authority: Mental Health Treatment Study; Research consultant or university: Westat; Role on the demonstration project: SSA awarded a contract to Westat in 2005 to implement and evaluate a project SSA designed between 2003 and 2005. Westat began working with the project sites in 2005 to prepare for implementation and began enrolling participants in the project in 2006. Implementation is scheduled to continue through 2010. Westat began the evaluation in 2008 and will complete it in 2011. Project: Projects conducted under SSI demonstration authority: Disability Program Navigator; (with Department of Labor (DOL)); Research consultant or university: University of Iowa College of Law, Law Health Policy and Disability Center; Department of Labor also awarded states cooperative agreements to implement the project; Role on the demonstration project: DOL awarded a contract to the University of Iowa College of Law, Law Health Policy and Disability Center to provide technical assistance for the project's implementation and to evaluate the project. SSA provided partial funding for the contract through an Interagency Agreement with DOL in 2002, 2003, and 2004. Implementation began in 2003. The University of Iowa did not evaluate the impact of the project because it could not meet SSA's data security requirements in order to obtain data needed for the evaluation. SSA officials told us that DOL plans to evaluate the project under a new contract with Mathematica Policy Research, but SSA is not funding that evaluation. Project: Projects conducted under SSI demonstration authority: Early Identification and Intervention Demonstration; Research consultant or university: George Washington University; Role on the demonstration project: SSA awarded a contract to George Washington University in 2004 to conduct preliminary research related to the Early Identification and Intervention project. SSA designed the demonstration project between 2004 and 2006 and issued a Request for Proposal for its implementation in 2007, but did not award a contract to implement or evaluate the project. Project: Projects conducted under SSI demonstration authority: Florida Freedom Initiative; Research consultant or university: Mathematica Policy Research; Role on the demonstration project: SSA funded a contract for Mathematica to design an evaluation for the project through an Interagency Agreement with the Department of Health and Human Services in 2004. Due to low enrollment, data needed for the evaluation was not collected by the State of Florida, which designed and implemented the project between 2003 and 2007, and the evaluation was not conducted. Project: Projects conducted under SSI demonstration authority: Homeless Outreach Projects and Evaluation; Research consultant or university: Westat; Role on the demonstration project: SSA awarded a contract to Westat in 2004 to evaluate a project SSA designed between 2003 and 2004. SSA implemented the project in 2004. Westat completed its evaluation in 2007. Project: Projects conducted under SSI demonstration authority: Interim Medical Benefits; Research consultant or university: None-project cancelled before a contract was awarded; Role on the demonstration project: SSA planned to issue a Request for Proposal to hire a professional research organization to design, implement, and evaluate the project but cancelled it before it awarded the contract. Project: Projects conducted under SSI demonstration authority: Pediatric Medical Unit; Research consultant or university: Association of University Centers on Disabilities (AUCD); Role on the demonstration project: SSA awarded a contract to AUCD in 2006 to design, implement, and evaluate the project; however, AUCD's design for the project began in 2004 under an extension of another contract. AUCD partially implemented the project in 2006 and 2007. SSA cancelled the project in 2008 before AUCD completed the implementation or evaluation. Project: Projects jointly authorized under DI and SSI authorities: State Partnership Initiative; Research consultant or university: Virginia Commonwealth University (VCU) (prime contractor); Mathematica Policy Research (subcontractor); SSA also awarded 12 states cooperative agreements to design and implement the project; Role on the demonstration project: SSA awarded multiple contracts to VCU starting in 1998 to provide technical assistance to the states that were implementing the project and to conduct an evaluation. VCU completed its evaluation in 2006. SSA awarded three contracts executed by VCU's subcontractor, Mathematica: one in 1998 to provide technical assistance to the states that were implementing the project, another in 1999 to design the evaluation, and one in 2003 to conduct the evaluation. Mathematica completed its evaluation in 2005. Project: Projects jointly authorized under DI and SSI authorities: Youth Transition Demonstration; Research consultant or university: Mathematica Policy Research (prime contractor); MDRC (subcontractor); Role on the demonstration project: SSA originally designed and implemented the project in 2003, but due to methodological limitations, SSA awarded a contract to Mathematica in 2005 to redesign, implement, and evaluate the project. Mathematica began implementing the new design in 2006. Implementation will continue until 2013. Mathematica will complete its evaluation in 2014. Source: GAO analysis of SSA data. [End of table] [End of section] Appendix IV: Comments from the Social Security Administration: Social Security: Social Security Administration: Baltimore MD: 21235-0001: The Commissioner: September 16, 2008: Mr. Daniel Bertoni: Director, Education, Workforce, and Income Security Issues: U.S. Government Accountability Office: 441 G Street, NW: Washington, D.C. 20548: Dear Mr. Bertoni: Thank you for the opportunity to review and comment on the draft report, "Social Security Disability: Management Controls Needed to Strengthen Demonstration Projects" (GAO-08-1053). Enclosed are our detailed comments to the draft report recommendations along with suggested technical revisions. If you have any questions, please contact Ms. Candace Skurnik, Director, Audit Management and Liaison Staff, at (410) 965-4636. Sincerely, Signed by: Michael J. Astrue: Enclosure: Comments On The Government Accountability Office (GAO) Draft Report, "Social Security Disability: Management Controls Needed To Strengthen Demonstration Projects" (GAO-08-1053): Thank you for the opportunity to review and comment on this draft report. We appreciate the report's recognition of our efforts to use the demonstration authorities we were provided and to include more rigorous methodologies in doing so. We are in general agreement with the tenor of the report and are pleased with the report's acknowledgement of management's efforts over the past year to improve the management of demonstration projects. We also appreciate that the report recognizes that while experimental designs are the "gold standard" for impact evaluations, quasi-experimental methods can be appropriate given all the considerations of our demonstration projects. Although this report contains only one recommendation for executive action, it does reference and comment on our progress in implementing three recommendations from a prior audit report (GAO-05-19). The comments that follow address all four recommendations. Following these comments, we have included other comments and suggested revisions on the content of the report. Current Recommendation: To improve the Social Security Administration's (SSA) management of its demonstration projects we recommend that the Commissioner of Social Security direct the Deputy Commissioner for the Office of Retirement and Disability Policy to: * Establish written policies, procedures, and mechanisms for managing, and operating its demonstration projects that are consistent with standard research practices and internal control standards in the federal government, including those for coordinating with internal and external stakeholders and sharing information with Congress. Comment: Although we generally concur with this recommendation, the report does not address existing processes and written procedures for managing and reviewing demonstration projects. For example, for staff and managers responsible for demonstrations, the performance management system provides a means for routinely reviewing demonstrations from project planning through implementation and evaluation. Our management and the Office of Management and Budget (OMB) routinely review demonstrations for the budget process, including the development and review of all documents required for the conduct of demonstration projects and all material associated with the budget cycle. We have extensive information on our intranet website to help Contracting Officer's Technical Representatives who are working on demonstration projects understand the OMB clearance process for Federal Register notices and survey instruments for data collection. Other efforts that facilitate the effective management of demonstration projects include receiving a monthly update on each demonstration project and including demonstration projects in the annual work plan for the Office of Program Development and Research. For some demonstrations, management requires more frequent updates and becomes more directly involved at critical points. For example, for the Benefit Offset National Demonstration, our management meets with the project team at least on a weekly basis. We agree with the need to develop a guidebook to assist our staff in the design, implementation, and evaluation phases of demonstration projects and the value of piloting demonstration projects before proceeding with full implementation (page 28). As the report indicates, we have developed a guidebook. However, our guidebook does not provide as much information as this report recommends and we consider it a work in progress. We also believe our Temporary Assistance to Needy Families- Supplemental Security Income (TANF-SSI) Disability Transition Project, which is a collaborative effort between us and the Department of Health and Human Services' Administration for Children and Families, may serve as a model for developing possible interventions for demonstrations and add to the information and guidance contained in the guidebook. Prior Recommendations for GAO Audit Report GAO-05-19: Recommendation 1: Develop a formal, comprehensive, long-term agenda for conducting demonstration projects. Comment: Our current demonstration project documentation, as presented in our work plan and budget, does provide a significant amount of information related to our demonstration project agenda. However, our ability to develop new demonstration projects and a corresponding agenda is limited by our statutory authority and available resources. As this report indicates, Section 234 of the Social Security Act, as amended, expired in December 2005. Since that time, we have not had the statutory authority to initiate new demonstration projects under title II of the Act. This impediment has a significant impact on the scope of demonstration projects we can undertake. In addition, as recognized in this report, demonstration projects require extensive resources and are "inherently complex and difficult to conduct" (page 7). Demonstration projects must be conducted on a scale that allows us to make reasonable assumptions regarding the applicability of the outcomes to the general beneficiary population. For example, the total estimated cost of our Accelerated Benefits Demonstration project is $43 million. However, we will use nearly $32 million to provide health benefits to the 1,200 participants. In addition, we are providing benefits counseling and employment supports to a subset of those applicants. Therefore, the primary costs associated with these projects are the interventions provided and the need to provide those interventions to a large sample of beneficiaries. We are, therefore, limited in the number of demonstration projects we can implement at any given time. Recommendation 2: Establish an expert panel to review and provide regular input on the design and implementation of demonstration projects from the early stages of a project through its final evaluation. Comment: We responded to the earlier audit report by employing a Technical Advisory Group (TAG) for demonstration projects. Each TAG consists of nationally renowned subject matter experts for the particular demonstration project. By using experts in a specific field for each project or development of a particular intervention, we obtain the specific expertise necessary, as opposed to a standing panel of experts who may not have the specific expertise. In addition, we have worked with experts from the research community to discuss specific demonstration project issues. Finally, changes in our management and component organization, as mentioned in the report, have now made it possible for us to utilize the expertise available within the agency. As noted in the report, current procedures in the guidebook call for the use of expert panels that include internal and external experts for the development of demonstration projects. The guidebook also references the need for input and review of the draft statement of work from external experts. We intend to follow these procedures in approaching new demonstration projects. In discussing this recommendation, the report notes that we did not regularly seek input from key stakeholders for Federal disability policy, such as the Social Security Advisory Board. The only new work that we have underway that may result in a demonstration project is the TANF-SSI Disability Transition Project. This work responds to a prior recommendation from not only the GAO, but also the Social Security Advisory Board. We also recently discussed plans for the Benefit Offset National Demonstration with the Social Security Advisory Board. Before any new demonstration project is initiated, we will follow the existing written instructions in the guidebook that call for collaboration with external stakeholders. Recommendation 3: Establish formal processes to ensure that, at the conclusion of each demonstration project, SSA fully considers and assesses the policy implications of its demonstration results, and clearly communicates SSA's assessment to Congress. Comment: As the report indicates, our current demonstration projects include explicit provisions for fully analyzing project results, and we will clearly communicate the results and the policy implications to Congress. We have also reinforced this need by placing the statutory reporting requirements in our guidebook. The current report, in discussing this prior recommendation, criticizes the content of our annual reports to Congress. We agree with the overall suggestion that the reports could be more informative. We have already recognized this need, and our recent reports have attempted to provide Congress with more information regarding project status and potential issues. [End of section] Appendix V: GAO Contact and Staff Acknowledgments: GAO Contact: Daniel Bertoni, (202) 512-7215 or bertonid@gao.gov: Staff Acknowledgments: In addition to the individual named above, key contributions to this report were made by Michael Collins, Assistant Director; Jason Holsclaw and Anne Welch, Analysts-in-Charge; Dana Hopings; Annamarie Lopata; Jean McSween. Additional support was provided by Kenneth Bombara; Daniel Concepcion; Jennifer Cook; Cindy Gilbert; Sharon Hermes; Joanie Lofgren; Joel Marus; Mimi Nguyen; Patricia Owens; Daniel Schwimer; Kris Trueblood; Kathy White; Charles Willson; Elizabeth Wood; and Jill Yost. [End of section] Related GAO Products: Federal Disability Programs: More Strategic Coordination Could Help Overcome Challenges to Needed Transformation. GAO-08-635. Washington, D.C.: May 20, 2008. Social Security Disability: Better Planning, Management, and Evaluation Could Help Address Backlogs. GAO-08-40. Washington, D.C.: December 7, 2007. High Risk Series: An Update. GAO-07-310. Washington, D.C.: January 2007. Federal Disability Assistance: Wide Array of Programs Needs to be Examined in Light of 21st Century Challenges. GAO-05-626. Washington, D.C.: June 2, 2005. Social Security Disability: Improved Processes for Planning and Conducting Demonstrations May Help SSA More Effectively Use Its Demonstration Authority. GAO-05-19. Washington, D.C.: November 4, 2004. SSA's Rehabilitation Programs. GAO/HEHS-95-253R. Washington, D.C.: September 7, 1995. Impact of Vocational Rehabilitation Services on the Social Security Disability Insurance (DI) Program. GAO/HRD-T-88-16. Washington, D.C.: May 26, 1988. Social Security: Little Success Achieved in Rehabilitating Disabled Beneficiaries. GAO/HRD-88-11. Washington, D.C.: December 7, 1987. Social Security: Observations on Demonstration Interviews with Disability Claimants. GAO/HRD-88-22BR. Washington, D.C.: December 3, 1987. Social Security: Demonstration Projects Concerning Interviews with Disability Claimants. GAO/HRD-87-35. Washington, D.C.: February 19, 1987. [End of section] Footnotes: [1] GAO, Social Security Disability: Improved Processes for Planning and Conducting Demonstrations May Help SSA More Effectively Use Its Demonstration Authority. GAO-05-19 (Washington, D.C.: Nov. 4, 2004). [2] GAO, High Risk Series: An Update, GAO-07-310 (Washington, D.C.: January 2007). [3] GAO, Designing Evaluations, GAO/PEMD-10.1.4 (Washington, D.C.: March 1991); GAO, Assessing the Reliability of Computer-Processed Data, GAO/OP-8.1.3 (Washington, D.C.: April 1991); P.H. Rossi, M.W. Lipsey, and H.E. Freeman, Evaluation: A Systematic Approach (California, 2004); B.R. Worthen, J.R. Sanders, and J.L. Fitzpatrick, Program Evaluation: Alternative Approaches and Practical Guidelines (New York, 1997). [4] GAO, Standards for Internal Control in the Federal Government, GAO/ AIMD-00-21.3.1 (Washington, D.C.: November 1999). These standards, issued pursuant to the requirements of the Federal Managers' Financial Integrity Act of 1982 (FMFIA), provide the overall framework for establishing and maintaining internal control. Also pursuant to FMFIA, the Office of Management and Budget issued Circular A-123, revised December 21, 2004, to provide the specific requirements for assessing the reporting on internal controls. Internal control standards and the definition of internal control in Circular A-123 are based on the GAO Standards for Internal Control in the Federal Government. [5] To be eligible for DI benefits, a worker must be (1) insured and (2) disabled according to the definition of disability. To be insured, they must have worked a minimum amount of time in employment covered by Social Security (similar to eligibility for Old-Age and Survivors Insurance benefits). However, for disability benefits, if an individual does not have 40 quarters of coverage (generally about 10 years), they must have 1 quarter of coverage (1 quarter of coverage is equal to $1,050 in 2008 and indexed to the annual increase in wages) for each year after 1950 or from age 21 up to the onset of disability. [6] DI demonstration authority was provided under 505(a) of the Social Security Disability Amendments of 1980 (Pub. L. No. 96-265). Another provision of the act, 505(b), amended 1110 of the Social Security Act of 1935 to provide SSA with similar demonstration authority for the SSI program, including authority to waive SSI program rules. [7] Pub. L. No. 108-203. [8] GAO-05-19. [9] GAO-05-19. [10] SSI demonstration authority focuses, in part, on prevention and reduction of dependency, aid in effecting coordination of planning between private and public welfare agencies, improving the administration and effectiveness of programs carried on or assisted under the Social Security Act, and ascertaining the feasibility of treating alcoholics and drug addicts to prevent the onset of irreversible medical conditions which may result in permanent disability. 42 U.S.C 1310. [11] 42 U.S.C. 434. [12] Pub. L. No. 106-170 302. [13] Individuals with serious mental illnesses now represent over a quarter (27 percent) of all DI recipients, and they account for the single largest diagnostic group (34 percent) on the SSI rolls. As part of our ongoing work on modernization of federal disability programs, we have raised the issue of planning for growth in the demand for services and benefits. See GAO, Federal Disability Assistance: Wide Array of Programs Needs to be Examined in Light of 21st Century Challenges. GAO- 05-626 (Washington, D.C.: June 2, 2005). [14] 42 U.S.C. 434 234(b). [15] Several of the demonstration projects experienced challenges meeting new data security requirements that SSA established in 2006 following the theft of a Department of Veterans Affairs computer, which heightened SSA's concerns about data security. SSA officials told us that DOL plans to evaluate the project under a new contract with Mathematica Policy Research, but SSA is not funding that evaluation and has a very limited role in planning and designing it. It is unclear what information will be available to use for making disability program and policy decisions. [16] This program has been renamed the Work Incentives Planning and Assistance (WIPA) program. [17] GAO-05-19. [18] The State Partnership Initiative was nearing completion at this time, and it was too late to make such improvements to it. [19] We did not assess the evaluation designs for the Disability Program Navigator and the Early Identification and Intervention Demonstration projects because SSA could not provide us with needed information. The Disability Program Navigator project concluded without an evaluation, and the Early Identification and Intervention Demonstration was cancelled before its design was finalized. [20] Social Security Act 234(b). [21] Research experts have noted that there is an inherent tradeoff between using experimental designs and generalizing results. [22] Each state designed its own project for the State Partnership Initiative. [23] Social Security Act 1110(a)(2) [24] Pub. L. No. 106-170 302. [25] GAO-05-19. [26] According to our analysis, the designs of the projects currently under way are considered strong or reasonable when assessed against professional research standards. [27] In response to our draft report SSA provided technical comments claiming an additional $600 million in savings resulting from a project cancellation. After analyzing the supporting documentation, we determined that the cost estimates that SSA provided did not directly support the proposed cost savings; therefore, we did not incorporate the suggested change into the report. [28] In 2007, SSA had moved DISP's operational component to another office in response to findings and recommendations made by its Inspector General. Agency officials told us that following the reorganization, DISP and the Office of Policy had similar missions, especially in regard to research. [29] GAO/AIMD-00-21.3.1. [30] GAO-05-19. [31] GAO-05-19. [32] The Social Security Advisory Board is an independent, bipartisan board created by Congress and appointed by the President and Congress to advise the President, Congress, and the Commissioner of Social Security on matters related to the Social Security and Supplemental Security Income programs. The National Council on Disability is an independent federal agency, composed of members appointed by the President by and with the advice and consent of the U.S. Senate, to provide advice to the President, Congress, and executive branch agencies to promote policies, programs, practices, and procedures that guarantee equal opportunity for all individuals with disabilities. [33] GAO/AIMD-00-21.3.1. [34] GAO/PEMD-10.1.4. [35] GAO-AIMD-00-21.3.1. [36] This office was merged with SSA's Office of Policy in February 2008 and is now called the Office of Retirement and Disability Policy (ORDP). [37] GAO, Designing Evaluations, GAO/PEMD-10.1.4 (Washington, D.C.: March 1991); GAO, Assessing the Reliability of Computer-Processed Data, GAO/OP-8.1.3 (Washington, D.C.: April 1991); P.H. Rossi, M.W. Lipsey, and H.E. Freeman, Evaluation: A Systematic Approach (California, 2004.) B.R. Worthen, J.R. Sanders, and J.L Fitzpatrick, Program Evaluation: Alternative Approaches and Practical Guidelines (New York, 1997). [38] Pub. L. No. 106-170 302. [39] Social Security Act of 1935 234(1)(C). GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office: 441 G Street NW, Room LM: Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548:

The Justia Government Accountability Office site republishes public reports retrieved from the U.S. GAO These reports should not be considered official, and do not necessarily reflect the views of Justia.