Social Security Disability
Management Controls Needed to Strengthen Demonstration Projects Gao ID: GAO-08-1053 September 26, 2008Since 1980, Congress has required the Social Security Administration (SSA) to conduct demonstration projects to test the effectiveness of possible changes to its Social Security Disability Insurance (DI) and Supplemental Security Income (SSI) programs that could decrease individuals' dependence on benefits or improve program administration. However, in 2004, GAO reported that SSA had not used its demonstration authority effectively. This follow-up report assesses (1) how SSA has used its demonstration authority to test DI and SSI program changes and what information these efforts have yielded and (2) what steps SSA has taken to improve the planning and management of its demonstration projects. To do this, GAO reviewed documents related to SSA's demonstration project management and the steps it took to implement the recommendations in the 2004 report, as well as the projects' designs, evaluations, and costs. GAO also interviewed officials from SSA, its contractors and project sites, and disability experts.
Over the last decade, SSA has initiated 14 demonstration projects under its authority to test possible DI and SSI policy and program changes; however, these projects have yielded limited information for influencing program and policy decisions. Of the 14 projects, SSA has completed 4, cancelled 5, and had 5 projects in progress as of June 2008. In total, SSA spent about $155 million on its projects as of April 2008, and officials anticipate spending another $220 million in the coming years on those projects currently under way. Yet, these projects have yielded limited information on the impacts of the program and policy changes they were testing. SSA did not conduct impact evaluations for two of its completed projects, and intended to evaluate five other projects, but could not do so because significant challenges led SSA to cancel them. SSA officials believe the five projects currently under way will yield useful information, but it is too early to tell. SSA has taken steps to improve its demonstration projects but continues to lack management controls to ensure that the projects yield reliable information for making disability policy decisions. SSA has used methodological designs that GAO determined were strong or reasonable when assessed against professional research standards for 11 of its 14 projects. SSA has also used external research professionals to work with the agency on the design, implementation, or evaluation of 12 of the projects, and appointed new program management to oversee its demonstration program. However, as of August 2008, SSA had not fully implemented the recommendations GAO made in 2004 and did not have written policies and procedures governing how it should review and operate its demonstration project program. Specifically, SSA does not have written policies and procedures for its managers and project officers to follow as they design, implement, and evaluate its demonstration projects. Absent such protocols, SSA did not always apply standard research practices, such as conducting pilot phases or obtaining sufficient stakeholder input, which led to data limitations and project cancellations.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-08-1053, Social Security Disability: Management Controls Needed to Strengthen Demonstration Projects
This is the accessible text file for GAO report number GAO-08-1053
entitled 'Social Security Disability: Management Controls Needed to
Strengthen Demonstration Projects' which was released on September 26,
2008.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
September 2008:
Social Security Disability:
Management Controls Needed to Strengthen Demonstration Projects:
Social Security Disability:
GAO-08-1053:
GAO Highlights:
Highlights of GAO-08-1053, a report to congressional requesters.
Why GAO Did This Study:
Since 1980, Congress has required the Social Security Administration
(SSA) to conduct demonstration projects to test the effectiveness of
possible changes to its Social Security Disability Insurance (DI) and
Supplemental Security Income (SSI) programs that could decrease
individuals‘ dependence on benefits or improve program administration.
However, in 2004, GAO reported that SSA had not used its demonstration
authority effectively. This follow-up report assesses (1) how SSA has
used its demonstration authority to test DI and SSI program changes and
what information these efforts have yielded and (2) what steps SSA has
taken to improve the planning and management of its demonstration
projects.
To do this, GAO reviewed documents related to SSA‘s demonstration
project management and the steps it took to implement the
recommendations in the 2004 report, as well as the projects‘ designs,
evaluations, and costs. GAO also interviewed officials from SSA, its
contractors and project sites, and disability experts
What GAO Found:
Over the last decade, SSA has initiated 14 demonstration projects under
its authority to test possible DI and SSI policy and program changes;
however, these projects have yielded limited information for
influencing program and policy decisions. Of the 14 projects, SSA has
completed 4, cancelled 5, and had 5 projects in progress as of June
2008. In total, SSA spent about $155 million on its projects as of
April 2008, and officials anticipate spending another $220 million in
the coming years on those projects currently under way. Yet, these
projects have yielded limited information on the impacts of the program
and policy changes they were testing. SSA did not conduct impact
evaluations for two of its completed projects, and intended to evaluate
five other projects, but could not do so because significant challenges
led SSA to cancel them. SSA officials believe the five projects
currently under way will yield useful information, but it is too early
to tell.
Table: Demonstration Project Costs and Information Yielded Since 1998:
DI (trust fund);
Funds spent (1998 to April 2008): $48.3 million;
Projected expenditures (April 2008 forward): $174 million.
SSI (appropriations);
Funds spent (1998 to April 2008): $107.2 million;
Projected expenditures (April 2008 forward): $45.6 million.
Total;
Funds spent (1998 to April 2008): $155.5 million;
Projected expenditures (April 2008 forward): $219.6 million.
Information yielded;
Funds spent (1998 to April 2008): Impact Information–2 of 14;
Other information–3 of 14; No information–6 of 14;
Projected expenditures (April 2008 forward): Information not yet
available–4 of 14.
Source: GAO analysis of SSA deminstration project estimated project
cost and evaluation results as of April 2008.
Note: Projects sum to greater than 14 because one project yielded
impact information and other information.
[End of table]
SSA has taken steps to improve its demonstration projects but continues
to lack management controls to ensure that the projects yield reliable
information for making disability policy decisions. SSA has used
methodological designs that GAO determined were strong or reasonable
when assessed against professional research standards for 11 of its 14
projects. SSA has also used external research professionals to work
with the agency on the design, implementation, or evaluation of 12 of
the projects, and appointed new program management to oversee its
demonstration program. However, as of August 2008, SSA had not fully
implemented the recommendations GAO made in 2004 and did not have
written policies and procedures governing how it should review and
operate its demonstration project program. Specifically, SSA does not
have written policies and procedures for its managers and project
officers to follow as they design, implement, and evaluate its
demonstration projects. Absent such protocols, SSA did not always apply
standard research practices, such as conducting pilot phases or
obtaining sufficient stakeholder input, which led to data limitations
and project cancellations.
What GAO Recommends:
GAO recommends that SSA establish written policies and procedures for
managing and operating its projects consistent with standard research
practices and internal control standards in the federal government. In
response, SSA generally agreed with GAO‘s recommendation and
acknowledged that its guidance is a work in progress.
To view the full product, including the scope and methodology, click on
[http://www.gao.gov/cgi-bin/getrpt?GAO-08-1053]. For more information,
contact Daniel Bertoni at (202) 512-7215 or bertonid@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
SSA Has Used Its Demonstration Authority to Initiate Several Projects
Since 1998, but Efforts Have Yielded Limited Information:
SSA Has Taken Steps to Improve Its Projects but Continues to Lack
Management Controls:
Conclusions:
Recommendation for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Additional Statutory Criteria for the Benefit Offset
National Demonstration Project:
Appendix III: SSA's Use of Research Contractors and Universities on the
Demonstration Projects:
Appendix IV: Comments from the Social Security Administration:
Appendix V: GAO Contact and Staff Acknowledgments:
Related GAO Products:
Tables:
Table 1: Overview of Demonstration Projects Initiated Since 1998:
Table 2: Total Estimated Costs for Demonstration Projects Currently in
Progress:
Table 3: Assessment of Demonstration Projects' Designs against
Professional Research Standards and Statutory Requirements:
Table 4: Reasons for Demonstration Projects' Cancellations:
Table 5: Additional Statutory Criteria for the Benefit Offset National
Demonstration Project:
Figure:
Figure 1: Projects Experienced Schedule Delays and Cancellations
Associated with Organizational Changes:
Abbreviations:
BOND: Benefit Offset National Demonstration:
DI: Social Security Disability Insurance:
DISP: Disability and Income Security Program:
DOL: Department of Labor:
FICA: Federal Insurance Contributions Act:
FMFIA: Federal Managers' Financial Integrity Act of 1982:
HOPE: Homeless Projects Outreach and Evaluation:
OPDR: Office of Program Development and Research:
PMU: Pediatric Medical Unit:
SSA: Social Security Administration:
SSI: Supplemental Security Income:
WIPA: Work Incentives Planning and Assistance:
United States Government Accountability Office:
Washington, DC 20548:
September 26, 2008:
The Honorable Michael R. McNulty:
Chairman:
The Honorable Sam Johnson:
Ranking Member:
Subcommittee on Social Security:
Committee on Ways and Means:
House of Representatives:
As of October 2007, the Social Security Administration (SSA) reported
that nearly 8.9 million disabled workers and their dependents were
receiving benefits under the Social Security Disability Insurance (DI)
program, while approximately 7.4 million individuals received federally
administered Supplemental Security Income (SSI) payments. In our prior
work, we have raised concerns about significant growth in the
disability rolls and challenges in devising return-to-work policies. To
address these challenges and encourage individuals capable of working
to return to work, Congress has required SSA to conduct demonstration
projects to test the effectiveness of possible program changes that
could decrease individuals' dependency on benefits. Over the last
decade, SSA has spent about $155 million dollars on these projects and
anticipates that it will spend another $220 million on those currently
under way.
To conduct these demonstrations, Congress authorized SSA to waive
certain program rules temporarily and use trust fund dollars and
appropriated funds to finance their development. Congress also required
that DI demonstration projects be of sufficient scope and conducted on
a wide enough scale to ensure a thorough evaluation of the program or
policy change under consideration. However, in 2004, we reported that
SSA had not used its DI demonstration authority effectively and that
the demonstration projects conducted in the late 1980s and early 1990s
had little effect on SSA's and Congress's consideration of disability
policy issues.[Footnote 1] We have also placed federal disability
programs on our high-risk list--a list of federal programs that need
attention and transformation. We did so, in part because of concerns
about significant growth in the disability rolls and challenges in
devising return-to-work policies.[Footnote 2] As you requested, we
reviewed the adequacy of SSA's current and planned demonstration
projects for both DI and SSI programs. Specifically, this report
assesses (1) how SSA has used its demonstration authority to test DI
and SSI program changes and what information these efforts have yielded
and (2) what steps SSA has taken to improve the planning and management
of its demonstration projects.
To complete our work, we reviewed the demonstration projects that SSA
planned, initiated, or terminated between the calendar years 1998 and
2008 in order to identify their key characteristics, including their
purpose, current status, and costs. To gain a better understanding of
the components of SSA's demonstration projects, we conducted interviews
with staff at sites where SSA implemented 9 of the 14 demonstration
projects. We selected sites that included ongoing, cancelled, and
completed projects, and represented diverse geographic regions
throughout the United States. To determine what information these
projects have yielded about the impacts of the policies and programs
being tested and lessons learned, we collected and reviewed available
evaluation information on each project. We collected data from SSA on
the funds spent to date, projected future expenditures, and total costs
for the demonstration projects, and determined that these data were
sufficiently reliable for the purposes of this report. We also reviewed
documents related to SSA's planning, implementation, and evaluation of
its demonstration projects, including the development of the projects'
methodological designs. We reviewed the projects' methodological
designs against professional research standards for demonstration
projects. These research standards were derived from GAO and recognized
academic criteria for conducting evaluation research. Our Office of
General Counsel reviewed these standards and found them consistent with
the authorizing statutes' methodological requirements.[Footnote 3]
Further, we identified relevant federal statutes governing DI and SSI
demonstration activities and examined other reviews of SSA
demonstration projects that we and SSA's Office of the Inspector
General had conducted. For both objectives, we interviewed SSA
officials, project officers, and research contractors that worked on
the demonstration projects, officials from SSA's Office of Inspector
General, and individuals from organizations that have a key role in
federal disability policy.
To address SSA's planning and management of its demonstration projects,
we interviewed SSA management and staff about the agency's policies,
guidance and procedures for developing and implementing demonstration
projects, and collected supporting documentation where available. We
assessed the adequacy of SSA's internal controls using the criteria in
GAO's Standards for Internal Control in the Federal Government, which
provide the overall framework for establishing and maintaining internal
control in the federal government.[Footnote 4]
We conducted this performance audit from November 2007 to August 2008
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives. Appendix I discusses our
scope and methodology in further detail.
Results in Brief:
Over the last decade, SSA has initiated 14 projects under its
demonstration authority to test both DI and SSI program changes--6
related to DI, 6 related to SSI, and 2 examining both programs jointly-
-however, these projects have yielded limited information to date. Of
the 14 projects, SSA has completed 4, cancelled 5, and had 5 projects
in progress as of August 2008. In general, SSA has used its
demonstration authority to conduct projects to consider changes to
program administration and benefits counseling, as well as to test
options for providing employment assistance and health benefits for
beneficiaries. In total, SSA had spent about $155 million on these
demonstration projects as of April 2008, and officials anticipate
spending another $220 million in the coming years on those projects
currently in progress. Yet, SSA's demonstration projects have yielded
limited information about the impacts of the policies and programs
being tested. We found that SSA did not conduct impact evaluations--
assessments of a project's effects compared to what would have happened
in its absence--for two of its completed projects because the data it
planned to use was not available. In addition, SSA intended to evaluate
the impacts of five other projects but could not because significant
challenges led it to cancel them in the early stages, and thus no data
was available to assess those policies and programs being tested.
Nonetheless, SSA was able to obtain some preliminary information from
one of its cancelled projects. SSA also obtained some information from
the two completed projects it evaluated, but the evaluations' findings
had limitations. In one case SSA could not separate the effects of its
project from the effects of a similar federal project that was also
under way. SSA has also obtained some preliminary information from one
of the five projects currently under way, and agency officials believe
that these projects will ultimately yield useful information. However,
it is too early to tell whether the information obtained will be used
to inform changes to disability program policy. These projects are
expected to continue until 2010 or later.
SSA has taken steps to improve its demonstration projects but continues
to lack written management controls to ensure that the projects yield
reliable information about their impacts. For 11 of the projects, SSA
has used experimental or quasi-experimental designs--methods commonly
used by research professionals for demonstration projects--that we
determined were strong or reasonable when assessed against professional
research standards. SSA has also used external research professionals
to work with the agency on the design, implementation, or evaluation of
12 of the 14 projects, and appointed new management in 2007 to oversee
the demonstration project program. The new management team subsequently
conducted an internal review of the 10 demonstration projects that were
under way at the time of their appointment and cancelled 5 that they
determined were unlikely to yield reliable information because they
faced significant limitations or challenges, or would have been
duplicative of other ongoing research. SSA's decision to cancel these
projects appears to have been data-driven and reasonable. However, as
of August 2008, SSA had not fully implemented the recommendations we
made in 2004 to help ensure the effectiveness of the demonstration
projects. Further, the agency does not have written policies and
procedures governing how it should review and operate its demonstration
program. Specifically, SSA does not have written policies and
procedures for its managers and project officers to follow as they
design, implement, and evaluate its demonstration projects. Absent such
protocols, SSA sometimes did not apply standard research practices,
such as conducting pilot phases or obtaining sufficient stakeholder
input, which led to data limitations and project cancellations.
We are making a recommendation to SSA that is intended to improve the
management of its demonstration project program by establishing written
policies, procedures, and mechanisms for managing and operating the
demonstration projects. These policies and procedures should be
consistent with standard research practices and internal control
standards in the federal government. In response to our draft report,
SSA generally agreed with our recommendation and the need to develop a
guidebook to assist its staff in the design, implementation, and
evaluation phases of its demonstration projects. Furthermore, SSA
considers its current guidebook a work in progress. SSA also stated
that the agency has taken steps in recent years to address our prior
recommendations. While we acknowledge SSA's efforts, we maintain that
additional steps are needed to fully implement them. SSA's comments are
reproduced in appendix IV.
Background:
Under the Social Security Act of 1935, as amended, SSA administers two
federal disability programs--DI and SSI--intended to provide benefits
to individuals with disabilities who are unable to work. The DI
insurance program provides monthly cash benefits to individuals who
have a Social Security work record and the amount of benefits is
related to prior earnings.[Footnote 5] The DI program is funded
primarily through a payroll tax required by the Federal Insurance
Contributions Act (FICA) and is levied on most workers, and DI benefits
are based on an individual's career earnings. The SSI program is a
means-tested entitlement program that provides monthly benefits to
aged, blind, or disabled individuals who have very limited income and
assets. The SSI program is funded through general revenues. Unlike the
DI benefit, the federal SSI benefit is a flat amount (adjusted for
other income the individual may have) and is not related to prior
earnings.
During the 1970s, as the number of disability awards and costs were
increasing significantly for the DI program, Congress enacted
legislation providing various work incentives to encourage
beneficiaries to return to work and, potentially, leave the benefit
rolls. To further these efforts, in 1980, Congress provided SSA with
the authority to conduct demonstration projects to evaluate the
effectiveness of policy alternatives that could encourage both DI and
SSI beneficiaries to re-enter the workforce.[Footnote 6] Under this
authority, SSA can temporarily waive DI and SSI program rules,
including rules regarding program eligibility and benefit
administration, in order to test the effect certain program changes
would have on beneficiaries' return-to-work rates and the size of the
DI and SSI benefit rolls. Because Congress has historically granted SSA
DI demonstration authority on a temporary basis, it is subject to
periodic review and renewal. Since first providing this authority in
1980, Congress has renewed it in 1986, 1989, 1994, 1999, and 2004.
However, in 2004, Congress only extended SSA's DI demonstration
authority through December 2005. As a result, SSA cannot initiate new
DI demonstration projects but can continue those projects that were
initiated on or before the December 2005 expiration date. However, SSA
can continue to initiate demonstration projects under its SSI
authority.[Footnote 7] In 2008, SSA requested that Congress reauthorize
its DI demonstration authority, and a bill was introduced to do so.
SSA's DI demonstration projects--unlike other SSA research activities-
-are paid for via the DI trust fund. Therefore, SSA is not required to
obtain congressional approval for DI demonstration expenditures,
although it is required to obtain approval from the Office of
Management and Budget for an annual apportionment of the trust funds
for these demonstrations. Unlike the DI projects, SSI demonstration
projects are funded from SSA's overall congressional research
appropriation. Although SSA's DI and SSI demonstration authorities are
separate, the agency's disability demonstration projects are sometimes
jointly authorized when they involve both DI and SSI beneficiaries and
applicants. When a demonstration project is conducted jointly under the
DI and SSI demonstration authorities, funding for the project is split
between trust fund (i.e., DI) and appropriated (i.e., SSI) sources.
SSA's Office of Program Development and Research (OPDR) provides
program analysis in support of DI and SSI programs. As part of their
responsibilities, OPDR--sometimes with the assistance of outside
research organizations--identifies the requirements for individual
disability program demonstration projects, including the basic
objectives, scope, and methodological standards for these projects.
OPDR project officers are primarily responsible for overseeing the
projects to ensure that they meet SSA's technical and programmatic
requirements.
As we have previously reported, demonstration projects examining the
impact of social programs aim to provide evidence of the feasibility or
effectiveness of a new approach or practice and are inherently complex
and difficult to conduct.[Footnote 8] Measuring outcomes, ensuring the
consistency and quality of data collected at various site locations,
establishing a causal connection between outcomes and program
activities, and separating out the influence of extraneous factors can
raise formidable technical and logistical problems. Although the
legislation granting SSA its demonstration authority does not require
the use of particular methodological approaches, SSA has historically
recognized that the law's general requirement for its demonstration
projects requires SSA to conduct its projects in a rigorous manner that
provides the agency with a reliable basis for making policy
recommendations. According to professional research standards, a
rigorous study should include a clearly stated research question and
methodology, including plans for data collection and evaluation, as
well as appropriate controls to determine if a relationship exists
between observed outcomes and the program change under examination (see
app. I).
As part of our prior work related to SSA DI demonstration authority we
reviewed two DI demonstration projects that SSA conducted in the late
1980s and early 1990s.[Footnote 9] At that time, we found that SSA had
not used its demonstration authority to extensively evaluate a wide
range of DI policy areas dealing with return to work and found that the
demonstration projects had little impact on SSA's and Congress's
consideration of DI policy issues. To facilitate close congressional
oversight and provide greater assurance that SSA will make effective
use of its authority, we recommended that SSA develop a formal agenda
for its demonstration projects, establish an expert panel to guide the
design and implementation of its demonstration projects, and establish
formal processes to ensure full consideration of demonstration project
results. We also identified several matters for Congress to consider,
including continuation of DI demonstration authority on a temporary
basis, establishment of additional reporting requirements for
demonstrations, and clearer specification of the methodological and
evaluation requirements of demonstrations.
SSA Has Used Its Demonstration Authority to Initiate Several Projects
Since 1998, but Efforts Have Yielded Limited Information:
Over the last decade, SSA has initiated 14 demonstration projects to
test policy and program changes, of which SSA has completed 4,
cancelled 5, and had 5 projects in progress as of August 2008. In
total, SSA had spent about $155 million on these demonstration projects
as of April 2008, and officials anticipate spending another $220
million in the coming years on those projects currently in progress.
However, these projects have yielded limited information for
influencing program and policy decisions. We found that SSA did not
conduct impact evaluations for two of its completed projects and
cancelled five projects prior to conducting formal evaluations; thus,
limited information is available.
Since 1998, SSA Has Initiated 14 DI and SSI Demonstration Projects in
an Effort to Test Policy and Administrative Changes:
Since 1998, SSA has initiated 14 projects under its demonstration
authority to test both DI and SSI program changes--6 related to DI, 6
related to SSI, and 2 examining both programs jointly (see table 1 for
an overview of each project). As of April 2008, SSA spent $80.3 million
on its completed projects, $7.1 million on cancelled projects, and
$68.2 million on those currently in progress.
Table 1: Overview of Demonstration Projects Initiated Since 1998:
Project: Projects conducted under DI demonstration authority:
Accelerated Benefits;
Policy issues studied: Projects conducted under DI demonstration
authority: Focused on provision of immediate health benefits to newly
entitled DI beneficiaries to test whether this results in improved
health and better return to work outcomes;
Status: Projects conducted under DI demonstration authority: Initiated
2004, in progress;
Funds spent (as of April 2008): Projects conducted under DI
demonstration authority: $3.6 million from the DI trust fund;
Information available: Projects conducted under DI demonstration
authority: Not yet available;
Completion expected 2011.
Project: Projects conducted under DI demonstration authority: Benefit
Offset - 4 State Pilot;
Policy issues studied: Projects conducted under DI demonstration
authority: Focused on assessment of pilot of a benefit offset to inform
the design of the Benefit Offset National Demonstration (BOND);
Status: Projects conducted under DI demonstration authority: Initiated
2003, in progress;
Funds spent (as of April 2008): Projects conducted under DI
demonstration authority: $4.5 million from the DI trust fund;
Information available: Projects conducted under DI demonstration
authority: Yes;
preliminary process and outcome information available showing some
increase in earnings by pilot participants and identifying operational
challenges, such as income reporting, encountered when implementing the
benefit offset;
Completion expected 2009.
Project: Projects conducted under DI demonstration authority: Benefit
Offset National Demonstration (BOND);
Policy issues studied: Projects conducted under DI demonstration
authority: Focused on assessment of a benefit offset that would allow
beneficiaries to remain eligible for DI benefits when earning above the
Substantial Gainful Activity level while offsetting these benefits by
$1 for every $2 in earnings;
it would also assess benefit counseling in conjunction with the offset;
Status: Projects conducted under DI demonstration authority: Initiated
1999, in progress;
Funds spent (as of April 2008): Projects conducted under DI
demonstration authority: $6.7 million from the DI trust fund;
Information available: Projects conducted under DI demonstration
authority: Not yet available; Completion date unknown[A].
Project: Projects conducted under DI demonstration authority:
California RISE (CA HIV/AI) Demonstration;
Policy issues studied: Projects conducted under DI demonstration
authority: Focused on provision of additional health and employment
services for DI beneficiaries with Human Immunodeficiency Virus (HIV)
or Auto-Immune (AI) deficiencies;
Status: Projects conducted under DI demonstration authority: Initiated
2003, cancelled 2007;
Funds spent (as of April 2008): Projects conducted under DI
demonstration authority: $2.1 million from the DI trust fund;
Information available: Projects conducted under DI demonstration
authority: No; project was not implemented.
Project: Projects conducted under DI demonstration authority: Early
Intervention Demonstration;
Policy issues studied: Projects conducted under DI demonstration
authority: Focused on provision of employment services to DI applicants
as an alternative to the DI program;
Status: Projects conducted under DI demonstration authority: Initiated
2000, cancelled 2007;
Funds spent (as of April 2008): Projects conducted under DI
demonstration authority: $2 million from the DI trust fund;
Information available: Projects conducted under DI demonstration
authority: No; project was not implemented.
Project: Projects conducted under DI demonstration authority: Mental
Health Treatment Study;
Policy issues studied: Projects conducted under DI demonstration
authority: Focused on provision of mental health and employment
services to DI beneficiaries with a primary impairment of schizophrenia
or affective disorder;
Status: Projects conducted under DI demonstration authority: Initiated
2003, in progress;
Funds spent (as of April 2008): Projects conducted under DI
demonstration authority: $17.2 million from the DI trust fund;
Information available: Projects conducted under DI demonstration
authority: Not yet available; Completion expected 2011.
Project: Projects conducted under SSI demonstration authority:
Disability Program Navigator,; conducted with the Department of Labor
(DOL);
Policy issues studied: Projects conducted under DI demonstration
authority: Focused on expansion of capacity in workforce investment
system to serve people with disabilities;
Status: Projects conducted under DI demonstration authority: Initiated
2002, completed 2007;
Funds spent (as of April 2008): Projects conducted under DI
demonstration authority: $12 million from appropriations;
Information available: Projects conducted under DI demonstration
authority: No; project was not evaluated; SSA officials told us that
DOL is pursuing a separate evaluation.
Project: Projects conducted under SSI demonstration authority: Early
Identification and Intervention Demonstration;
Policy issues studied: Projects conducted under DI demonstration
authority: Focused on provision of funding for programs that increase
developmental screening and early identification of children with
developmental delays and/or disabilities;
Status: Projects conducted under DI demonstration authority: Initiated
2004, cancelled 2007;
Funds spent (as of April 2008): Projects conducted under DI
demonstration authority: $50,000 from appropriations;
Information available: Projects conducted under DI demonstration
authority: No; project was not implemented.
Project: Projects conducted under SSI demonstration authority: Florida
Freedom Initiative;
Policy issues studied: Projects conducted under DI demonstration
authority: Focused on assessment of waivers of SSI program rules to
foster greater self-sufficiency among SSI recipients in the state of
Florida;
Status: Projects conducted under DI demonstration authority: Initiated
2003, completed 2007;
Funds spent (as of April 2008): Projects conducted under DI
demonstration authority: $100,000 from appropriations;
Information available: Projects conducted under DI demonstration
authority: No; project was not evaluated because too few participants
enrolled to produce enough data for the evaluation.
Project: Projects conducted under SSI demonstration authority: Homeless
Outreach Projects and Evaluation;
Policy issues studied: Projects conducted under DI demonstration
authority: Focused on provision of services to people with disabilities
who are chronically homeless, including assistance with filing
disability benefit applications and accessing mainstream treatment and
services;
Status: Projects conducted under DI demonstration authority: Initiated
2003, completed 2008;
Funds spent (as of April 2008): Projects conducted under DI
demonstration authority: $21.4 million from appropriations;
Information available: Projects conducted under DI demonstration
authority: Yes; impact information available showing that project
participants received faster decisions from SSA about whether to
allow/deny benefits; however, the contractor identified limitations
with its evaluation because a similar program that another federal
agency initiated after the SSA project began could have influenced the
impact on the Homeless Outreach Projects and Evaluation participants.
Project: Projects conducted under SSI demonstration authority: Interim
Medical Benefits;
Policy issues studied: Projects conducted under DI demonstration
authority: Focused on provision of medical benefits to child and adult
SSI applicants who are likely to be awarded benefits following the
disability determination process;
Status: Projects conducted under DI demonstration authority: Initiated
2004, cancelled 2007;
Funds spent (as of April 2008): Projects conducted under DI
demonstration authority: $0;
Information available: Projects conducted under DI demonstration
authority: No; project was not implemented.
Project: Projects conducted under SSI demonstration authority:
Pediatric Medical Unit (PMU);
Policy issues studied: Projects conducted under DI demonstration
authority: Focused on provision of pediatric expertise to disability
adjudicators to assist in making determinations on childhood disability
cases;
Status: Projects conducted under DI demonstration authority: Initiated
2004, cancelled 2008;
Funds spent (as of April 2008): Projects conducted under DI
demonstration authority: $2.9 million from appropriations;
Information available: Projects conducted under DI demonstration
authority: Yes; some information from implementation experiences
identified potential problems with childhood disability applications;
PMU information could inform future research on interdisciplinary
approaches to disability determination.
Project: Projects jointly authorized under DI and SSI demonstration
authorities: State Partnership Initiative;
Policy issues studied: Projects conducted under DI demonstration
authority: Focused on assessment of waivers of SSI benefit rules and
other employment support interventions for DI and SSI beneficiaries
provided in 12 states;
Status: Projects conducted under DI demonstration authority: Initiated
1998, completed 2006;
Funds spent (as of April 2008): Projects conducted under DI
demonstration authority: $46.7 million,; $8.5 million from the DI trust
fund and $38.2 million from appropriations;
Information available: Projects conducted under DI demonstration
authority: Yes; some information from implementation experiences, as
well as impact information showing that benefits counseling and
employment services sometimes increased employment, but reduced
earnings or did not affect them. However, the contractor identified
some limitations with its impact evaluation because it is based on
preliminary data.
Project: Projects jointly authorized under DI and SSI demonstration
authorities: Youth Transition Demonstration;
Policy issues studied: Projects conducted under DI demonstration
authority: Focused on provision of services to help youth SSI
beneficiaries and other youth with disabilities make the transition
from school to work and/or continue their education; it would also
assess various waivers of disability program rules;
Status: Projects conducted under DI demonstration authority: Initiated
2003, in progress;
Funds spent (as of April 2008): Projects conducted under DI
demonstration authority: $36.2 million,; $3.7 million from the DI Trust
fund and $32.5 million from appropriations;
Information available: Projects conducted under DI demonstration
authority: Not yet available; Completion expected 2014.
Project: Total;
Policy issues studied: Projects conducted under DI demonstration
authority: [Empty];
Status: Projects conducted under DI demonstration authority: 4
completed; 5 cancelled; 5 in progress;
Funds spent (as of April 2008): Projects conducted under DI
demonstration authority: $155.5 million; ($48.3 million from DI trust
fund, $107.2 million from appropriations);
Information available: Projects conducted under DI demonstration
authority: Impact information-2 of 14; Other information-3 of 14; No
information-6 of 14;
Information not yet available-4 of 14.
Source: GAO analysis of SSA's demonstration projects and budget data.
Notes: In the column Information Available, projects sum to greater
than 14 because 1 project yielded impact information and other
information. In some cases, SSA officials provided us with estimated
initiation dates because the agency had not maintained complete
documentation on all demonstrations projects initiated since 1998. For
the State Partnership Initiative, SSA officials were unable to verify
the actual budget amounts spent in 1998 and 1999 for this project
because SSA no longer maintains this information.
[A] As of August 2008, SSA had not determined an expected completion
date for the BOND project.
[End of table]
While SSA initiated 14 projects over the past 10 years, the agency has
only completed 4 of them to date. These completed projects generally
focused on reducing individuals' dependency on the SSI program by
primarily testing program waivers and other changes in program
administration, as outlined in its SSI demonstration
authority.[Footnote 10] We also found that SSA cancelled five projects
during this period, citing significant challenges that would have
limited the agency's ability to obtain reliable information from them.
SSA had five projects in progress as of August 2008. These projects
generally addressed topics outlined in the authorizing legislation for
DI demonstrations and included strategies to return individuals to work
and reduce the growth of certain subgroups of beneficiaries. For
example, the legislation required projects to test various incentives
to increase DI beneficiaries work activity.[Footnote 11] In addition,
the Ticket to Work and Work Incentives Improvement Act of 1999 provided
demonstration authority for a benefit reduction, rather than complete
benefit termination, when beneficiaries had earnings that exceeded a
certain level.[Footnote 12] To address this provision, SSA initiated
the Benefit Offset National Demonstration (BOND) shortly after passage
of the statute. Another project in progress, the Mental Health
Treatment Study, is focused on identifying strategies for providing
mental health treatment and employment supports for certain DI
beneficiaries with mental illnesses.[Footnote 13] As of April 2008,
officials estimated that the total costs for the five projects
currently in progress would be about $288 million--about $220 million
more than the $68 million already expended (see table 2).
Table 2: Total Estimated Costs for Demonstration Projects Currently in
Progress:
Project: Accelerated Benefits;
Year of initiation: 2004;
Completion date: 2011;
Total estimated cost: $45 million.
Project: Benefit Offset - 4 State Pilot;
Year of initiation: 2005;
Completion date: 2009;
Total estimated cost: $8.5 million.
Project: Benefit Offset National Demonstration (BOND);
Year of initiation: 1999;
Completion date: unknown[A];
Total estimated cost: $95 million.
Project: Mental Health Treatment Study;
Year of initiation: 2003;
Completion date: 2011;
Total estimated cost: $52.5 million.
Project: Youth Transition Demonstration;
Year of initiation: 2003;
Completion date: 2014;
Total estimated cost: $86.7 million; ($78 million from appropriations
and $8.7 million from DI trust fund).
Project: Total;
Year of initiation: [Empty];
Completion date: [Empty];
Total estimated cost: $287.7 million; ($209.7 million from DI trust
fund, $78 million from appropriations).
Source: GAO analysis of SSA's demonstration projects and budget data.
Notes: Expected total cost figures provided by SSA are estimates.
[A] As of August 2008, SSA has not determined an expected completion
date for the BOND project.
[End of table]
Demonstration Projects Have Yielded Limited Information to Influence
Program and Policy Changes:
Despite using its demonstration authority to examine various issues,
SSA's demonstration projects have yielded limited information for
influencing program and policy decisions. As required under its
demonstration authority, SSA's demonstration projects should be
conducted in such a way to permit a thorough evaluation of alternative
methods under consideration.[Footnote 14] However, we found that SSA
had not conducted impact evaluations--assessments of a project's
effects compared to what would have happened in its absence--for two of
its completed projects, the Disability Program Navigator and the
Florida Freedom Initiative. Thus, no information about the impacts of
the program and policy changes being tested was available for making
decisions about disability policy. The Disability Program Navigator
project, which SSA conducted with the Department of Labor (DOL), was
not evaluated because the evaluation contractor could not meet SSA's
data security requirements established after the project was already in
progress, and thus could not access the necessary data.[Footnote 15]
SSA developed a plan to evaluate the Florida Freedom Initiative after
they became concerned about the state's evaluation plans. However, SSA
did not conduct an evaluation because staff at the state level
conducting the project did not enroll enough participants in the
project to meet sample size requirements. Thus, there was not enough
data available to conduct a reliable evaluation.
Furthermore, SSA intended to evaluate the impacts of policies and
programs being tested in five other projects but could not do so
because the significant challenges those projects faced led SSA to
cancel them in the early stages. Specifically, four of these projects
were cancelled prior to implementation, and thus no data was available
to conduct the evaluations of those policy and programs being tested.
The other cancelled project--the Pediatric Medical Unit demonstration-
-was partially implemented but not evaluated because the project did
not establish the comparison group needed for the analysis. The project
also did not it enroll enough participants at some implementation sites
to meet the sample size requirements needed to generate data for a
reliable evaluation. However, SSA was able to obtain some preliminary
information on how the project's strategy appeared to be working at two
site locations and is considering how to use it.
Although SSA did conduct evaluations for two of the completed projects-
-the Homeless Outreach Projects and Evaluation (HOPE) project and the
State Partnership Initiative project--we found that these projects also
yielded little information about the impacts of the strategies being
tested because the reported evaluation results could not reliably
demonstrate the projects' effects. For example, an outcome evaluation
of the HOPE project showed that although disability program applicants
assisted by the project received faster decisions from SSA about
whether to allow or deny benefits, another federal agency initiated a
similar project even though the HOPE project was under way. Therefore,
SSA's evaluation results were weakened, in part because researchers
could not separate the effects of the SSA project from the effects of
the other federal project. While SSA did not obtain reliable impact
evaluation results from this project, agency officials told us that
they did obtain a great deal of information about the process of
conducting this type of demonstration project.
For the State Partnership Initiative, we found that SSA did conduct an
impact evaluation when the project ended, but data available at that
time were incomplete, and thus information about the impact of the
project may not be a reliable indicator of the project's long-term
effects. SSA's contractors recommended that a final evaluation be
conducted once all the data were collected to assess whether the
preliminary results were valid. However, SSA management chose not to
pursue further evaluation because the preliminary results indicated
that the project was not successful at increasing earnings enough to
allow individuals returning to work to exit the rolls and no longer be
dependent on disability benefits. Nonetheless, SSA's contractors and
agency officials said that lessons learned from implementing the State
Partnership Initiative have influenced the agency's subsequent approach
to return beneficiaries to work. For example, SSA used the job
descriptions of benefits planners, as well as data systems from this
project, to design the agency's national Benefits Planning and Outreach
program.[Footnote 16]
SSA has also begun to obtain some information from one of the five
projects currently under way. SSA has used preliminary results of the
Benefit Offset - 4 State Pilot to aid in the design of the BOND
project. Each of the four states conducting this pilot has provided an
interim report to SSA detailing lessons learned from the implementation
of this project. Because the pilot and BOND both test a benefit offset
in conjunction with other DI program changes, SSA officials and the
BOND project contractor believe that states' experiences implementing
this pilot will help SSA identify and resolve operational issues before
rolling BOND out nationally. In addition, the four states have
conducted preliminary impact evaluations for the pilot project and
expect to complete final evaluations once the project's implementation
and data collection phases are over.
SSA also plans to conduct impact evaluations of the other demonstration
projects it had in progress as of August 2008. While they have the
potential to yield reliable results, it is too early to tell whether
they will ultimately be useful for informing DI and SSI policy and
program changes. These projects address issues outlined in the
demonstration authority statutes and disability programs more broadly,
and SSA officials believe they will yield useful information. For
example, SSA officials anticipate that the results of the Accelerated
Benefits demonstration project could help policymakers determine
whether to eliminate the 24 month waiting period for Medicare that DI
beneficiaries encounter under current law. SSA officials also
anticipate that demonstration projects in progress could yield key
information on how to improve outcomes for certain subgroups of
beneficiaries. For example, SSA officials said that the Youth
Transition Demonstration, which targets young people with disabilities
as they transition from school to work, could identify strategies for
improving the self-sufficiency of these beneficiaries and thus reduce
their dependence on the disability programs. Most of SSA's current
demonstration projects are expected to continue until 2010 or later
before generating final evaluation results that could inform changes to
disability program policy.
SSA Has Taken Steps to Improve Its Projects but Continues to Lack
Management Controls:
SSA has taken steps to improve its demonstration projects, in part by
applying more rigorous methodologies than it did for the projects SSA
initiated prior to 1998; however, it has not fully implemented GAO's
recommendations from 2004 and does not have written policies and
procedures in place to ensure that projects are routinely reviewed and
effectively managed so that they yield reliable information about their
impacts. As a result, some projects faced challenges, such as low
participation rates or data collection problems, which were significant
enough to hinder the agency's ability to evaluate the projects' impacts
as planned. In addition, without comprehensive written policies and
procedures governing how SSA manages and operates its demonstration
program, the project objectives, designs, and evaluation plans may be
impacted during times of organizational change.
SSA Has Taken Steps to Improve Its Demonstration Projects:
SSA has improved its demonstration projects by applying more rigorous
methodologies than it did prior to 1998, contracting with professional
researchers and appointing new management for the program.
Specifically, SSA is applying more rigorous evaluation methodologies to
the projects it has initiated since 1998 than it did to the projects
initiated in the late 1980s and early 1990s. At the time of our prior
report, SSA officials acknowledged that the limited rigor of those
earlier projects reduced their usefulness and indicated that the agency
had placed a new emphasis on ensuring that its projects going forward
would be more rigorously designed.[Footnote 17] Of the 14 projects that
SSA has initiated since 1998, 13 were early enough in the planning or
design stages at that time to give SSA an opportunity to make such
improvements.[Footnote 18] Since that time, SSA has completed much of
the design work for its 14 projects and provided us with detailed
design information for 12 of them, enabling us to assess the rigor of
these projects' designs for our current review.
Our current analysis shows that SSA did use more rigorous methodologies
for the projects initiated over the last decade than for its earlier
projects. SSA is now using methodologies known as experimental or quasi-
experimental designs, which are commonly used by research professionals
conducting demonstration projects to estimate the impacts of program or
policy changes. On the basis of our assessment, we determined that 11
of the 12 projects' designs were strong or reasonable when assessed
against professional research standards (see table 3).[Footnote 19] We
compared each project's design against GAO and recognized academic
criteria for conducting evaluation research, which were also consistent
with statutory requirements that DI projects be generally sufficient in
scope and planned in such a way to permit a thorough evaluation of the
program or policy changes under consideration. We also determined that
the projects currently under way could provide some reliable results if
implemented and evaluated as designed.
Despite this progress, we found that SSA did not always meet additional
DI and SSI statutory requirements regarding the general applicability
of the projects' results and the use of expert advice, respectively.
The authorizing statute for DI demonstration projects requires that the
results derived from the projects will obtain generally in the
operation of the disability program.[Footnote 20] While one of the six
DI projects, the BOND project, has been designed to yield nationally
representative information about the impacts of the project, the
statute does not require that the results be applicable to all DI
beneficiaries nationwide.[Footnote 21] However, the results should
apply to a larger group of beneficiaries than just those that
participated in the demonstration project, and SSA may be able to apply
the results from three other DI projects--the Accelerated Benefits
demonstration project, the Benefit Offset - 4 State Pilot, and the
Mental Health Treatment Study--more generally because it plans to
implement and evaluate the projects in a consistent manner at multiple
sites. In addition, one of the two jointly authorized projects--the
State Partnership Initiative--did not yield generally applicable
results because the projects were not implemented consistently across
each state.[Footnote 22] The authorizing statute for SSI projects
requires the Commissioner of SSA to obtain the advice and
recommendations of specialists who are competent to evaluate the
proposed projects as to the soundness of their design, the
possibilities of securing productive results, the adequacy of resources
to conduct the proposed research or demonstrations, and their
relationship to other similar research or demonstrations already
completed or in process before entering into a contract, grant, or
cooperative agreement for the project.[Footnote 23] However, SSA
obtained advice from experts for only two of the six SSI projects.
Finally, SSA generally met other design criteria required by statute
for the BOND project (see app. II).[Footnote 24]
Table 3: Assessment of Demonstration Projects' Designs against
Professional Research Standards and Statutory Requirements:
Project: Projects conducted under DI demonstration authority:
Accelerated Benefits (In progress);
Methodology: Experimental;
Assessment against professional research standards[A]: Strong;
Assessment against additional DI statutory requirements[B]: Meets to
some extent;
Assessment against additional SSI statutory requirements[C]: NA.
Project: Projects conducted under DI demonstration authority: Benefit
Offset - 4 State Pilot[D[(IN] In progress);
Methodology: Experimental;
Assessment against professional research standards[A]: Reasonable;
Assessment against additional DI statutory requirements[B]: Meets to
some extent;
Assessment against additional SSI statutory requirements[C]: NA.
Project: Projects conducted under DI demonstration authority: Benefit
Offset National Demonstration (In progress);
Methodology: Experimental;
Assessment against professional research standards[A]: Strong;
Assessment against additional DI statutory requirements[B]: Meets;
Assessment against additional SSI statutory requirements[C]: NA.
Project: Projects conducted under DI demonstration authority:
California RISE (CA HIV/AI) (Cancelled);
Methodology: Experimental;
Assessment against professional research standards[A]: Reasonable;
Assessment against additional DI statutory requirements[B]: Does not
meet;
Assessment against additional SSI statutory requirements[C]: NA.
Project: Projects conducted under DI demonstration authority: Early
Intervention Demonstration (Cancelled);
Methodology: Experimental;
Assessment against professional research standards[A]: Strong;
Assessment against additional DI statutory requirements[B]: Does not
meet;
Assessment against additional SSI statutory requirements[C]: NA.
Project: Projects conducted under DI demonstration authority: Mental
Health Treatment Study (In progress);
Methodology: Experimental;
Assessment against professional research standards[A]: Reasonable;
Assessment against additional DI statutory requirements[B]: Meets to
some extent;
Assessment against additional SSI statutory requirements[C]: NA.
Project: Projects conducted under SSI demonstration authority:
Disability Program Navigator (Completed);
Methodology: Unknown;
Assessment against professional research standards[A]: Could not
assess;
Assessment against additional DI statutory requirements[B]: NA;
Assessment against additional SSI statutory requirements[C]: Meets to
some extent.
Project: Projects conducted under SSI demonstration authority: Early
Identification and Intervention Demonstration (Cancelled);
Methodology: Unknown;
Assessment against professional research standards[A]: Could not
assess;
Assessment against additional DI statutory requirements[B]: NA;
Assessment against additional SSI statutory requirements[C]: Does not
meet.
Project: Projects conducted under SSI demonstration authority: Florida
Freedom Initiative (Completed);
Methodology: Quasi- experimental;
Assessment against professional research standards[A]: Relatively weak;
Assessment against additional DI statutory requirements[B]: NA;
Assessment against additional SSI statutory requirements[C]: Does not
meet.
Project: Projects conducted under SSI demonstration authority: Homeless
Outreach Projects and Evaluation (Completed);
Methodology: Quasi-experimental;
Assessment against professional research standards[A]: Reasonable;
Assessment against additional DI statutory requirements[B]: NA;
Assessment against additional SSI statutory requirements[C]: Does not
meet.
Project: Projects conducted under SSI demonstration authority: Interim
Medical Benefits (Cancelled);
Methodology: Quasi- experimental;
Assessment against professional research standards[A]: Reasonable;
Assessment against additional DI statutory requirements[B]: NA;
Assessment against additional SSI statutory requirements[C]: Does not
meet.
Project: Projects conducted under SSI demonstration authority:
Pediatric Medical Units (Cancelled);
Methodology: Experimental;
Assessment against professional research standards[A]: Reasonable;
Assessment against additional DI statutory requirements[B]: NA;
Assessment against additional SSI statutory requirements[C]: Meets.
Project: Projects jointly authorized under DI and SSI demonstration
authorities: State Partnership Initiative[D,E[(completed);
Methodology: Mixed-experimental;
quasi-experimental;
Assessment against professional research standards[A]: Mixed-some
strong;
some reasonable;
Assessment against additional DI statutory requirements[B]: Does not
meet;
Assessment against additional SSI statutory requirements[C]: Meets to
some extent.
Project: Projects jointly authorized under DI and SSI demonstration
authorities: Youth Transition Demonstration[E[(IN] In progress);
Methodology: Experimental;
Assessment against professional research standards[A]: Strong;
Assessment against additional DI statutory requirements[B]: Meets to
some extent;
Assessment against additional SSI statutory requirements[C]: Meets.
Source: GAO analysis of SSA's demonstration projects' designs.
[A] Appendix I discusses professional research standards in further
detail.
[B] The Social Security Act § 234(b) requires that in addition to the
demonstration projects being of sufficient scope and carried out on a
wide enough scale to permit a thorough evaluation of the alternative
methods under consideration, the projects should also give assurance
that the results derived from the projects obtain generally in the
operation of the disability program.
[C] The Social Security Act § 1110(a)(1) requires the Commissioner of
SSA to obtain the advice and recommendations of specialists who are
competent to evaluate the proposed projects as to the soundness of
their design, the possibilities of securing productive results, the
adequacy of resources to conduct the proposed research or
demonstrations, and their relationship to other similar research or
demonstrations already completed or in process.
[D] Multiple evaluations were conducted for this demonstration project.
[E] This project must meet the requirements specified in both the DI
and SSI statutes because it is authorized under both of them.
[End of table]
Evaluation methodologies:
Experimental designs involve random assignment of study participants to
either a treatment group or a control group. The treatment group is
subjected to the new program or policy, and the control group is not.
The strength of the experimental design is in its assurance that those
who experience the treatment are like those that do not experience the
treatment in all important ways, except for the difference of receiving
the treatment itself. Thus, after the study is over, researchers can
usually attribute observed differences in outcomes between the
treatment and control groups to the new policy or program. Research
experts generally consider experimental designs to be the "gold
standard" for impact evaluations. However, they may not always be
feasible or may not necessarily provide the optimal approach for
conducting a demonstration because of various implementation or ethical
considerations. Quasi-experimental methods-- reasonable but less
rigorous alternatives to experimental designs-- include study
participants that receive a treatment, but do not involve the use of
randomized control groups and instead rely on the nonrandom selection
of a comparison group of nonparticipants with characteristics similar
to the study participants. Quasi-experimental methods may instead be
preferable or may need to be used in conjunction with experimental
methods to produce the most rigorous results. For instance, quasi-
experimental methods may be preferred when there is a chance that a
randomized experimental approach will not remain intact, such as when
participants in the control group are likely to receive elements of the
policy or program under evaluation. However, because quasi-experiments
do not randomly assign participants to treatment and control groups,
the assurance that the two groups are actually similar in all critical
ways except for receiving the treatment is missing, and they cannot
establish cause and effect between the program under study and the
observed outcomes. Nonetheless, they can be used to determine whether a
relationship exists between the strategy being tested and the observed
outcomes.
To further improve the demonstration projects' planning and
methodological rigor, SSA has used external research professionals to
work with the agency on the design, implementation, or evaluation of 12
of the 14 projects. SSA officials have acknowledged the need for
additional expertise to design and implement methodologically rigorous
demonstration projects.[Footnote 25] Thus, SSA has awarded, or planned
to award, contracts and cooperative agreements to research consultants
and universities with such expertise to evaluate 12 of its 14 projects
(see app. III). In nine cases, these researchers also worked on the
design or implementation of the projects. For example, for the
Accelerated Benefits demonstration project, an SSA research contractor
also designed how and where the project would be conducted and managed
its implementation so that the data needed for the evaluation it plans
to conduct will be available. We also found that SSA and most of these
researchers communicated regularly when collaborating on these projects
and researchers submitted monthly or quarterly progress reports to SSA,
which included information on expenditures, progress, and areas of
concern that needed to be addressed.
SSA also appointed new program management in 2007. Since that time, the
new management team has conducted an internal review of the 10
demonstration projects that were under way at the time of their
appointment. SSA officials told us that all projects underwent a
thorough review that was conducted by the Acting Associate Commissioner
for Program Development and Research and others with appropriate
expertise. Documents we obtained indicate that the review identified
the projects' strengths, weaknesses, and whether they were likely to
yield reliable, useful results. For example, SSA considered whether a
project's sample size and site selection were appropriate, if it had
been implemented in accordance with its design, and if it faced any
challenges that would prevent researchers from conducting a rigorous
evaluation of its results. SSA concluded that five of its projects
would continue, expecting that they were likely to yield reliable
impact information; these five projects are currently in
progress.[Footnote 26] However, the agency did need to make significant
changes to strengthen one of the five projects' designs--the BOND
project. At an earlier point in the design phase, SSA expanded the BOND
project's scope to include multiple components in addition to the
benefit offset, such as a health benefits package. The new management
team subsequently determined that the cost estimates and program
complexity associated with several of those components raised questions
about the feasibility of implementing the project and significantly
scaled back the scope of the study.
For the other five projects the new management was reviewing, SSA
determined they faced significant limitations or challenges--such as
poorly chosen implementation sites and low participation--that made it
highly unlikely for them to obtain reliable results or would have been
duplicative of other ongoing research (see table 4). Thus, SSA
cancelled those projects in 2007 and 2008. Although the only
information SSA has obtained from these projects are some lessons
learned from the Pediatric Medical Unit project, the agency projected
that it would have spent another $82 million had it not canceled
them.[Footnote 27] Based on the information SSA provided us about the
challenges facing the projects and the expected future costs of
conducting them, its decisions to cancel the five projects appear to
have been data-driven and reasonable.
Table 4: Reasons for Demonstration Projects' Cancellations:
Project: California RISE (CA HIV/AI) Demonstration;
Program focus: DI;
Phase of project when cancelled: Design;
Amount spent prior to cancellation: $2.1 million;
Reasons SSA cited for cancellation: Reliable evaluation results were
unlikely because similar services were already available in the state
selected for implementation, potentially introducing control group
contamination and problems recruiting enough participants for the
study. Also, nationally representative results were unlikely because of
unique conditions in the state; Significant coordination problems also
existed between the implementation and evaluation components of the
project.
Project: Early Identification and Intervention Demonstration;
Program focus: SSI;
Phase of project when cancelled: Solicitation of proposals;
Amount spent prior to cancellation: $50,000;
Reasons SSA cited for cancellation: Reliable evaluation results were
unlikely because the Request for Proposal for this study did not
require the use of a rigorous evaluation methodology; The project also
had little connection to SSA's mission, and there were competing needs
for SSA's research resources.
Project: Early Intervention Demonstration;
Program focus: DI;
Phase of project when cancelled: Design; Amount spent prior to
cancellation: $2 million;
Reasons SSA cited for cancellation: This project was initiated as a
stand-alone demonstration project but was incorporated into the BOND
project in 2005. It added significant complexity to BOND and became
very costly to carry it out.
Project: Interim Medical Benefits;
Program focus: SSI;
Phase of project when cancelled: Prior to solicitation of proposals;
Amount spent prior to cancellation: $0;
Reasons SSA cited for cancellation: This project was very similar to
the Accelerated Benefits demonstration project, and information from it
would have been duplicative.
Project: Pediatric Medical Unit (PMU);
Program focus: SSI;
Phase of project when cancelled: Implementation;
Amount spent prior to cancellation: $2.9 million;
Reasons SSA cited for cancellation: While the design for this project
was reasonable in that it specified the use of an experimental design
and establishment of a comparison group, when SSA implemented the
project, the comparison group was not established. This prevented SSA
from obtaining the level of rigor it wanted for this project;
Insufficient sample size also made reliable evaluation results unlikely
because some of the participating Disability Determination Services
were not referring enough cases to the PMU for review. In addition, the
low referral rates made the cost per participant very high.
Project: Total;
Program focus: [Empty];
Phase of project when cancelled: [Empty];
Amount spent prior to cancellation: $7.05 million; ($4.1 million from
DI trust fund, $2.95 million from appropriations);
Reasons SSA cited for cancellation: [Empty].
Source: GAO analysis of SSA's demonstration projects and budget data.
[End of table]
Further, SSA consolidated its research expertise by merging the Office
of Disability and Income Security Programs (DISP) with SSA's Office of
Policy in February 2008, creating the Office of Retirement and
Disability Policy. As of June 2008, each office's research unit remains
intact and no formal organizational changes were made to the
demonstration program, but agency officials told us that the merger has
facilitated communication and strengthened relationships between
researchers within the agency.[Footnote 28] For example, experts from
the former Office of Policy's research unit routinely review and
provide input on the demonstration projects' designs and evaluations.
SSA Continues to Lack Policies, Procedures, and Other Mechanisms to
Ensure Projects Yield Reliable Information:
While SSA is taking steps to generally improve its demonstration
projects' designs and address specific project limitations, it does not
have policies, procedures, and mechanisms to ensure that demonstration
projects will yield reliable information about the impacts of the
programs they are testing. According to internal control standards in
the federal government, federal agencies should have policies,
procedures, and mechanisms in place to provide reasonable assurance
that a program's objectives are being achieved.[Footnote 29] However,
we found that, as of August 2008, SSA had not fully implemented the
recommendations we made in 2004 to help ensure the effectiveness of the
demonstration projects.[Footnote 30] Specifically, SSA continues to
lack:
* a formal, comprehensive, long-term agenda for conducting
demonstration projects;
* an expert panel to review and provide regular input on the design and
implementation of demonstration projects from the early stages of a
project through its final evaluation; and:
* a formal process for fully considering the potential policy
implications of its demonstration projects' results and fully apprising
Congress of the results and their policy implications.
We did find that SSA has developed a limited research agenda for its
projects that lacks basic details about the projects, including their
objectives, schedules, and costs. The agenda was also developed without
broadly consulting key internal and external stakeholders to obtain
their input. SSA officials told us that they do not plan to update the
agenda to reflect that some of the projects have been cancelled. In
contrast, other federal agencies that conduct research have published
much more detailed research agendas and update them regularly. For
example, the Department of Education's National Institute on Disability
and Rehabilitation Research publishes a 5-year plan that outlines
priorities for rehabilitation research, demonstration projects,
training and related activities, explains the basis for such
activities, and publishes the plan for public comment before submitting
it to Congress.
In addition, agency officials said that SSA planned to continue using
experts only on an ad hoc basis, citing how potential conflicts of
interest could pose challenges to serve on an expert panel as we had
recommended.[Footnote 31] SSA established ad hoc panels or consulted
with outside experts for 8 of its 14 demonstration projects, including
4 of those that are currently in progress. Many of SSA's project
officers and contractors reported that this was a positive experience,
and SSA management has told us that they plan to continue using experts
in the future. However, under this approach SSA may miss the
opportunity to obtain advice more broadly on the demonstration program.
For example, the panels and experts that SSA used were brought on board
after the agency decided to initiate the demonstration projects;
therefore, they were not in position at an early enough point in time
to help SSA consider whether a demonstration project or an alternative
research approach was, in fact, the best way to meet the agency's
needs. Furthermore, SSA did not regularly seek input from the Social
Security Advisory Board or the National Council on Disability, which
both play key roles in federal disability policy and could be in a
position to advise SSA more broadly on the demonstration
projects.[Footnote 32]
In addition, our prior work found that SSA had not sufficiently
provided information on the status and results of its demonstration
projects to Congress. In our current review, we found that SSA
regularly submits annual reports about the DI demonstration projects to
its congressional oversight committees. While SSA meets its statutory
requirements by submitting these reports, the information in them is
generally limited to descriptions of the projects' objectives and the
dates of upcoming milestones. Similarly, the information that SSA
reports about its SSI demonstration projects is limited to brief
descriptions in the agency's annual congressional budget
justifications. Key information that could help Congress monitor the
progress of the demonstration projects--including project costs,
potential risks and obstacles to their success, or the policy
implications of their results--was rarely included in the annual
reports or budget justifications. However, SSA officials also told us
that they sometimes share additional information with Congress about
the demonstration projects. For example, SSA officials told us they met
with congressional committees in October and November 2007 to share
information about its design plans for the BOND project.
In addition to not fully addressing our prior recommendations, SSA does
not have written policies and procedures governing how it should review
and operate its demonstration program. Accordingly, SSA does not have a
written policy requiring SSA management to review its project officer's
demonstration projects on a regular basis. Standards for internal
control in the federal government state that managers should compare a
program's actual performance against expected targets and analyze
significant differences.[Footnote 33] Although the new program
management team reviewed each of the demonstration projects at the time
of their appointment, SSA does not have a written policy requiring such
a review process periodically throughout the design, implementation,
and evaluation phases of each project. SSA's lack of a policy to
systematically review each project on a periodic basis contributed to
problems sometimes going undetected after the projects were
implemented, and they did not yield the data needed for their
evaluations. For example, because SSA was not actively involved in
implementing or monitoring the Florida Freedom Initiative demonstration
project, it was not in a position to take steps to ensure that the
project proceeded as planned, and staff at the state level failed to
enroll enough participants to generate data for the evaluation that SSA
planned. Therefore, no evaluation was conducted for this project.
In addition, we found that SSA does not have written procedures for its
project officers to follow as they design, implement, and evaluate its
demonstration projects. Such procedures could be used to ensure that
standard research practices, such as conducting pilot phases and
including internal and external stakeholders, are applied when planning
and implementing the demonstration projects. Specifically, SSA does not
require staff to regularly use pilots to test projects' underlying
assumptions, operational logistics, or feasibility before they are
implemented. As a result, SSA planned or conducted pilots or phased
implementations for only 8 of its 14 projects, although GAO criteria
for evaluation research emphasize the importance of conducting pilots,
as they are a critical test of a project's design.[Footnote 34] At
least four of the projects that did not include pilot phases
experienced the type of logistical challenges that pilots are intended
to identify. For example, the Homeless Outreach Projects and Evaluation
demonstration project experienced start-up delays because of
compatibility problems between the contractor's online data collection
system and the computer systems at the 41 sites where the project was
implemented. If SSA had conducted a pilot phase for this project, it
may have detected these issues at a smaller number of sites and
developed a plan to resolve them prior to implementing the full
project.
In addition, SSA does not have written procedures directing its project
officers or contractors to routinely consult with internal and external
stakeholders when planning the demonstration projects. We found that at
least 11 of the 14 projects experienced challenges or limitations
because SSA had not obtained sufficient input from, or coordinated
effectively with, internal and external stakeholders. For example, SSA
officials told us that they were aware that the Benefit Offset - 4
State Pilot project and the Benefit Offset National Demonstration
project required a change to internal SSA processes for calculating DI
benefits but did not coordinate with key internal stakeholders early on
to determine how to make this change, and no systematic process was put
in place. As a result, SSA had to calculate payments by hand for the
Benefit Offset - 4 State Pilot, and the BOND project's implementation
has been delayed while SSA now works with its internal stakeholders to
determine how to make the needed changes. Furthermore, SSA officials,
contractors, and representatives from the various demonstration
projects' implementation sites told us that there was little input from
internal stakeholders and that internal coordination problems existed
on at least seven of these projects, including three of those that were
cancelled. We also found that lack of coordination or communication
with external stakeholders led to challenges in at least seven of these
projects, including four of those that were cancelled. For example,
coordination problems between the two contractors for the California
Rise project resulted in the project's components being designed in
isolation from each other, which complicated the evaluation plans and
eventually contributed to the project's cancellation. In addition, SSA
did not always include the prospective implementation sites in the
planning and design of its projects, although they could have provided
insight into the feasibility and logistical requirements of the
project.
While SSA has periodically provided direction informally to its project
officers, some project officers told us that more formal guidance would
have helped them to better understand what steps were necessary and
expected, and we concluded from our discussions with others that such
guidance would have been helpful. SSA recognizes that the program's
lack of written procedures is a limitation and is drafting a guidebook
on standard research practices for staff to follow when planning and
designing demonstration projects. Although an SSA official told us that
this document is a work in progress, it appears that the handbook will
include key procedures for designing a project, such as identifying
what data is needed for the evaluation and how it should be obtained.
It also directs project officers to assemble a team for project
development that includes staff from across the agency who will be able
to collaborate and provide the input necessary to address the multiple
components of the demonstration project. The draft guidebook also
includes provisions for assembling a research panel composed of
internal and external experts. This panel would review proposed
research projects and identify those that present the most promising
opportunities, taking into consideration the extent to which prior
research has already addressed the topic. However, in its current form,
the guidebook provides little direction for the implementation or
evaluation phases of the demonstration projects, and SSA officials had
not finalized it as of May 2008.
Without comprehensive written policies and procedures governing how SSA
manages and operates its demonstration programs, the project
objectives, designs, and evaluation plans may be impacted during times
of organizational change. Because government operating conditions
continually change, agencies should have mechanisms in place to
identify and address any special risks arising from such changes,
especially those caused by hiring new personnel to occupy key positions
in the agency.[Footnote 35] However, because SSA lacks mechanisms such
as a standing advisory panel or written policies and procedures to
provide continuity for its demonstration program when organizational
changes occur, it cannot guarantee that institutional knowledge about
the projects is shared or that the impacts of such changes are
considered as the projects progress. SSA has experienced several
organizational changes since the first of these projects was initiated
in 1998, which have included the demonstration program's relocation
from the Office of Policy to the Office of Disability and Income
Support Programs in 2002, program management's replacement in 2007, and
the Office of Policy and Office of Disability and Income Support
Program's merger in 2008. At least six of SSA's projects experienced
schedule delays and cancellations, in part because newly appointed
officials made significant changes to some projects or determined that
because others faced significant limitations or potential challenges it
was not in the agency's interests to continue them (see fig. 1). While
certain management actions may be reasonable, SSA's lack of written
policies and procedures governing how such steps are taken leaves
current and future projects vulnerable to disruption. For example, we
found that the Benefit Offset National Demonstration project is still
in the design phase after 9 years, during which time it has gone
through numerous revisions by different program managers and was moved
from one office to another. As of August 2008, an interagency working
group was determining how to implement this administratively complex
project. SSA has put the project's implementation and evaluation on
hold until this issue has been resolved.
Figure 1: Projects Experienced Schedule Delays and Cancellations
Associated with Organizational Changes:
This figure is a chart showing projects which experienced schedule
delays and cancellations associated with organizational changes.
[See PDF for image]
Source: GAO analysis of SSA data.
[A] As of August 2008, SSA had not determined the expected
implementation or evaluation date for the BOND project.
[End of figure]
Conclusions:
For over two decades, SSA has had the authority to conduct
demonstration projects to test strategies that could address the
challenges posed by the low rate of return into the workforce and the
growing number of applicants and disabled beneficiaries. However, the
agency has missed opportunities to identify ways to modernize DI and
SSI programs and policies because it has generally not conducted the
demonstration projects effectively. Since 1998 alone, SSA has spent
over $150 million on 14 demonstration projects; yet these projects have
generated limited information about the impacts of the strategies that
were being tested. Although many of these projects were generally well
designed, SSA's lack of written polices, procedures, and mechanisms for
managing and operating these projects is one of the key reasons the
projects SSA has completed and cancelled to date were generally not
implemented and evaluated in a way that yielded reliable, data-driven
impact information. As a result, Congress, SSA, and other organizations
that play a critical role in federal disability policy continue to lack
key information about important issues, such as the impact of providing
health care or employment supports to DI and SSI beneficiaries as a
means to help beneficiaries achieve self sufficiency and leave the
rolls.
SSA's five demonstration projects currently in progress have the
potential to identify solutions to some of SSA's challenges. However,
if SSA does not address the limitations in the way it manages and
operates demonstration projects, these projects may encounter the same
challenges that past projects have faced, and SSA could again have
little to show for its efforts. Given that SSA estimates it will spend
approximately $220 million over the next several years to complete
these projects, it is important that steps be taken to make the
projects less vulnerable to the challenges and organizational changes
they could encounter in the future. SSA's actions to review its
demonstration projects and to begin drafting guidance to help staff
better plan and design its projects are encouraging first steps. As SSA
officials work toward finalizing this guidance, it is also necessary
for the agency to address its lack of written policies and procedures
for managing and operating its projects during their implementation and
evaluation phases. SSA should also take action to fully implement the
recommendations we made in 2004. Implementing those recommendations by
fully developing its research agenda, establishing an expert panel to
advise it about the projects on a regular basis, and improving its
communications with Congress could help improve the effectiveness and
transparency of its demonstration program going forward.
Recommendation for Executive Action:
To improve SSA's management of its demonstration projects we recommend
that the Commissioner of Social Security direct the Deputy Commissioner
for the Office of Retirement and Disability Policy to:
* establish written policies, procedures, and mechanisms for managing
and operating its demonstration projects that are consistent with
standard research practices and internal control standards in the
federal government, including those for coordinating with internal and
external stakeholders and sharing information with Congress.
Agency Comments and Our Evaluation:
We obtained written comments on a draft of this report from SSA, which
are reproduced in appendix IV. We incorporated technical comments we
received throughout the report, as appropriate.
In response to our draft report, SSA generally agreed with our
recommendation and acknowledged the need to develop a guidebook to
assist its staff in the design, implementation, and evaluation phases
of its demonstration projects. SSA further discussed its existing
processes and written procedures for managing and reviewing its
programs, including the demonstration project program. While
noteworthy, we continue to believe SSA needs to establish written
procedures that incorporate professional research standards and
internal control mechanisms for ensuring that the demonstration
projects yield reliable information about their impacts. SSA considers
its current guidebook a work in progress. Further, SSA stated that the
agency has taken steps in recent years to address our prior
recommendations. While we acknowledge SSA's efforts, the agency needs
to take additional steps to fully implement them. For example, SSA
continues to lack a standing expert panel to review and provide regular
input on the demonstration projects overall, even though it has
employed subject matter experts for some of its demonstration projects.
Although SSA officials have raised concerns about the difficulty of
establishing an expert panel because research contractors serving on
the panel would be precluded from working on individual projects, we
continue to believe that such a panel could be established. As
previously recommended, this panel should also include SSA's key
research personnel and outside disability experts in addition to
researchers.
We are sending copies of this report to the Commissioner of SSA and
other interested parties. We will also make copies available to others
upon request. In addition, the report will be available at no charge on
the GAO Web site at [hyperlink, http://www.gao.gov].
If you or your staffs have any questions concerning this report, please
contact me at (202) 512-7215. Contact points for our offices of
Congressional Relations and Public Affairs may be found on the last
page of this report. GAO staff who made key contributions to this
report are listed in appendix V.
Signed by:
Daniel Bertoni:
Director, Education, Workforce, and Income Security:
[End of section]
Appendix I: Scope and Methodology:
To determine how SSA has used its current demonstration authority, we
reviewed legislation authorizing the Social Security Administration
(SSA) to conduct Disability Insurance (DI) and Supplemental Security
Income (SSI) demonstration projects, prior GAO report and SSA's Office
of the Inspector General (OIG) reports, and reports to Congress on the
demonstration authority. We reviewed documents from SSA and from
publicly available sources, including the Federal Register and reports
by other research organizations. We interviewed current and former SSA
officials in the Office of Disability and Income Security
Programs,[Footnote 36] specifically, the Office of Program Development
and Research (OPDR) and the Office of Research, Evaluation and
Statistics who had responsibility for, or involvement in, the
demonstration projects. We also interviewed research contractors that
worked on the demonstration projects and individuals from organizations
that have a key role in federal disability policy. In addition, we
interviewed staff from sites where SSA implemented 9 of the 14
demonstration projects. We selected sites that included ongoing,
cancelled and completed projects, and represented diverse geographic
regions throughout the United States.
To better understand SSA's DI and SSI demonstration projects, we
reviewed SSA documents describing the purpose, design, and status for
all demonstration projects that were in progress, completed, or had
been cancelled prior to completion. These documents included requests
for proposals, project plans and schedules, interim or final project
reports, and reports to Congress from 1998 to 2008. We used these
documents to identify key characteristics of the projects, including
the policy issues addressed, use of contractors, the authority used to
conduct each project, project timelines, and information resulting from
each project. We also examined the issues SSA tested or was in the
process of testing in its demonstration program. We reviewed the
authorizing statutes for the DI and SSI demonstration programs, as well
as requirements for specific demonstrations included in the Ticket to
Work and Work Incentives Improvement Act of 1999 to determine the
extent to which the projects in SSA's demonstration program address
statutory requirements. For projects cancelled during this time period,
we collected cancellation memos and other documentation to determine
SSA's reasons for the cancellations.
To describe the costs associated with the program, we collected
expenditures data from SSA for each project--including funds spent to
date for each project and total anticipated funding for the projects
that are currently in progress. We assessed the reliability of the
budget data by (1) manually checking the required data elements, (2)
reviewing existing information about the data and the system that
produced them, and (3) interviewing agency officials knowledgeable
about the data. When we found discrepancies, we brought them to the
attention of SSA officials and worked with them to correct the
discrepancies before conducting our analyses. Based on these efforts,
we determined that the data were sufficiently reliable for the purposes
of this report.
To assess the extent to which SSA's demonstration projects were
designed in accordance with professional research standards and
statutory criteria, we reviewed the most current information that SSA
provided about each project, either an evaluation design or final
evaluation report. These design and evaluation methodologies were
assessed against professional research standards, consistent with the
authorizing statutes' methodological requirements and GAO's and
recognized academic criteria for conducting evaluation
research.[Footnote 37] Key components of the professional research
standards include:
* methodological rigor of the project's design and evaluation and their
appropriateness given the purpose of the research (e.g. use of an
experimental or quasi-experimental design for an impact evaluation);
* appropriate handling of any problems encountered when implementing
the evaluation's design, such as participant attrition or insufficient
sample sizes;
* appropriate handling of any problems encountered with the data, such
as missing values or variables;
* appropriate variables to ensure internal and external validity, given
the evaluation's design;
* appropriate data analysis and statistical models, such as frequencies
or multivariate analysis, given the evaluation's design; and:
* overall strength of the evaluation design and analysis.
To assess the appropriateness of each study's methodology for answering
the research questions, we developed two data collection instruments
based on these professional research standards--one for evaluation
designs and one for the final evaluation reports. We then examined the
strengths and weaknesses of the evaluation designs and final reports,
taking into consideration the project's objectives, resource
constraints, methodological approach, technical adequacy of plans for
data collection and analysis, and when available, the presentation of
the findings. A social scientist read and coded each evaluation design
or final report. A second social scientist reviewed each completed data
collection instrument and the relevant documentation to verify the
accuracy of every coded item. For each DI demonstration project, we
also reviewed the reports to ascertain whether they met statutory
requirements that the project's results be broadly applicable to
relevant segments of the DI beneficiary population, not just the
project participants. For each SSI demonstration project, we also
interviewed agency officials to determine whether SSA met its statutory
obligation to obtain the advice and recommendations of specialists who
are competent to evaluate the projects as to the soundness of their
design, the possibilities of securing productive results, the adequacy
of resources to conduct them, and their relationship to other similar
research or demonstrations already in progress. We also identified key
provisions of the demonstration authority statutes to assess SSA's
compliance with congressional reporting requirements.
To address SSA's planning and management of its demonstration projects,
we interviewed SSA management and staff about the agency's policies,
guidance and processes on developing and implementing demonstration
projects, and collected supporting documentation where available. We
assessed the adequacy of SSA's internal controls using the criteria in
GAO's Standards for Internal Control in the Federal Government, GAO/
AIMD 00-21.3.1, dated November 1999. These standards, issued pursuant
to the requirements of the Federal Managers' Financial Integrity Act of
1982 (FMFIA), provide the overall framework for establishing and
maintaining internal control in the federal government. Also pursuant
to FMFIA, the Office of Management and Budget issued Circular A-123,
revised December 21, 2004, to provide the specific requirements for
assessing the reporting on internal controls. Internal control
standards and the definition of internal control in Circular A-123 are
based on the GAO Standards for Internal Control in the Federal
Government.
[End of section]
Appendix II: Additional Statutory Criteria for the Benefit Offset
National Demonstration Project:
In addition to meeting professional research standards and the mandated
methodological requirements for the DI demonstration projects, the
Benefit Offset National Demonstration (BOND) project is required to
address a number of specific issues.[Footnote 38] We found that BOND
met all of these requirements except the one to test the project's
effect on induced entry to and reduced exit from the DI rolls (table
5).
Table 5: Additional Statutory Criteria for the Benefit Offset National
Demonstration Project:
Statutory requirement: The effects, if any, of induced entry into the
project and reduced exit from the project;
SSA action: SSA considered including an experimental analysis of
whether a benefit offset would induce entry to the DI program. In
consultation with their research contractor, agency officials
determined that the best possible approach would be cost prohibitive in
terms of resources and time, and very complex, and still would be
unlikely to produce solid, reliable estimates of what induced demand
would occur if a DI benefit offset were implemented nationally. SSA
will explore alternative approaches to estimating potential entry
effects.
Statutory requirement: The extent, if any, to which the project being
tested is affected by whether it is in operation in a locality within
an area under the administration of the Ticket to Work and Self-
Sufficiency Program established under section 1148 of the Social
Security Act;
SSA action: An SSA official told us the effects of the Ticket to Work
(Ticket) program could not be evaluated as intended in the statute
because the two programs did not run concurrently, although it was
originally expected that they would. The statutory criteria presume
that the Ticket program would be operating in some of the same
locations as the BOND project, but not all, so that some BOND
participants would be eligible to participate in the Ticket program,
but others would not be eligible. The Ticket program has now been
implemented nationwide, and most SSA beneficiaries are now eligible to
participate in it, thus the two groups needed for this evaluation no
longer exist. However, the agency official and SSA's research
contractor told us that SSA will be able to determine which BOND
participants are also participating in the Ticket program, and the
influence of the Ticket program on these participants would be
considered in the BOND evaluation.
Statutory requirement: The savings that accrue to the Federal Old-Age
and Survivors Insurance Trust fund, the Federal Disability Insurance
Trust fund, and other federal programs under the project being tested;
SSA action: The BOND project design includes a benefit-cost analysis
that will estimate the net budgetary effects of the benefit offset and
counseling provided in each treatment package separately for
participants, society, SSA, the federal government, and state and local
governments. Agency officials and the research contractor told us that
the savings that accrue to the Federal Old-Age and Survivors Insurance
Trust fund, the Federal Disability Insurance Trust fund, and other
Federal programs will be part of that analysis.
Statutory requirement: The annual cost (including net cost) of the
project and the annual cost (including net cost) that would have been
incurred in the absence of the project;
SSA action: The BOND project design includes a benefit-cost analysis
that will estimate the net budgetary effects of the benefit offset and
counseling provided in each treatment package separately for
participants, society, SSA, the federal government, and state and local
governments. Agency officials and the research contractor told us that
the annual cost (including net cost) of the project and the annual cost
(including net cost) that would have been incurred in the absence of
the project will be part of that analysis.
Statutory requirement: The determinants of return to work, including
the characteristics of the beneficiaries who participate in the
project;
SSA action: The BOND project design includes a participation analysis
that will examine the proportion of beneficiaries that take up the
offer of the benefit offset and/or counseling, the timing and intensity
of their participation, the characteristics of participants and
nonparticipants, and the reasons for participation and
nonparticipation.
Statutory requirement: The employment outcomes, including wages,
occupations, benefits, and hours worked, of beneficiaries who return to
work as a result of participation in the project;
SSA action: The BOND project design includes an impact analysis of the
benefit offset and benefits counseling on levels of employment,
earnings and benefits, wages, and occupations. Other employment
analyses such as effects on regularity of work will also be conducted.
Impacts on other dimensions will also be investigated, such as health
status, function in activities of daily living, taxes paid, and family
income.
Source: GAO analysis of statutory requirements and SSA data.
[End of table]
In addition, while not included as part of the BOND demonstration, SSA
plans to conduct research on variations in the amount of the offset as
a proportion of earned income to determine the appropriate offset
disregard, in accordance with the DI authorizing statute's
requirements.[Footnote 39]
[End of section]
Appendix III: SSA's Use of Research Contractors and Universities on the
Demonstration Projects:
Project: Projects conducted under DI demonstration authority:
Accelerated Benefits;
Research consultant or university: MDRC (prime contractor); Mathematica
Policy Research (subcontractor);
Role on the demonstration project: SSA awarded MDRC a contract in 2006
for the project's design, implementation, and evaluation. MDRC's design
for the project and evaluation expanded on preliminary design criteria
SSA developed between 2004 and 2005, and published in a Request for
Proposal. MDRC finalized the design in collaboration with Mathematica
Policy Research and SSA. MDRC and Mathematica began enrolling
participants in the project in October 2007. Implementation will
continue through 2011. MDRC is conducting the impact evaluation and
Mathematica is conducting the process evaluation. SSA expects the
evaluation to be completed in 2011.
Project: Projects conducted under DI demonstration authority: Benefit
Offset - 4 State Pilot;
Research consultant or university: None;
SSA awarded contracts directly to the states that are implementing the
pilot;
Role on the demonstration project: NA.
Project: Projects conducted under DI demonstration authority: Benefit
Offset National Demonstration (BOND);
Research consultant or university: Abt Associates;
Role on the demonstration project: SSA awarded Abt Associates a
contract in 2004 for the project's design and will award Abt a second
contract for its implementation and evaluation contingent on successful
completion of the design. Abt's design for the project and evaluation
is based on preliminary design criteria SSA developed between 1999 and
2004 and published in a Request for Proposal. Abt is finalizing the
design and planning for implementation in collaboration with SSA.
Project: Projects conducted under DI demonstration authority:
California RISE (CA HIV/AI) Demonstration;
Research consultant or university: Mathematica Policy Research;
SSA also awarded the California Department of Rehabilitation a
cooperative agreement in 2006 to design the employment supports for the
project, and implement the project;
Role on the demonstration project: SSA awarded Mathematica Policy
Research a contract in 2006 to design and implement the health services
part of the project and to design and conduct the evaluation.
Mathematica's design for the heath services component and the
evaluation is based on preliminary design criteria SSA developed
between 2004 and 2006 and published in a Request for Proposal.
Mathematica worked on the design in 2006 and 2007, and SSA reviewed and
provided input on it. Mathematica expected to implement the project in
2007 and complete its evaluation in 2011, but SSA cancelled the project
in 2007 before implementation began.
Project: Projects conducted under DI demonstration authority: Early
Intervention Demonstration;
Research consultant or university: Disability Research Institute
(Rutgers University); Abt Associates;
Role on the demonstration project: SSA awarded the Disability Research
Institute a contract in 2000 for Rutgers University to design,
implement, and evaluate an Early Intervention pilot project. Rutgers
proposed a design for the project and evaluation and submitted an
evaluation design report for SSA's approval in 2002. Rutgers planned to
implement the project in 2003 and begin its evaluation in 2004, but the
project remained under design until the contract expired in 2005; SSA
modified Abt's contract for the BOND project in 2005 to incorporate
Early Intervention into BOND's design, implementation, and evaluation.
Abt proposed options to incorporate the Early Intervention project, but
SSA later decided to cancel it and eliminated it from Abt's contract in
2007.
Project: Projects conducted under DI demonstration authority: Mental
Health Treatment Study;
Research consultant or university: Westat;
Role on the demonstration project: SSA awarded a contract to Westat in
2005 to implement and evaluate a project SSA designed between 2003 and
2005. Westat began working with the project sites in 2005 to prepare
for implementation and began enrolling participants in the project in
2006. Implementation is scheduled to continue through 2010. Westat
began the evaluation in 2008 and will complete it in 2011.
Project: Projects conducted under SSI demonstration authority:
Disability Program Navigator; (with Department of Labor (DOL));
Research consultant or university: University of Iowa College of Law,
Law Health Policy and Disability Center; Department of Labor also
awarded states cooperative agreements to implement the project;
Role on the demonstration project: DOL awarded a contract to the
University of Iowa College of Law, Law Health Policy and Disability
Center to provide technical assistance for the project's implementation
and to evaluate the project. SSA provided partial funding for the
contract through an Interagency Agreement with DOL in 2002, 2003, and
2004. Implementation began in 2003. The University of Iowa did not
evaluate the impact of the project because it could not meet SSA's data
security requirements in order to obtain data needed for the
evaluation. SSA officials told us that DOL plans to evaluate the
project under a new contract with Mathematica Policy Research, but SSA
is not funding that evaluation.
Project: Projects conducted under SSI demonstration authority: Early
Identification and Intervention Demonstration;
Research consultant or university: George Washington University;
Role on the demonstration project: SSA awarded a contract to George
Washington University in 2004 to conduct preliminary research related
to the Early Identification and Intervention project. SSA designed the
demonstration project between 2004 and 2006 and issued a Request for
Proposal for its implementation in 2007, but did not award a contract
to implement or evaluate the project.
Project: Projects conducted under SSI demonstration authority: Florida
Freedom Initiative;
Research consultant or university: Mathematica Policy Research;
Role on the demonstration project: SSA funded a contract for
Mathematica to design an evaluation for the project through an
Interagency Agreement with the Department of Health and Human Services
in 2004. Due to low enrollment, data needed for the evaluation was not
collected by the State of Florida, which designed and implemented the
project between 2003 and 2007, and the evaluation was not conducted.
Project: Projects conducted under SSI demonstration authority: Homeless
Outreach Projects and Evaluation;
Research consultant or university: Westat;
Role on the demonstration project: SSA awarded a contract to Westat in
2004 to evaluate a project SSA designed between 2003 and 2004. SSA
implemented the project in 2004. Westat completed its evaluation in
2007.
Project: Projects conducted under SSI demonstration authority: Interim
Medical Benefits;
Research consultant or university: None-project cancelled before a
contract was awarded;
Role on the demonstration project: SSA planned to issue a Request for
Proposal to hire a professional research organization to design,
implement, and evaluate the project but cancelled it before it awarded
the contract.
Project: Projects conducted under SSI demonstration authority:
Pediatric Medical Unit;
Research consultant or university: Association of University Centers on
Disabilities (AUCD);
Role on the demonstration project: SSA awarded a contract to AUCD in
2006 to design, implement, and evaluate the project; however, AUCD's
design for the project began in 2004 under an extension of another
contract. AUCD partially implemented the project in 2006 and 2007. SSA
cancelled the project in 2008 before AUCD completed the implementation
or evaluation.
Project: Projects jointly authorized under DI and SSI authorities:
State Partnership Initiative;
Research consultant or university: Virginia Commonwealth University
(VCU) (prime contractor); Mathematica Policy Research (subcontractor);
SSA also awarded 12 states cooperative agreements to design and
implement the project;
Role on the demonstration project: SSA awarded multiple contracts to
VCU starting in 1998 to provide technical assistance to the states that
were implementing the project and to conduct an evaluation. VCU
completed its evaluation in 2006. SSA awarded three contracts executed
by VCU's subcontractor, Mathematica: one in 1998 to provide technical
assistance to the states that were implementing the project, another in
1999 to design the evaluation, and one in 2003 to conduct the
evaluation. Mathematica completed its evaluation in 2005.
Project: Projects jointly authorized under DI and SSI authorities:
Youth Transition Demonstration;
Research consultant or university: Mathematica Policy Research (prime
contractor); MDRC (subcontractor);
Role on the demonstration project: SSA originally designed and
implemented the project in 2003, but due to methodological limitations,
SSA awarded a contract to Mathematica in 2005 to redesign, implement,
and evaluate the project. Mathematica began implementing the new design
in 2006. Implementation will continue until 2013. Mathematica will
complete its evaluation in 2014.
Source: GAO analysis of SSA data.
[End of table]
[End of section]
Appendix IV: Comments from the Social Security Administration:
Social Security:
Social Security Administration:
Baltimore MD:
21235-0001:
The Commissioner:
September 16, 2008:
Mr. Daniel Bertoni:
Director, Education, Workforce, and Income Security Issues:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, D.C. 20548:
Dear Mr. Bertoni:
Thank you for the opportunity to review and comment on the draft
report, "Social Security Disability: Management Controls Needed to
Strengthen Demonstration Projects" (GAO-08-1053).
Enclosed are our detailed comments to the draft report recommendations
along with suggested technical revisions.
If you have any questions, please contact Ms. Candace Skurnik,
Director, Audit Management and Liaison Staff, at (410) 965-4636.
Sincerely,
Signed by:
Michael J. Astrue:
Enclosure:
Comments On The Government Accountability Office (GAO) Draft Report,
"Social Security Disability: Management Controls Needed To Strengthen
Demonstration Projects" (GAO-08-1053):
Thank you for the opportunity to review and comment on this draft
report. We appreciate the report's recognition of our efforts to use
the demonstration authorities we were provided and to include more
rigorous methodologies in doing so. We are in general agreement with
the tenor of the report and are pleased with the report's
acknowledgement of management's efforts over the past year to improve
the management of demonstration projects. We also appreciate that the
report recognizes that while experimental designs are the "gold
standard" for impact evaluations, quasi-experimental methods can be
appropriate given all the considerations of our demonstration projects.
Although this report contains only one recommendation for executive
action, it does reference and comment on our progress in implementing
three recommendations from a prior audit report (GAO-05-19). The
comments that follow address all four recommendations. Following these
comments, we have included other comments and suggested revisions on
the content of the report.
Current Recommendation:
To improve the Social Security Administration's (SSA) management of its
demonstration projects we recommend that the Commissioner of Social
Security direct the Deputy Commissioner for the Office of Retirement
and Disability Policy to:
* Establish written policies, procedures, and mechanisms for managing,
and operating its demonstration projects that are consistent with
standard research practices and internal control standards in the
federal government, including those for coordinating with internal and
external stakeholders and sharing information with Congress.
Comment:
Although we generally concur with this recommendation, the report does
not address existing processes and written procedures for managing and
reviewing demonstration projects. For example, for staff and managers
responsible for demonstrations, the performance management system
provides a means for routinely reviewing demonstrations from project
planning through implementation and evaluation. Our management and the
Office of Management and Budget (OMB) routinely review demonstrations
for the budget process, including the development and review of all
documents required for the conduct of demonstration projects and all
material associated with the budget cycle. We have extensive
information on our intranet website to help Contracting Officer's
Technical Representatives who are working on demonstration projects
understand the OMB clearance process for Federal Register notices and
survey instruments for data collection.
Other efforts that facilitate the effective management of demonstration
projects include receiving a monthly update on each demonstration
project and including demonstration projects in the annual
work plan for the Office of Program Development and Research. For some
demonstrations, management requires more frequent updates and becomes
more directly involved at critical points. For example, for the Benefit
Offset National Demonstration, our management meets with the project
team at least on a weekly basis.
We agree with the need to develop a guidebook to assist our staff in
the design, implementation, and evaluation phases of demonstration
projects and the value of piloting demonstration projects before
proceeding with full implementation (page 28). As the report indicates,
we have developed a guidebook. However, our guidebook does not provide
as much information as this report recommends and we consider it a work
in progress. We also believe our Temporary Assistance to Needy Families-
Supplemental Security Income (TANF-SSI) Disability Transition Project,
which is a collaborative effort between us and the Department of Health
and Human Services' Administration for Children and Families, may serve
as a model for developing possible interventions for demonstrations and
add to the information and guidance contained in the guidebook.
Prior Recommendations for GAO Audit Report GAO-05-19:
Recommendation 1:
Develop a formal, comprehensive, long-term agenda for conducting
demonstration projects.
Comment:
Our current demonstration project documentation, as presented in our
work plan and budget, does provide a significant amount of information
related to our demonstration project agenda. However, our ability to
develop new demonstration projects and a corresponding agenda is
limited by our statutory authority and available resources.
As this report indicates, Section 234 of the Social Security Act, as
amended, expired in December 2005. Since that time, we have not had the
statutory authority to initiate new demonstration projects under title
II of the Act. This impediment has a significant impact on the scope of
demonstration projects we can undertake.
In addition, as recognized in this report, demonstration projects
require extensive resources and are "inherently complex and difficult
to conduct" (page 7).
Demonstration projects must be conducted on a scale that allows us to
make reasonable assumptions regarding the applicability of the outcomes
to the general beneficiary population. For example, the total estimated
cost of our Accelerated Benefits Demonstration project is $43 million.
However, we will use nearly $32 million to provide health benefits to
the 1,200 participants. In addition, we are providing benefits
counseling and employment supports to a subset of those applicants.
Therefore, the primary costs associated with these projects are the
interventions provided and the need to provide those interventions to a
large sample of beneficiaries. We are, therefore, limited in the number
of demonstration projects we can implement at any given time.
Recommendation 2:
Establish an expert panel to review and provide regular input on the
design and implementation of demonstration projects from the early
stages of a project through its final evaluation.
Comment:
We responded to the earlier audit report by employing a Technical
Advisory Group (TAG) for demonstration projects. Each TAG consists of
nationally renowned subject matter experts for the particular
demonstration project. By using experts in a specific field for each
project or development of a particular intervention, we obtain the
specific expertise necessary, as opposed to a standing panel of experts
who may not have the specific expertise. In addition, we have worked
with experts from the research community to discuss specific
demonstration project issues. Finally, changes in our management and
component organization, as mentioned in the report, have now made it
possible for us to utilize the expertise available within the agency.
As noted in the report, current procedures in the guidebook call for
the use of expert panels that include internal and external experts for
the development of demonstration projects. The guidebook also
references the need for input and review of the draft statement of work
from external experts. We intend to follow these procedures in
approaching new demonstration projects. In discussing this
recommendation, the report notes that we did not regularly seek input
from key stakeholders for Federal disability policy, such as the Social
Security Advisory Board. The only new work that we have underway that
may result in a demonstration project is the TANF-SSI Disability
Transition Project. This work responds to a prior recommendation from
not only the GAO, but also the Social Security Advisory Board. We also
recently discussed plans for the Benefit Offset National Demonstration
with the Social Security Advisory Board. Before any new demonstration
project is initiated, we will follow the existing written instructions
in the guidebook that call for collaboration with external
stakeholders.
Recommendation 3:
Establish formal processes to ensure that, at the conclusion of each
demonstration project, SSA fully considers and assesses the policy
implications of its demonstration results, and clearly communicates
SSA's assessment to Congress.
Comment:
As the report indicates, our current demonstration projects include
explicit provisions for fully analyzing project results, and we will
clearly communicate the results and the policy implications to
Congress. We have also reinforced this need by placing the statutory
reporting requirements in our guidebook.
The current report, in discussing this prior recommendation, criticizes
the content of our annual reports to Congress. We agree with the
overall suggestion that the reports could be more informative. We have
already recognized this need, and our recent reports have attempted to
provide Congress with more information regarding project status and
potential issues.
[End of section]
Appendix V: GAO Contact and Staff Acknowledgments:
GAO Contact:
Daniel Bertoni, (202) 512-7215 or bertonid@gao.gov:
Staff Acknowledgments:
In addition to the individual named above, key contributions to this
report were made by Michael Collins, Assistant Director; Jason Holsclaw
and Anne Welch, Analysts-in-Charge; Dana Hopings; Annamarie Lopata;
Jean McSween. Additional support was provided by Kenneth Bombara;
Daniel Concepcion; Jennifer Cook; Cindy Gilbert; Sharon Hermes; Joanie
Lofgren; Joel Marus; Mimi Nguyen; Patricia Owens; Daniel Schwimer; Kris
Trueblood; Kathy White; Charles Willson; Elizabeth Wood; and Jill Yost.
[End of section]
Related GAO Products:
Federal Disability Programs: More Strategic Coordination Could Help
Overcome Challenges to Needed Transformation. GAO-08-635. Washington,
D.C.: May 20, 2008.
Social Security Disability: Better Planning, Management, and Evaluation
Could Help Address Backlogs. GAO-08-40. Washington, D.C.: December 7,
2007.
High Risk Series: An Update. GAO-07-310. Washington, D.C.: January
2007.
Federal Disability Assistance: Wide Array of Programs Needs to be
Examined in Light of 21st Century Challenges. GAO-05-626. Washington,
D.C.: June 2, 2005.
Social Security Disability: Improved Processes for Planning and
Conducting Demonstrations May Help SSA More Effectively Use Its
Demonstration Authority. GAO-05-19. Washington, D.C.: November 4, 2004.
SSA's Rehabilitation Programs. GAO/HEHS-95-253R. Washington, D.C.:
September 7, 1995.
Impact of Vocational Rehabilitation Services on the Social Security
Disability Insurance (DI) Program. GAO/HRD-T-88-16. Washington, D.C.:
May 26, 1988.
Social Security: Little Success Achieved in Rehabilitating Disabled
Beneficiaries. GAO/HRD-88-11. Washington, D.C.: December 7, 1987.
Social Security: Observations on Demonstration Interviews with
Disability Claimants. GAO/HRD-88-22BR. Washington, D.C.: December 3,
1987.
Social Security: Demonstration Projects Concerning Interviews with
Disability Claimants. GAO/HRD-87-35. Washington, D.C.: February 19,
1987.
[End of section]
Footnotes:
[1] GAO, Social Security Disability: Improved Processes for Planning
and Conducting Demonstrations May Help SSA More Effectively Use Its
Demonstration Authority. GAO-05-19 (Washington, D.C.: Nov. 4, 2004).
[2] GAO, High Risk Series: An Update, GAO-07-310 (Washington, D.C.:
January 2007).
[3] GAO, Designing Evaluations, GAO/PEMD-10.1.4 (Washington, D.C.:
March 1991); GAO, Assessing the Reliability of Computer-Processed Data,
GAO/OP-8.1.3 (Washington, D.C.: April 1991); P.H. Rossi, M.W. Lipsey,
and H.E. Freeman, Evaluation: A Systematic Approach (California, 2004);
B.R. Worthen, J.R. Sanders, and J.L. Fitzpatrick, Program Evaluation:
Alternative Approaches and Practical Guidelines (New York, 1997).
[4] GAO, Standards for Internal Control in the Federal Government, GAO/
AIMD-00-21.3.1 (Washington, D.C.: November 1999). These standards,
issued pursuant to the requirements of the Federal Managers' Financial
Integrity Act of 1982 (FMFIA), provide the overall framework for
establishing and maintaining internal control. Also pursuant to FMFIA,
the Office of Management and Budget issued Circular A-123, revised
December 21, 2004, to provide the specific requirements for assessing
the reporting on internal controls. Internal control standards and the
definition of internal control in Circular A-123 are based on the GAO
Standards for Internal Control in the Federal Government.
[5] To be eligible for DI benefits, a worker must be (1) insured and
(2) disabled according to the definition of disability. To be insured,
they must have worked a minimum amount of time in employment covered by
Social Security (similar to eligibility for Old-Age and Survivors
Insurance benefits). However, for disability benefits, if an individual
does not have 40 quarters of coverage (generally about 10 years), they
must have 1 quarter of coverage (1 quarter of coverage is equal to
$1,050 in 2008 and indexed to the annual increase in wages) for each
year after 1950 or from age 21 up to the onset of disability.
[6] DI demonstration authority was provided under § 505(a) of the
Social Security Disability Amendments of 1980 (Pub. L. No. 96-265).
Another provision of the act, § 505(b), amended §1110 of the Social
Security Act of 1935 to provide SSA with similar demonstration
authority for the SSI program, including authority to waive SSI program
rules.
[7] Pub. L. No. 108-203.
[8] GAO-05-19.
[9] GAO-05-19.
[10] SSI demonstration authority focuses, in part, on prevention and
reduction of dependency, aid in effecting coordination of planning
between private and public welfare agencies, improving the
administration and effectiveness of programs carried on or assisted
under the Social Security Act, and ascertaining the feasibility of
treating alcoholics and drug addicts to prevent the onset of
irreversible medical conditions which may result in permanent
disability. 42 U.S.C §1310.
[11] 42 U.S.C. § 434.
[12] Pub. L. No. 106-170 § 302.
[13] Individuals with serious mental illnesses now represent over a
quarter (27 percent) of all DI recipients, and they account for the
single largest diagnostic group (34 percent) on the SSI rolls. As part
of our ongoing work on modernization of federal disability programs, we
have raised the issue of planning for growth in the demand for services
and benefits. See GAO, Federal Disability Assistance: Wide Array of
Programs Needs to be Examined in Light of 21st Century Challenges. GAO-
05-626 (Washington, D.C.: June 2, 2005).
[14] 42 U.S.C. 434 § 234(b).
[15] Several of the demonstration projects experienced challenges
meeting new data security requirements that SSA established in 2006
following the theft of a Department of Veterans Affairs computer, which
heightened SSA's concerns about data security. SSA officials told us
that DOL plans to evaluate the project under a new contract with
Mathematica Policy Research, but SSA is not funding that evaluation and
has a very limited role in planning and designing it. It is unclear
what information will be available to use for making disability program
and policy decisions.
[16] This program has been renamed the Work Incentives Planning and
Assistance (WIPA) program.
[17] GAO-05-19.
[18] The State Partnership Initiative was nearing completion at this
time, and it was too late to make such improvements to it.
[19] We did not assess the evaluation designs for the Disability
Program Navigator and the Early Identification and Intervention
Demonstration projects because SSA could not provide us with needed
information. The Disability Program Navigator project concluded without
an evaluation, and the Early Identification and Intervention
Demonstration was cancelled before its design was finalized.
[20] Social Security Act § 234(b).
[21] Research experts have noted that there is an inherent tradeoff
between using experimental designs and generalizing results.
[22] Each state designed its own project for the State Partnership
Initiative.
[23] Social Security Act § 1110(a)(2)
[24] Pub. L. No. 106-170 § 302.
[25] GAO-05-19.
[26] According to our analysis, the designs of the projects currently
under way are considered strong or reasonable when assessed against
professional research standards.
[27] In response to our draft report SSA provided technical comments
claiming an additional $600 million in savings resulting from a project
cancellation. After analyzing the supporting documentation, we
determined that the cost estimates that SSA provided did not directly
support the proposed cost savings; therefore, we did not incorporate
the suggested change into the report.
[28] In 2007, SSA had moved DISP's operational component to another
office in response to findings and recommendations made by its
Inspector General. Agency officials told us that following the
reorganization, DISP and the Office of Policy had similar missions,
especially in regard to research.
[29] GAO/AIMD-00-21.3.1.
[30] GAO-05-19.
[31] GAO-05-19.
[32] The Social Security Advisory Board is an independent, bipartisan
board created by Congress and appointed by the President and Congress
to advise the President, Congress, and the Commissioner of Social
Security on matters related to the Social Security and Supplemental
Security Income programs. The National Council on Disability is an
independent federal agency, composed of members appointed by the
President by and with the advice and consent of the U.S. Senate, to
provide advice to the President, Congress, and executive branch
agencies to promote policies, programs, practices, and procedures that
guarantee equal opportunity for all individuals with disabilities.
[33] GAO/AIMD-00-21.3.1.
[34] GAO/PEMD-10.1.4.
[35] GAO-AIMD-00-21.3.1.
[36] This office was merged with SSA's Office of Policy in February
2008 and is now called the Office of Retirement and Disability Policy
(ORDP).
[37] GAO, Designing Evaluations, GAO/PEMD-10.1.4 (Washington, D.C.:
March 1991); GAO, Assessing the Reliability of Computer-Processed Data,
GAO/OP-8.1.3 (Washington, D.C.: April 1991); P.H. Rossi, M.W. Lipsey,
and H.E. Freeman, Evaluation: A Systematic Approach (California, 2004.)
B.R. Worthen, J.R. Sanders, and J.L Fitzpatrick, Program Evaluation:
Alternative Approaches and Practical Guidelines (New York, 1997).
[38] Pub. L. No. 106-170 § 302.
[39] Social Security Act of 1935 § 234(1)(C).
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
