Social Security Disability
Improved Processes for Planning and Conducting Demonstrations May Help SSA More Effectively Use Its Demonstration Authority Gao ID: GAO-05-19 November 4, 2004Since 1980, the Congress has required the Social Security Administration (SSA) to conduct demonstration projects to test the effectiveness of possible program changes that could encourage individuals to return to work and decrease their dependence on Disability Insurance (DI) benefits. To conduct these demonstrations, the Congress authorized SSA, on a temporary basis, to waive certain DI and Medicare program rules and to use Social Security Trust Funds. The Congress required GAO to review SSA's use of its DI demonstration authority and to make a recommendation as to whether this authority should be made permanent.
SSA has not used its demonstration authority to extensively evaluate a wide range of DI policy areas dealing with return to work. Despite being given the authority to assess a broad range of policy alternatives, SSA has, until very recently, focused its demonstration efforts mostly on a relatively narrow set of policy issues--those dealing with the provision of vocational rehabilitation and employment services. SSA's recently proposed or initiated demonstrations have begun to address a broader range of policy issues, such as provisions to reduce, rather than terminate, benefits based on earnings above a certain level. However, the agency has no systematic processes or mechanisms for ensuring that it is adequately identifying and prioritizing those issues that could best be addressed through use of its demonstration authority. For example, the agency has not developed a formal demonstration research agenda explicitly identifying its broad vision for using its DI demonstration authority and explaining how ongoing or proposed demonstration projects support achievement of the agency's goals and objectives. SSA's demonstration projects have had little impact on the agency's and the Congress' consideration of DI policy issues. This is due, in part, to methodological limitations that have prevented SSA from producing project results that are useful for reliably assessing DI policy alternatives. In addition, SSA has not established a formal process for ensuring that its demonstration results are fully considered for potential policy implications. For example, SSA does not maintain a comprehensive record of its demonstration results that could be used to build a body of knowledge for informing policy decisions and planning future research. Furthermore, SSA's reporting of demonstration project results has been insufficient in ensuring that the Congress is fully apprised of these results and their policy implications.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-05-19, Social Security Disability: Improved Processes for Planning and Conducting Demonstrations May Help SSA More Effectively Use Its Demonstration Authority
This is the accessible text file for GAO report number GAO-05-19
entitled 'Social Security Disability: Improved Processes for Planning
and Conducting Demonstrations May Help SSA More Effectively Use Its
Demonstration Authority' which was released on November 04, 2004.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Committee on Finance, U.S. Senate, and the Committee on
Ways and Means, House of Representatives:
United States Government Accountability Office:
GAO:
November 2004:
SOCIAL SECURITY DISABILITY:
Improved Processes for Planning and Conducting Demonstrations May Help
SSA More Effectively Use Its Demonstration Authority:
GAO-05-19:
GAO Highlights:
Highlights of GAO-05-19, a report to Committee on Finance, U.S.
Senate, and the Committee on Ways and Means, House of Representatives:
Why GAO Did This Study:
Since 1980, the Congress has required the Social Security
Administration (SSA) to conduct demonstration projects to test the
effectiveness of possible program changes that could encourage
individuals to return to work and decrease their dependence on
Disability Insurance (DI) benefits. To conduct these demonstrations,
the Congress authorized SSA, on a temporary basis, to waive certain DI
and Medicare program rules and to use Social Security Trust Funds. The
Congress required GAO to review SSA‘s use of its DI demonstration
authority and to make a recommendation as to whether this authority
should be made permanent.
What GAO Found:
SSA has not used its demonstration authority to extensively evaluate a
wide range of DI policy areas dealing with return to work. Despite
being given the authority to assess a broad range of policy
alternatives, SSA has, until very recently, focused its demonstration
efforts mostly on a relatively narrow set of policy issues”those
dealing with the provision of vocational rehabilitation and employment
services. SSA‘s recently proposed or initiated demonstrations have
begun to address a broader range of policy issues, such as provisions
to reduce, rather than terminate, benefits based on earnings above a
certain level. However, the agency has no systematic processes or
mechanisms for ensuring that it is adequately identifying and
prioritizing those issues that could best be addressed through use of
its demonstration authority. For example, the agency has not developed
a formal demonstration research agenda explicitly identifying its broad
vision for using its DI demonstration authority and explaining how
ongoing or proposed demonstration projects support achievement of the
agency‘s goals and objectives.
SSA‘s demonstration projects have had little impact on the agency‘s and
the Congress‘ consideration of DI policy issues. This is due, in part,
to methodological limitations that have prevented SSA from producing
project results that are useful for reliably assessing DI policy
alternatives. In addition, SSA has not established a formal process
for ensuring that its demonstration results are fully considered for
potential policy implications. For example, SSA does not maintain a
comprehensive record of its demonstration results that could be used
to build a body of knowledge for informing policy decisions and
planning future research. Furthermore, SSA‘s reporting of demonstration
project results has been insufficient in ensuring that the Congress is
fully apprised of these results and their policy implications.
What GAO Recommends:
GAO recommends that SSA develop a formal agenda for its demonstrations,
establish an expert panel to guide the design and implementation of
demonstrations, and establish formal processes to ensure full
consideration of demonstration results. GAO also identifies several
matters for the Congress to consider, including continuation of DI
demonstration authority on a temporary basis, establishment of
additional reporting requirements for demonstrations, and clearer
specification of the methodological and evaluation requirements for
demonstrations.
In its comments on a draft of this report, SSA agreed with GAO‘s
recommendations.
www.gao.gov/cgi-bin/getrpt?GAO-05-19.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Robert E. Robertson at
(202) 512-7215 or RobertsonR@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
SSA Has Not Used Its Demonstration Authority to Evaluate a Wide Range
of DI Return-to-Work Policy Issues:
SSA's Demonstration Projects Have Had Little Influence on Consideration
of DI Policy Changes:
Conclusions:
Recommendations:
Matters for Congressional Consideration:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Comments from the Social Security Administration:
Appendix III: GAO Contacts and Staff Acknowledgments:
GAO Contacts:
Staff Acknowledgments:
Tables:
Table 1: Completed or Nearly Completed Projects Conducted under SSA's
DI Demonstration Authority:
Table 2: Recently Proposed or Initiated Projects Conducted under SSA's
DI Demonstration Authority:
Abbreviations:
DI: Disability Insurance:
HHS: Department of Health and Human Services:
OASDI: Old-Age and Survivors Insurance and Disability Insurance:
OPDR: Office of Program Development and Research:
RDP: Research Demonstration Program:
RFP: request for proposal:
RSVP: Referral System for Vocational Rehabilitation Providers:
SGA: substantial gainful activity:
SPI: State Partnership Initiative:
SSI: Supplemental Security Income:
SSA: Social Security Administration:
SSAB: Social Security Advisory Board:
United States Government Accountability Office:
Washington, DC 20548:
November 4, 2004:
The Honorable Charles E. Grassley:
Chairman:
The Honorable Max Baucus:
Ranking Minority Member:
Committee on Finance:
U. S. Senate:
The Honorable William M. Thomas:
Chairman:
The Honorable Charles B. Rangel:
Ranking Minority Member:
Committee on Ways and Means:
House of Representatives:
From 1982 to 2002, the number of disabled workers receiving benefits
under the Disability Insurance (DI) program doubled from 2.6 million to
5.5 million, while payments quadrupled from about $14.8 billion to $60
billion. Although technological and medical advances, along with social
changes, have increased the potential for some people with disabilities
to participate in the labor force, less than 1 percent of DI
beneficiaries leave the rolls each year because they have reentered the
workforce. The enactment of various DI work incentives that are
intended to encourage beneficiaries to work--and, potentially, to leave
the rolls--has also had little discernible impact on beneficiaries'
success in returning to the workforce. This low rate of return to work,
coupled with escalating growth in DI program enrollment and benefit
costs over the past few decades, threatens to exhaust the DI Trust Fund
by 2029. Yet relatively little is known about the factors that help
beneficiaries overcome employment challenges and disincentives and that
inhibit them from achieving an earnings level that leads to self-
sufficiency. GAO has designated federal disability programs as a high-
risk area, in part because of the significant growth in the DI program
and the long-standing challenges in devising effective return-to-work
policies.[Footnote 1]
In response to these challenges, the Congress has, since 1980, required
the Social Security Administration (SSA) to conduct demonstration
projects to test the effectiveness of possible program changes that
could encourage individuals to return to work and decrease their
dependence on DI benefits. In fostering return to work, these
demonstrations and the program changes they test are intended to
produce savings in the Trust Funds or improve DI program
administration. To achieve these objectives, SSA's DI demonstration
authority contains several key features that provide SSA with a
potentially valuable tool for assessing the effectiveness of policy
alternatives. One of these features is SSA's authority to waive certain
DI and Medicare program rules. For example, when conducting
demonstrations, SSA is permitted to exempt certain beneficiaries from
requirements that workers with disabilities earn below a certain amount
to remain eligible for DI benefits or that they wait 24 months to
become eligible for Medicare benefits. Another key aspect of SSA's
demonstration authority is the requirement that DI demonstration
projects be of sufficient scope and conducted on a wide enough scale to
ensure a thorough evaluation and results that are applicable to the DI
program as a whole. In addition, the legislation authorized SSA to use
DI Trust Fund and Old-Age and Survivors Insurance Trust Fund monies to
pay for the demonstrations and required SSA to periodically report to
the Congress on its demonstration activities, providing, when
appropriate, recommendations for legislative or administrative
changes. The Congress initially granted SSA this demonstration
authority for a 5-year period but subsequently renewed the authority
several times, most recently extending it through 2005.
The Ticket to Work and Work Incentives Improvement Act (P.L. 106-170)
mandated that GAO review SSA's use of its DI demonstration authority
and provide a recommendation as to whether SSA should be given this
authority permanently. To develop a basis on which to make this
recommendation, we examined the extent to which (1) SSA has used the
demonstration authority to test a range of program changes and (2)
SSA's demonstration authority has been used to inform policy decisions.
In conducting our examination, we reviewed relevant federal statutes
and regulations governing DI demonstration activities and interviewed
SSA officials with current and prior responsibility for demonstration
projects. We also reviewed documents related to SSA's planning,
implementation, and evaluation of demonstration projects, including
agency reports to the Congress, public notices, and reports analyzing
project results. In addition, we examined other reviews of SSA's
demonstrations (including prior GAO and Inspector General reports) and
interviewed officials from disability research, advisory, and advocacy
organizations. Furthermore, we obtained information on research and
demonstrations conducted by other federal agencies. We performed our
work in accordance with generally accepted government auditing
standards between October 2003 and August 2004. For more details about
our scope and methodology, see appendix I.
Results in Brief:
SSA has not used its demonstration authority to extensively evaluate a
wide range of DI policy areas dealing with return to work. Despite
being given the authority to assess a broad range of policy
alternatives, SSA has, until very recently, focused its demonstration
efforts mostly on a relatively narrow set of policy issues--those
dealing with the provision of vocational rehabilitation and employment
services. But even in that policy area, SSA's use of DI demonstration
authority has not been comprehensive and, therefore, did not
extensively address key policy issues that the agency is currently
grappling with in its efforts to implement a new approach for providing
return-to-work assistance. SSA's recently proposed or initiated
demonstrations have begun to address a broader range of policy issues,
such as provisions to reduce, rather than terminate, benefits based on
earnings above a certain level. However, the agency has no systematic
processes or mechanisms for ensuring that it is adequately identifying
and prioritizing those issues that could best be addressed through use
of its demonstration authority. For example, the agency has not
developed a formal demonstration research agenda explicitly identifying
its broad vision for using its DI demonstration authority and
explaining how ongoing or proposed demonstration projects support
achievement of the agency's goals and objectives.
SSA's demonstration projects have had little impact on the agency's and
the Congress' consideration of DI policy issues. This is due, in part,
to methodological limitations that have prevented SSA from producing
project results that are useful for reliably assessing DI policy
alternatives. In addition, SSA has not established a formal process for
ensuring that its demonstration results are fully considered for
potential policy implications. For example, SSA does not maintain a
comprehensive record of its demonstration results that could be used to
build a body of knowledge for informing policy decisions and planning
future research. Furthermore, SSA's reporting of demonstration project
results has been insufficient in ensuring that the Congress is fully
apprised of these results and their policy implications.
This report contains recommendations to help ensure the effectiveness
of SSA's DI demonstration efforts. It also identifies several matters
that the Congress may wish to consider, including continuation of SSA's
demonstration authority on a temporary basis. In its comments on a
draft of this report, SSA agreed with our recommendations. SSA also
agreed that it has not, in the past, used its demonstration authority
to extensively evaluate DI policy, but it noted that its recently
initiated or proposed demonstrations will play a vital role in testing
program and policy changes. SSA's comments appear in appendix II.
Background:
Established in 1956, DI is an insurance program that provides monthly
cash benefits to workers who are unable to work because of severe long-
term disability. Workers who have worked long enough and recently
enough are insured for coverage under the DI program. To meet the
definition of disability under the DI program, an individual must have
a medically determinable physical or mental impairment that (1) has
lasted or is expected to last at least 1 year or to result in death and
(2) prevents the individual from engaging in substantial gainful
activity (SGA). Individuals are considered to be engaged in SGA if they
have countable earnings above a certain dollar level.[Footnote 2] Once
a person is on the disability rolls, benefits continue until (1) the
beneficiary dies, (2) the beneficiary becomes eligible for Social
Security retirement benefits at full retirement age,[Footnote 3] (3)
SSA determines that the beneficiary is no longer eligible for benefits
because his or her earned income exceeds the SGA level, or (4) SSA
decides that the beneficiary's medical condition has improved to the
point that he or she is no longer considered disabled. In 2002, SSA
paid about $60 billion in DI cash benefits to 5.5 million disabled
workers,[Footnote 4] with average monthly benefits amounting to $834
per person.[Footnote 5] In addition to receiving cash assistance,
beneficiaries automatically qualify for Medicare after 24 months of DI
entitlement.
During the 1970s, as the number of disability awards increased
significantly and resulted in substantial cost increases for the DI
program, the Congress became concerned about the growth of the DI
program and program rules that provided disincentives to returning to
work. To encourage DI beneficiaries to return to work--and,
potentially, to leave the benefit rolls--the Congress has, over the
years, enacted legislation providing various work incentives. Such
incentives include a trial work period during which beneficiaries may
earn any amount for 9 months within a 60-month period and still receive
full cash and medical benefits and continued Medicare coverage that
allows beneficiaries to maintain eligibility for Medicare for at least
39 months following a trial work period as long as medical disability
continues.
In an effort to further address these issues, the Congress, in 1980,
required SSA to conduct demonstration projects[Footnote 6] to evaluate
the effectiveness of policy alternatives that could encourage DI
beneficiaries to reenter the workforce.[Footnote 7] A key aspect of
this demonstration authority is SSA's ability to waive DI and Medicare
program rules to the extent needed in conducting these projects. The
legislation granting DI demonstration authority also identified a
variety of policy alternatives for SSA to consider testing, including
(1) alternative ways of treating DI beneficiaries' work-related
activity such as methods allowing for a reduction in benefits based on
earnings and (2) modifications in other rules, such as the trial work
period and Medicare eligibility waiting period, that may serve as
obstacles to DI beneficiaries returning to work. In addition, this
legislation identified several requirements pertaining to the design
and evaluation of DI demonstration projects. In particular, these
projects were required to be of sufficient scope and carried out on a
wide enough scale to permit a thorough evaluation of the policy
alternatives studied such that the results would be generally
applicable to the operation of the DI program. The law additionally
required SSA to submit reports to the Congress announcing the
initiation of DI demonstration projects as well as periodic reports
describing the status of these projects and a final report on all
projects carried out under the demonstration authority. SSA was
directed to make recommendations, when appropriate, for legislative or
administrative changes in its reports to the Congress.
Another important aspect of SSA's DI demonstration authority is that
unlike other SSA research activities, which are funded through
congressional appropriations, these projects can be paid for with DI
Trust Fund and Old-Age and Survivors Insurance Trust Fund monies.
Therefore, SSA is not required to obtain congressional approval for DI
demonstration expenditures, although it is required to receive approval
from the Office of Management and Budget for an annual apportionment of
Trust Funds for these demonstrations.
SSA's DI demonstration authority has always been granted on a temporary
basis and therefore has been subject to periodic review and renewal by
the Congress.[Footnote 8] After initially granting this authority for a
5-year period, the Congress subsequently renewed it several times, in
1986, 1989, 1994, 1999, and 2004. The renewal of SSA's authority has
sometimes been delayed so that SSA has, on several occasions, gone
without DI demonstration authority. For example, after its
demonstration authority expired in June 1996, SSA was not again granted
DI demonstration authority until December 1999. Most recently, the
Congress extended this demonstration authority through December
2005.[Footnote 9]
In addition to granting this general DI demonstration authority, the
Congress may enact legislative mandates for SSA to conduct specific DI
demonstration projects. For example, the Ticket to Work and Work
Incentives Improvement Act of 1999 required SSA to conduct a
demonstration to assess the effectiveness of a benefit offset program
under which DI benefits are reduced by $1 for every $2 in earnings
(above a certain level) by a beneficiary. SSA's authority to conduct
this demonstration is similar in some respects to the authority it has
under its general DI demonstration statute. For instance, the statute
allows waiver of DI and Medicare program provisions to carry out this
benefit offset demonstration. However, some differences exist between
the two authorities. In particular, the benefit offset demonstration
authority provides a more detailed and comprehensive list of
demonstration objectives for SSA to fulfill than does SSA's general
authority. For example, the benefit offset demonstration authority
lists six "matters to be determined," which include assessments of
project costs; savings to the Trust Funds; and project effects on
employment outcomes such as wages, occupations, benefits, and hours
worked.
Regardless of the authority under which they are carried out,
demonstration projects examining the impact of social programs are
inherently complex and difficult to conduct. Measuring outcomes,
ensuring the consistency and quality of data collected at various
sites, establishing a causal connection between outcomes and program
activities, and separating out the influence of extraneous factors
raise formidable technical and logistical problems. Thus, these
projects generally require a planned study and considerable time and
expense. Adding to these complexities are other administrative or
statutory requirements affecting SSA's DI demonstrations. For example,
SSA's policy is that its demonstration projects must not make those who
participate in the project worse off, which could limit the specific
types of policy alternatives the agency can study or the methods used
to study such alternatives.
Although the legislation granting DI demonstration authority does not
prescribe the use of particular methodological approaches, SSA has
repeatedly recognized that the law's general requirements for
demonstration evaluations require SSA to conduct these projects in a
rigorous manner that provides the agency with a reliable basis for
making policy recommendations.[Footnote 10] Rigorous methods are
required to estimate the net impact of a tested disability policy
option because many other factors, such as the economy, can influence
whether a beneficiary returns to work. In an August 2002 report to the
SSA Commissioner, an SSA advisory panel stated that it is widely agreed
that experimental designs,[Footnote 11] "when feasible from operational
and budgetary perspectives and when they can be conducted without
serious threats to their validity, are the best methodology for
determining the effects of changes in government programs."[Footnote
12] In addition, SSA officials and other researchers have noted the
advantages of experimental designs in providing policymakers with more
clear-cut results that are less subject to debate than results derived
from other methods.[Footnote 13] However, when experimental designs are
not feasible or desirable, the use of quasi-experimental
designs[Footnote 14] offers a reasonably rigorous evaluation
alternative that may, under certain circumstances, offer advantages
over experimental designs.[Footnote 15]
Other factors may also limit the rigor of DI demonstrations, including
insufficient sample sizes, inconsistency in demonstration design or
implementation across multiple project sites, and deficiencies in data
collection. Such design, implementation, and evaluation weaknesses may
hamper SSA's use of project results as a basis for making policy
recommendations because they limit the agency's ability to (1) control
for factors external to the demonstration, (2) generalize demonstration
results to a wider population of DI beneficiaries, and (3) isolate the
effects of specific policy interventions from the overall effects
produced by a demonstration.
The Office of Program Development and Research (OPDR) is the entity
within SSA that develops and implements demonstration projects for the
DI and Supplemental Security Income (SSI) Programs.[Footnote 16] OPDR
program and research staff--sometimes with the assistance of outside
research organizations--identifies the broad outlines and requirements
of disability program demonstration projects, including the basic
objectives, scope, and methodological standards for these projects. SSA
then issues formal notices requesting public or private sector
organizations to submit offers to conduct the demonstration projects,
which may include development of a detailed design plan, provision of
technical support, collection of project data, or evaluation of project
results. On the basis of SSA's review of submitted proposals and bids,
the agency may enter into grants, cooperative agreements, or
contractual arrangements with one or more organizations to carry out
demonstration projects.[Footnote 17] For example, a single
demonstration may involve cooperative agreements with states to design
and implement projects as well as contracts with one or more research
institutions to provide technical assistance to the states and evaluate
demonstration results. Project managers in OPDR have the primary
responsibility for overseeing demonstration projects to ensure that
they meet SSA's technical and programmatic requirements. OPDR
collaborates with SSA's Office of Acquisition and Grants in issuing
formal project notices and solicitations and, subsequently, in
overseeing grant or contract performance.
SSA Has Not Used Its Demonstration Authority to Evaluate a Wide Range
of DI Return-to-Work Policy Issues:
SSA has not used its demonstration authority to extensively evaluate a
wide range of DI policy areas dealing with return to work. Until very
recently, SSA has focused its demonstration efforts primarily on a
relatively narrow set of policy issues dealing with the provision of
vocational rehabilitation and employment services, despite being given
the authority to assess a much broader range of policy alternatives.
Even in the area of vocational rehabilitation and employment issues,
SSA's use of DI demonstration authority has not been comprehensive and,
therefore, did not extensively address key policy issues that the
agency is currently grappling with under its Ticket to Work program.
SSA's recently initiated or proposed demonstrations have begun to
address a broader range of policy issues. However, the agency has no
systematic processes or mechanisms for ensuring that it is adequately
identifying and prioritizing those issues that could best be addressed
through use of its demonstration authority.
SSA Has Not Used Its Demonstration Authority Extensively:
The DI demonstration projects that SSA has conducted since 1980 have
not extensively addressed a wide range of return-to-work policy issues.
Since first being granted DI demonstration authority 24 years ago, SSA
has used this authority to complete four projects, with another project
nearing completion.[Footnote 18] Total costs for these projects amount
to at least $107 million, of which about $42 million was paid for from
the Old-Age and Survivors Insurance and Disability Insurance (OASDI)
Trust Funds.[Footnote 19]
The legislation granting DI demonstration authority to SSA provided the
agency with an opportunity to examine a broad set of return-to-work
policy alternatives and even identified some specific alternatives for
SSA to consider studying, including (1) reducing, rather than
terminating, benefits based on earnings; (2) lengthening the trial work
period; (3) decreasing the 24-month waiting period for Medicare
benefits; (4) altering program administration; (5) earlier referral of
beneficiaries for rehabilitation; and (6) using employers and others to
stimulate new forms of vocational rehabilitation. The projects SSA has
conducted thus far have focused predominantly on the latter category of
issues involving vocational rehabilitation and have focused to a lesser
extent--or not at all--on other key policy issues affecting return to
work (see table 1). More specifically, examination of policy
alternatives dealing with the provision of vocational rehabilitation
and employment services has been the primary objective of four of the
five completed or nearly completed demonstrations. Although two of
these projects also examined other DI return-to-work policy issues--
such as the possible effects of changes in program work incentives and
alterations in the provision of medical benefits--they did so to only a
limited extent. None of the projects looked at other potentially
significant DI policy issues, such as the possibility of changing SSA's
benefit structure to allow for a reduction in benefits, rather than a
complete cutoff of benefits, based on earnings.
Table 1: Completed or Nearly Completed Projects Conducted under SSA's
DI Demonstration Authority:
Project: Research Demonstration Program (RDP);
Policy issues studied: Mostly focused on provision of vocational
rehabilitation and employment services but also touched on other policy
issues such as changes in SSA work incentives;
Year initiated: 1987;
Year completed: Most projects under the RDP appear to have been
completed by the early 1990s.;
Cost: $32 million, about half of which was funded through the Trust
Funds, with the other half coming from appropriations.
Project: Pain Assessment Instruments Development Project;
Policy issues studied: Focused on the testing of pain assessment
questionnaires for determining the impact of pain on an applicant's or
beneficiary's ability to return to work;
Year initiated: 1990;
Year completed: 1994;
Cost: A $3.8 million contract was awarded by SSA on this project, but
it is unclear if there were any additional project costs. All of the
costs of this project appear to have been funded through the Trust
Funds..
Project: Project Network;
Policy issues studied: Focused on provision of vocational
rehabilitation and employment services;
Year initiated: 1991;
Year completed: 1999;
Cost: $25.4 million, about $13.7 million of it funded through the Trust
Funds, with the remainder coming from appropriations.
Project: Project Referral System for Vocational Rehabilitation
Providers (Project RSVP);
Policy issues studied: Focused on provision of vocational
rehabilitation;
Year initiated: 1997;
Year completed: No identifiable completion date;
project was eventually superseded by the Ticket to Work program;
Cost: No information available.
Project: State Partnership Initiative (SPI);
Policy issues studied: Mostly focused on provision of vocational
rehabilitation and employment services but also touched on other policy
issues such as increased availability of health insurance;
Year initiated: 1998;
Year completed: Expected completion in 2006;
Cost: Projected to cost $46.1 million, about $8.7 million of it funded
through the Trust Funds and $37.4 million coming from appropriations.
Source: GAO analysis of SSA data.
Note: Most of these projects involved both DI and SSI beneficiaries and
therefore were jointly conducted and funded under SSA's DI
demonstration authority and its SSI demonstration authority.
[End of table]
Furthermore, SSA has not used its DI demonstration authority to
comprehensively examine issues involving vocational rehabilitation,
including key policy issues with which the agency is currently
grappling. For example, SSA did not extensively test key elements of
what eventually became the Ticket to Work program.[Footnote 20]
Although the ticket program was not formally proposed by SSA in a
legislative package until 1997, as early as 1989, in an annual report
to the Congress on SSA's demonstration activities, SSA noted that among
its ideas for improving SSA's ability to assist beneficiaries in
returning to work was a voucher program that could be used to pay for
vocational rehabilitation services from private providers. SSA told the
Congress that such a program, as well as other possible policy changes,
would need to be thoroughly tested as a prerequisite to developing a
new nationwide program. However, only one project completed under SSA's
DI demonstration authority--Project Referral System for Vocational
Rehabilitation Providers (Project RSVP), initiated in 1997--addressed
an issue directly relevant to the ticket program, namely, the use of a
contractor to perform certain administrative functions for an expanded
vocational rehabilitation referral and reimbursement program. But our
review of project documentation and our discussions with SSA officials
indicate that Project RSVP was more of an effort to make an operational
change in the way SSA managed its vocational rehabilitation program
than a study to evaluate the advantages and disadvantages of such a
change. In fact, we could not identify any end product or final results
for this project.
SSA also made another attempt, ultimately unsuccessful, to directly
address issues related to establishment of a ticket program. In the
Omnibus Budget Reconciliation Act of 1990, the Congress mandated that
SSA use its DI demonstration authority to assess the advantages and
disadvantages of permitting DI beneficiaries to select from among both
public and private vocational rehabilitation providers. But in January
1993, SSA reported to the Congress that it would be unable to conduct
this demonstration because of an insufficient number of providers
willing to participate in the project. SSA explained that the
performance-based reimbursement provisions of the proposed project
appeared to be the reason why providers were reluctant to
participate.[Footnote 21] Despite the Congress' expressed interest in
these issues, SSA did not attempt to identify alternative ways to carry
out such a demonstration. In particular, given that SSA remained very
interested in the expanded use of private rehabilitation providers for
the DI program, the difficulties encountered in recruiting providers
for the demonstration should have highlighted the need for SSA to
further study the issue of provider reimbursement before proceeding
with any policy initiatives in this area. SSA's current Deputy
Commissioner for Disability and Income Security Programs told us that
if SSA had used its demonstration authority to study these types of
issues in the 1990s, SSA might have been able to identify and possibly
resolve these issues then rather than struggling to do so now. In
addition, such information could have been helpful in the Congress'
consideration of the ticket legislation's merits as it deliberated
whether to enact this program.
SSA's Recently Proposed Demonstrations Address a Broader Range of
Policy Issues:
In contrast to the completed and nearly completed demonstration
projects, SSA's more recent projects, which are generally in the early
planning or proposal stages, represent a much more wide-ranging set of
demonstrations (see table 2). For example, the projects, as currently
described, will deal with a variety of issues such as early provision
of cash and medical benefits and a change in the benefit payment
structure to allow a benefit offset for beneficiaries earning above the
SGA level. This more comprehensive approach to demonstrations is due in
part to legislative changes. For example, the Ticket to Work Act
mandated that SSA conduct a benefit offset demonstration and also
permitted SSA, for the first time, to conduct demonstrations involving
DI applicants, thereby allowing SSA to test ideas such as early
provision of cash and medical benefits and vocational rehabilitation
services to individuals who have not yet entered the disability rolls.
In addition, SSA has recently placed a high priority on conducting
disability demonstration projects that examine the key issues affecting
beneficiaries' return to work. This priority was reflected in the SSA
Commissioner's September 25, 2003, testimony before the House Committee
on Ways and Means, Subcommittee on Social Security, in which she
announced several new demonstrations as part of a broader strategy to
improve the DI and SSI programs. SSA estimates that these recently
proposed and initiated projects will cost about $357 million, $293
million of which will be paid for from the OASDI Trust Funds.
Table 2: Recently Proposed or Initiated Projects Conducted under SSA's
DI Demonstration Authority:
Project: DI Benefit Offset;
Policy issues studied: Focused on assessment of a benefit offset that
would allow beneficiaries to retain their eligibility for benefits when
earning above the SGA level while offsetting these benefits by $1 for
every $2 in earnings;
it would also assess various employment support interventions in
conjunction with the offset;
Actual/planned year of initiation: 2004;
Expected project duration: 6 years;
Expected cost: $106 million, all from the Trust Funds.
Project: Early Intervention;
Policy issues studied: Focused on provision of cash and medical
benefits and employment supports to DI applicants;
Actual/planned year of initiation: 2005;
Expected project duration: Pilot is expected to last 4 years;
duration of full demonstration (if conducted) is unknown;
Expected cost: $34.4 million, of which $28.4 million is expected to be
funded through the Trust Funds.
Project: Interim Medical Benefits;
Policy issues studied: Focused on provision of medical benefits to DI
applicants with no medical insurance whose condition is likely to
improve with treatment;
Actual/planned year of initiation: Project in preliminary stage;
too early to determine start date;
Expected project duration: 6 years;
Expected cost: $59.8 million, of which $54.9 million is expected to be
funded through the Trust Funds;
(These costs apply to both the Interim and Ongoing Medical Benefits
demonstrations).
Project: Ongoing Medical Benefits;
Policy issues studied: Focused on provision of medical benefits to DI
beneficiaries who want to work but have no affordable access to health
insurance;
Actual/planned year of initiation: Project in preliminary stage;
too early to determine start date, although SSA has proposed starting a
pilot project in 2005;
Expected project duration: Project in preliminary stage;
too early to determine duration;
Expected cost: See above.
Project: Mental Health Treatment Study;
Policy issues studied: Focused on provision of outpatient mental health
treatment and vocational rehabilitation services to DI beneficiaries
for whom a mental health disorder is the primary diagnosis;
Actual/planned year of initiation: Project in preliminary stage;
too early to determine start date;
Expected project duration: 6 years;
Expected cost: $66.9 million, of which $59.7 million is expected to be
funded through the Trust Funds.
Project: Temporary Allowance;
Policy issues studied: Focused on provision of immediate cash and
medical benefits to DI applicants likely to benefit from aggressive
medical care;
Actual/planned year of initiation: Project in preliminary stage;
too early to determine start date;
Expected project duration: 5 years;
Expected cost: $32 million, all from the Trust Funds.
Project: Predictive Modeling[A];
Policy issues studied: Focused on analyses to develop screening tools
that will be used to identify candidates for several of SSA's other
demonstration projects;
Actual/planned year of initiation: 2004;
Expected project duration: 2 years;
Expected cost: $9 million, of which $7.2 million is expected to be
funded through the Trust Funds.
Project: Youth Transition Process Demonstration[B];
Policy issues studied: Focused on states' development of integrated
service delivery systems to improve employment outcomes for youth
transitioning to adulthood;
Actual/planned year of initiation: 2003;
Expected project duration: 6 years;
Expected cost: $48.5 million, of which $4.9 million is expected to be
funded through the Trust Funds.
Source: GAO analysis of SSA data.
Note: Some of these projects are expected to involve both DI and SSI
beneficiaries and therefore will be jointly conducted and funded under
SSA's DI demonstration authority and its SSI demonstration authority.
[A] This is not a demonstration project designed to assess policy
alternatives. Instead, this project is intended to develop a specific
product for use in certain other SSA demonstrations.
[B] This project is focused on the SSI program. However, SSI
beneficiaries who are dually eligible for DI benefits may also
participate in this demonstration.
[End of table]
SSA Lacks a Formal Process for Establishing a Demonstration Research
Agenda:
Despite SSA's recent broadening of the scope of its projects, the
agency does not have in place any systematic processes for identifying
and assessing potential issues that could be well suited for study
under SSA's demonstration project authority. Therefore, there is no
assurance that the agency will, in future demonstration efforts,
maintain its current focus on a broad array of return-to-work policy
issues. Our discussions with SSA officials and review of a study
examining earlier demonstration efforts indicate that the agency's
agenda for demonstration projects is subject to significant change over
time resulting, in part, from changes in executive branch and SSA
leadership and senior management. The effects of such changes may
include termination of projects or significant delays and modifications
in their planning and implementation. For example, in its 1994 report
examining SSA's Research Demonstration Program, the agency's Inspector
General noted that changes in SSA leadership had disrupted the
accomplishment of RDP objectives. The disability research and advisory
officials we spoke with also indicated that SSA's project priorities
and decisions are significantly influenced by larger political and
organizational changes, which may prevent SSA from focusing on long-
term research objectives. One advisory official noted that these
difficulties in long-term planning have occurred despite the fact that
the Congress--in making SSA an independent agency and establishing a 6-
year term for the SSA Commissioner--intended that SSA would be better
able to engage in the type of long-range planning required to address
its program needs.
SSA's approach for identifying and prioritizing demonstrations has
varied through the years. Soon after being granted DI demonstration
authority in 1980, SSA developed a detailed demonstration research plan
to directly address the policy issues identified in SSA's authorizing
legislation. However, our discussions with SSA officials and review of
internal agency documents indicate that the plan was never acted upon
because of competing organizational priorities and concerns over the
potential cost of the demonstrations and possible technical
limitations, such as the adequacy of systems support. Consequently, as
its DI demonstration authority was due to expire in 1985, SSA had not
used it to conduct any demonstrations. In the second half of the 1980s,
after its demonstration authority was renewed, SSA changed course.
Partly on the basis of solicitation of ideas from the public, SSA
identified priority areas dealing mostly with vocational rehabilitation
and employment services issues for which it would issue grants to
public and private organizations to conduct demonstrations. The
specific priority areas identified changed from year to year as SSA
attempted to stimulate, test, and coordinate effective approaches
toward employment assistance.
In its required 1991 annual report to the Congress on its DI
demonstration activities, SSA said that it was proceeding with broader
testing of key elements of a comprehensive employment and
rehabilitation system. But our review of agency documents and
discussions with SSA officials indicate that SSA has not developed a
formal, comprehensive, long-term agenda for conducting DI demonstration
projects. Senior SSA officials told us that the agency's current
demonstration project decisions are, to some extent, based on
discussions with outside research, advocacy, and other groups. But SSA
has no formal mechanisms and requirements in place to ensure that the
agency obtains such input and to decide how such input should be
factored in with other considerations in determining the agency's
demonstration priorities.
The need for explicit planning concerning SSA research, including
demonstrations, has been identified in past reviews of SSA's disability
programs. For example, in 1998, the Social Security Advisory Board
(SSAB) noted the need for SSA to develop a comprehensive, long-range
research and program evaluation plan for DI and SSI that would guide
the agency's research and define priorities.[Footnote 22] SSAB also
said that SSA's research plan should reflect broad consultation with
the Congress, other agencies, SSAB, and others and recommended the
establishment of a permanent research advisory panel to advise in the
development of a long-range plan. In a 1996 report on SSA's disability
programs, the National Academy of Social Insurance noted the "dearth of
rigorous research on the disability benefit programs" since the 1980s
and said that SSA needed a comprehensive, long-range research program
to address this deficiency.[Footnote 23] In addition, officials from
disability research, advisory, and advocacy groups told us that they
believe the establishment of a formal research agenda or an advisory
panel with regard to demonstration projects would be helpful in
ensuring that SSA adequately identifies its demonstration priorities
and maintains its commitment to these priorities even in the face of
political or administrative changes.
SSA's Demonstration Projects Have Had Little Influence on Consideration
of DI Policy Changes:
SSA's demonstration projects have had little influence on the agency's
and the Congress' consideration of DI policy issues. This is due, in
part, to methodological limitations that have prevented SSA from
producing project results that are useful for reliably assessing DI
policy alternatives. In addition, SSA lacks a formal process for fully
considering the potential policy implications of its demonstration
results. Furthermore, SSA's reports on demonstration projects have not
fully apprised the Congress of project results and their policy
implications.
Limitations in Project Design, Implementation, and Evaluation Lessen
Usefulness of Demonstration Results:
The demonstration projects SSA has conducted under its DI demonstration
authority have generally not been designed, implemented, or evaluated
in a rigorous enough manner to allow the agency to reliably assess the
advantages and disadvantages of specific policy alternatives. While
SSA's major DI demonstrations have varied significantly in their
methodological rigor, all of them have experienced at least some
significant methodological limitations.[Footnote 24] For example,
SSA's first major DI demonstration, the Research Demonstration Program,
was characterized by a number of fundamental design and evaluation
flaws such as the limited scope and small sample sizes of the RDP
projects and the limited use of control groups. In its 1994 report on
the RDP, the Department of Health and Human Services' (HHS) Inspector
General noted that because of such limitations, "grantees were unable
to conduct research that SSA deemed necessary for definitive tests of
alternatives to help beneficiaries obtain work."[Footnote 25] In
addition, SSA did not develop a plan for evaluating the overall RDP
results as part of its initial project design. In a required 1994
annual report to the Congress on its demonstration activities, SSA
acknowledged that the lack of a rigorous project design and the
omission of a strong evaluation component limited the ways in which the
project results could be generalized. But SSA also described a number
of "observations" that resulted from the RDP and noted that this
project had helped to identify the agency's future demonstration
priorities. However, given the significant limitations of the RDP, it
is unlikely that its results could have provided a reliable basis for
effectively establishing such priorities.
In its next major DI demonstration effort, Project Network, which was
initiated as the RDP projects were being completed, SSA avoided many of
the major shortcomings of the RDP. For example, Project Network was
rigorously designed, using an experimental approach based on the random
assignment of beneficiaries to treatment and control groups. As a
result, this project produced some reasonably clear results, which SSA
thoroughly evaluated in an effort to assess the overall impact of the
tested policy alternatives. Despite its generally rigorous design,
Project Network also had some limitations that may have, to some
extent, limited its usefulness for policy consideration. For example,
in examining the effects of a case management approach for providing
vocational rehabilitation services, Project Network used four different
service delivery models.[Footnote 26] Although the Project Network
evaluation provided information on the overall effects of a case
management approach, it did not provide a basis for reliably assessing
and comparing the separate effects of the four models even though such
an assessment may have provided useful information for policy
considerations. In addition, Project Network did not produce results
that could be generalized to the larger population of beneficiaries,
which, in turn, limited SSA's ability to assess whether the tested
policy should be implemented on a nationwide basis.[Footnote 27]
As was the case with Project Network, SSA has made a significant effort
under its State Partnership Initiative demonstration to avoid some of
the problems encountered under the RDP. For example, SSA contracted
with two research institutions to design an evaluation plan for the
demonstration and to provide assistance with technical issues and data
collection to the various states conducting this demonstration. Our
discussions with SSA and contractor officials who have been involved in
this demonstration as well as our own review of SPI project documents
indicate that the efforts of the contractors appear to have introduced
a certain degree of rigor in the design, implementation, and,
potentially, evaluation of this demonstration.[Footnote 28] For
example, SSA's contractors have indicated that the SPI "core
evaluation" will likely produce useful results regarding the effects on
beneficiary employment of the overall package of policy alternatives
tested under the demonstration. But despite these efforts, the SPI
design also has a number of limitations that could substantially reduce
the usefulness of its results for evaluating the effects of the
demonstration's individual policy alternatives. For example, SSA gave
each of the 12 participating states significant discretion in designing
and conducting projects, which resulted in 12 distinct state projects.
Each project tested different combinations of policy alternatives,
applied different research methods to study these alternatives, and
used varying approaches to select beneficiaries for participation in
the project. SSA officials told us that such differences across
projects make it unlikely that SPI will produce final results that
allow for reliable evaluations of specific policy alternatives on a
national level. SSA and one of its SPI contractors have also noted
other potential limitations in the design and implementation of SPI,
such as problems with the quality of states' data collection, that may
detract from SSA's ability to evaluate specific policy
alternatives.[Footnote 29]
SSA officials currently responsible for planning and conducting DI
demonstrations acknowledged that the agency's past demonstrations have
generally not provided useful information for policy making largely
because of the limited rigor with which these projects were conducted.
However, they emphasized that the agency has, over the past couple of
years, placed a new emphasis on ensuring that DI demonstrations are
rigorously designed so that the results can be used to effectively
evaluate specific policy options and develop recommendations. In
particular, the officials noted the importance of using, whenever
feasible, an experimental approach in its demonstration projects and of
ensuring that demonstration results can be generalized to the larger
population of DI beneficiaries.[Footnote 30] The officials also
emphasized the need for SSA to hire additional staff with the expertise
needed to carry out methodologically rigorous demonstration projects.
Aside from the SPI demonstration, all of SSA's other current DI
demonstrations are in the early design phase or have been proposed only
recently. Therefore, we were not able to assess the methodological
rigor of these projects. However, our review of SSA's request for
proposal (RFP) for its Benefit Offset demonstration indicates that SSA
is making a serious effort to comprehensively and rigorously study this
policy issue. For example, SSA has proposed using an experimental
design with random assignment to treatment and control groups.
Nevertheless, the scope and complexity of SSA's proposal suggest that
this will be a very challenging project for SSA to carry out
successfully, and that the agency will need to ensure that its project
design avoids some of the pitfalls that have limited the usefulness of
past demonstrations, such as insufficient sample size and lack of
uniformity in tested interventions across sites.
SSA Does Not Ensure That Project Results Are Adequately Considered and
Communicated:
SSA does not have procedures or processes in place to ensure that
project results--regardless of any limitations that they may have--are
fully considered by senior officials within the agency for their policy
implications or their implications for future SSA research and
demonstrations. Without such processes, projects that begin with the
support of senior managers under one administration may not receive
adequate attention from a new group of senior managers under a future
administration. Our discussions with current and former SSA officials
and with officials from disability research, advocacy, and advisory
organizations indicate that such shifting priorities have been the norm
for SSA's DI demonstration projects. For example, several of these
officials told us that when Project Network was completed in 1999, its
results were not formally reviewed and considered by senior SSA
managers, in part because of the changes in presidential
administrations and in senior agency leadership that had occurred since
the start of the project. Officials from one of the groups we spoke
with told us that SSA's consideration of project results could be
improved by the establishment of a panel to review project results and
explore their policy implications.
An additional factor that could limit SSA's consideration of
demonstration results is the lack of an adequate historical record--
reflecting the outcomes and the problems or issues encountered--of the
various projects that the agency has conducted under its demonstration
authority.[Footnote 31] SSA has not maintained a formal record of its
disability demonstration project activities and results, so basic
information on these projects--such as project notices, design
documents, and evaluation documents--is in some cases no longer
available. As a result, information on some projects can be obtained
only by relying on the recollection of SSA employees who were around
when the study was conducted. While formal document retention
requirements may not dictate that SSA maintain such information,
several SSA officials told us that the agency would benefit from an
institutional record of demonstration activity. According to these
officials, such a record would constitute a body of knowledge that the
agency should be building to improve DI return-to-work policies. This
becomes even more important in light of the expected retirement of a
large percentage of SSA staff during this decade.[Footnote 32]
In addition to having shortcomings in its consideration of
demonstration results, SSA has not sufficiently communicated the status
and results of its demonstration projects to the Congress. Although SSA
had been required to issue various reports to the Congress regarding
its DI demonstration projects, it has not always produced such reports.
For example, although SSA was required to submit final reports on the
use of its demonstration authority in 1985, 1990, 1993, and 1996, the
only final report that SSA submitted was in 1996. In addition, SSA did
not submit annual reports on its demonstration activities in 7 of the
16 years in which these reports were required. Furthermore, when these
reports have been produced, they have not provided all of the
information needed to fully inform the Congress of demonstration
activities and results. For example, our review of these reports
indicates that they have frequently lacked key information such as a
discussion of a project's potential policy implications, its
limitations, and the costs of conducting the project.
Conclusions:
In allowing SSA to waive program provisions and use OASDI Trust Fund
dollars, SSA's DI demonstration authority provides the agency with a
special, and potentially very valuable, means of studying policy
alternatives to improve the agency's return-to-work programs. SSA has
spent tens of millions of dollars from the OASDI Trust Funds to conduct
these projects--in addition to tens of millions of dollars from SSA's
general appropriations--and expects to spend hundreds of millions more
within the next 10 years. While these amounts may be small as a
percentage of the total Trust Funds, they nevertheless represent a
substantial use of increasingly limited federal resources. After having
this authority for more than two decades, SSA has yet to use it to
propose or assess major policy options that could result in savings to
the Trust Funds. Because SSA's use of its DI demonstration authority
has yet to achieve the Congress' intended results--and because SSA is
permitted to draw on increasingly limited Trust Funds to conduct these
demonstrations--we believe it is important for the Congress to maintain
close oversight of SSA's use of this authority. We also believe that
such oversight would be a greater challenge if the Congress were to
grant this demonstration authority on a permanent basis.
As the DI Trust Fund approaches exhaustion, the need for programmatic
improvements becomes greater and greater. As part of a broader effort
to address this need, SSA has recently initiated or proposed a number
of DI demonstration projects that, according to SSA officials, are
geared toward producing useful and methodologically sound results. Such
results could provide an important basis for SSA to address some of the
long-standing issues that have led GAO to identify federal disability
programs as a high-risk area. However, the challenges SSA has
historically faced in conducting demonstration projects and the
potential for changing priorities to adversely affect long-range
research plans suggests that, in the long run, SSA may be unable to
fulfill these demonstration goals. This is especially likely if SSA
continues its informal approach to prioritizing and planning
demonstrations and assessing their results. Without more formal
mechanisms for establishing its commitment to effective and thorough DI
demonstrations--including the submission of regular reports to the
Congress on the results and implications of its demonstration projects-
-SSA will be unable to ensure that the extensive amount of time,
effort, and funding devoted to these demonstrations is well spent.
Recommendations:
To help ensure the effectiveness of SSA's DI demonstration projects, we
recommend that the Commissioner of Social Security take the following
actions:
* Develop a formal agenda reflecting the agency's long-term plans and
priorities for conducting DI demonstration projects. In establishing
this agenda, SSA should consult broadly with key internal and external
stakeholders, including SSA advisory groups, disability researchers,
and the Congress.
* Establish an expert panel to review and provide regular input on the
design and implementation of demonstration projects from the early
stages of a project through its final evaluation. Such a panel should
include SSA's key research personnel as well as outside disability
experts and researchers. SSA should establish guidelines to ensure that
its project plans and activities adequately address the issues or
concerns raised by the panel or provide a clear rationale for not
addressing such issues.
* Establish formal processes to ensure that, at the conclusion of each
demonstration project, SSA fully considers and assesses the policy
implications of its demonstration results and clearly communicates
SSA's assessment to the Congress. Such processes should ensure that SSA
consults sufficiently with internal and external experts in its review
of demonstration project results and that SSA issues a report to the
Congress clearly identifying (1) major project outcomes, (2) major
project limitations, (3) total project costs, (4) any policy options or
recommendations, (5) expected costs and benefits of proposed options or
recommendations, and (6) any further research or other actions needed
to clarify or support the project's results. Another key aspect of such
formal processes should be a requirement that SSA maintain a
comprehensive record of DI demonstration projects. This record would
help SSA in establishing an empirically based body of knowledge
regarding possible return-to-work strategies and in deriving the full
value of its substantial investments in demonstration projects.
Matters for Congressional Consideration:
To facilitate close congressional oversight and provide greater
assurance that SSA will make effective use of its DI demonstration
authority, the Congress should consider the following actions:
* Continue to provide DI demonstration authority to SSA on a temporary
basis but allow SSA to complete all projects that have been initiated
prior to expiration of this authority. This would provide SSA with
greater certainty and stability in its efforts to plan and conduct
demonstration projects while preserving the Congress' ability to
periodically reassess and reconsider SSA's overall use of DI
demonstration authority.
* Require that SSA periodically provide a comprehensive report to the
Congress summarizing the results and policy implications of all of its
DI demonstration projects. The due date for this report could either
coincide with the expiration of SSA's DI demonstration authority or, if
this authority is made permanent or extended for a period greater than
5 years, be set for every 5 years. Such reports could serve as a basis
for the Congress' assessment of SSA's use of its demonstration
authority and its consideration of whether this authority should be
renewed.
* Establish reporting requirements that more clearly specify what SSA
is expected to communicate to the Congress in its annual reports on DI
demonstrations. Among such requirements could be a description of all
SSA projects that the SSA Commissioner is considering conducting or is
conducting some preliminary work on. For each demonstration project
that the agency is planning or conducting, SSA should provide clear
information on the projects' specific objectives, potential costs, key
milestone dates (e.g., actual or expected dates for RFP, award of
contracts or grants, start of project operations, completion of
operations, completion of analysis, and final report), potential
obstacles to project completion, and the types of policy alternatives
that SSA might consider pursuing depending on the results of the
demonstration. This would provide the Congress with a more complete
understanding of the direction and progress of SSA in its efforts to
fulfill its DI demonstration requirements.
* More clearly specify the methodological and evaluation requirements
for DI demonstrations to better ensure that such projects are designed
in the most rigorous manner possible and that their results are useful
for answering specific policy questions and for making, where
appropriate, well-supported policy recommendations. Such requirements
should not be entirely prescriptive given the need for SSA to have
sufficient flexibility for choosing the right methodological approach
based on the specific circumstances and objectives of a particular
demonstration project. However, the requirements could call for SSA to
choose, to the extent practical and feasible, the most rigorous methods
possible in conducting these demonstrations. Whatever methods are
ultimately selected, SSA should be sure that the methods used will
allow for a reliable assessment of the potential effect on the DI
program of the individual policy alternatives being studied. Finally,
SSA's legislative requirements could be revised to include a more
explicit list of project objectives--such as assessments of specific
employment outcomes, costs and benefits, and Trust Fund savings--
similar to the language that was included under Sections 302(b)(1) and
(b)(2) of the Ticket to Work and Work Incentives Improvement Act.
Agency Comments and Our Evaluation:
In commenting on a draft of this report, SSA agreed with our
recommendations. SSA agreed that in the past it has not used its
demonstration authority to extensively evaluate DI policy but noted
that its recently initiated or proposed demonstrations will play a
vital role in testing program and policy changes. SSA also agreed that
the use of experts in developing demonstration projects is very useful
and commented that it has used the expertise of particular individuals
on an ad hoc basis and plans to continue to use the advice and
recommendations of experts in the development of future demonstrations.
Finally, SSA agreed that a central source of information regarding the
results and policy implications of disability demonstrations needs to
be established and stated that it planned to fully analyze the results
of demonstration projects to inform DI policy decisions. SSA's comments
appear in appendix II.
Copies of this report are being sent to the Commissioner of SSA,
appropriate congressional committees, and other interested parties. The
report is also available at no charge on GAO's Web site at http://
www.gao.gov. If you have any questions about this report, please
contact me at (202) 512-7215. Other contacts and staff acknowledgments
are listed in appendix III.
Signed by:
Robert E. Robertson:
Director, Education, Workforce, and Income Security Issues:
[End of section]
Appendix I: Scope and Methodology:
To address the mandated objectives, we reviewed legislation authorizing
the Social Security Administration (SSA) to conduct Disability
Insurance (DI) demonstration projects, Congressional reports related to
this legislation, and SSA regulations governing DI demonstration
activities. We also examined internal SSA memorandums and planning
documents discussing proposals to conduct demonstration projects and
the nature, purpose, requirements, and distinguishing features of SSA's
demonstration authority. We interviewed a wide range of current and
former SSA officials who have had involvement in or responsibility for
conducting disability program demonstration projects, including
officials from the Office of Disability and Income Security Programs
(ODISP) and two offices operating under ODISP--the Office of Program
Development and Research and the Office of Employment Support Programs-
-as well as officials from the Office of the Chief Actuary, the Office
of Acquisition and Grants, the Office of Budget, the Office of
Strategic Management, and the Office of Research, Evaluation, and
Statistics. We also interviewed officials from disability research,
advisory, and advocacy organizations. In addition, we examined other
reviews of SSA's disability demonstration and research programs,
including prior GAO and Inspector General reports and reports from
disability research and advisory groups. We also reviewed SSA budget
documents identifying agency spending on disability program
demonstrations and SSA testimony describing agency priorities related
to the DI program in general and demonstration projects in particular.
In addition, we examined SSA's strategic plan, annual performance
plans, and annual accountability reports.
To obtain detailed information on SSA's DI demonstration projects, we
reviewed various documents related to SSA's design, implementation, and
evaluation of demonstration projects including agency reports to the
Congress; public notifications of demonstration projects issued in the
Federal Register; contract, grant, and cooperative agreement
solicitation and award notices issued in the Federal Register or in the
Commerce Business Daily; and project reports submitted to SSA by
grantees or contractors, including project design and evaluation
documents. We used information from these sources to identify key
characteristics and outcomes of each project, including its broad
goals, specific study objectives, types of program waivers applied,
methodology, actual or expected costs, funding sources, major project
milestones including actual or expected initiation and completion
dates, project duration, involvement of outside contractors and
grantees, key project strengths and limitations, and final project
results, including any recommendations that may have been made. The
type and extent of information we obtained for each demonstration
project varied widely, in large part because SSA has not maintained
comprehensive documentation on its prior demonstrations. In addition,
documentation on SSA's more recent demonstrations was very limited
given that these projects are in the early planning and design stages.
To provide a broader context for understanding SSA's use of its
demonstration authority, we reviewed other federal agencies'
legislative authorities for conducting demonstration and research
activities. We also examined reports from GAO and other organizations
that evaluated demonstration and research projects conducted by other
federal agencies or that identified key evaluation and methodological
issues related to such projects.
We performed our work at SSA headquarters in Baltimore, Maryland, and
at various locations in Washington, D.C. We conducted our work between
October 2003 and August 2004 in accordance with generally accepted
government auditing standards.
[End of section]
Appendix II: Comments from the Social Security Administration:
SOCIAL SECURITY:
The Commissioner:
October 21,2004:
Mr. Robert E. Robertson:
Director, Education, Workforce and Income Security Issues:
U.S. Government Accountability Office:
Room 5-T-57:
441 G Street, NW:
Washington, D.C. 20548:
Dear Mr. Robertson:
Thank you for the opportunity to review and comment on the draft report
"Social Security Disability: Improved Processes for Planning and
Conducting Demonstrations May Help SSA More Effectively Use Its
Demonstration Authority" (GAO-05-19). Our comments on the report are
enclosed.
If you have any questions, please have your staff contact Candace
Skumik, Director, Audit Management and Liaison Staff, at (410) 965-
4636.
Sincerely,
Signed by:
Jo Anne B. Barnhart:
Enclosures (2):
COMMENTS ON THE GOVENMENT ACCOUNTABILITY OFFICE (GAO) DRAFT REPORT
"SOCIAL SECURITY DISABILITY: IMPROVED PROCESSES FOR PLANNING AND
CONDUCTING DEMONSTRATIONS MAY HELP THE SOCIAL SECURITY ADMINISTRATION
MORE EFFECTIVELY USE ITS DEMONSTRATION AUTHORITY" (GAO-05-19):
We appreciate the opportunity to comment on the draft report concerning
the Social Security Administration's (SSA) use of demonstration
authority granted by Congress to test the overall effectiveness of
possible program changes that could encourage individuals to return to
work and decrease their dependence on Disability Insurance (DI)
benefits.
We agree with your finding that, in the past, we have not used the
demonstration authority to extensively evaluate DI policy. We are
pleased that the report acknowledges recent changes and that "... SSA's
recently proposed or initiated demonstrations have begun to address a
broader range of policy issues..." We believe that SSA has made several
key changes in policy that will address this issue.
As the Commissioner's new approach to the disability determination
process is developed, our disability demonstrations will play a vital
role in testing program and policy changes that will enable SSA to
provide the appropriate supports and services to applicants and
beneficiaries that will best serve their individual needs. The ultimate
goal of these disability demonstration projects is to identify
individuals, as early as possible, who could retain or attain
employment with adequate supports or services.
As decisions and plans are made regarding the implementation of the
Commissioner's new approach, which emphasizes return to work at each
step of the disability determination process, SSA is coordinating the
implementation with the activities constituting our comprehensive Work
Opportunity Vision. SSA's Work Opportunity Vision includes the
disability demonstration projects, plans to expand and simplify work
incentives, and our proposed improvements to the Ticket Program. These
coordinated activities will have a combined impact on streamlining the
return to work process and assisting our applicants and beneficiaries
in maximizing their economic self-sufficiency through employment.
We also agree with your recommendations and are particularly pleased
that the report recognizes our increased effort to address disability
program improvements, as evidenced by our recently initiated and
proposed disability demonstration projects. The outcomes of these
projects should go a long way towards the removal of our disability
program from the Federal disability programs high-risk list.
Our responses to the specific recommendations are provided below:
Recommendation 1:
SSA should develop a formal agenda reflecting the Agency's long-term
plans and priorities for conducting DI demonstration projects. In
establishing this agenda, SSA should consult broadly with key internal
and external stakeholders, including SSA advisory groups, disability
researchers, and Congress.
Response:
We agree and already have begun work on a formal disability research
agenda that would answer key questions associated with disability
policy making. We see the research conducted through our disability
demonstration projects as an integral part of the management of SSA's
programs and as a key component of our Work Opportunity Vision and the
Commissioner's new approach to the disability determination process. We
anticipate the results from our research will inform DI policy
decisions and may direct future studies on particular issues or
populations. We plan to consult with interested groups and individuals
as development of the agenda proceeds.
Recommendation 2:
SSA should establish an expert panel to review and provide regular
input on the design and implementation of demonstration projects from
the early stages of a project through its final evaluation. Such a
panel should include SSA's key research personnel as well as outside
disability experts and researchers. SSA should establish guidelines to
ensure that its project plans and activities adequately address the
issues or concerns raised by the panel or provide a clear rationale for
not addressing such issues.
Response:
We agree that the use of experts in developing demonstration projects
is very useful. However, we have used the expertise of particular
individuals on mad hoc basis. This allows us to tap individuals with
particular expertise for project development, but also allows those
experts to participate in some of the contracts. To eliminate conflicts
of interest as we develop these projects, experts we utilize in the
project development must forgo participation in the project. While we
recognize the value of this expert advice, we do not want to limit the
use of experts in the implementation of these disability
demonstrations. We plan to continue to use the advice and
recommendations of experts in the development of future demonstrations.
Recommendation 3:
SSA should establish formal processes to ensure that, at the conclusion
of each demonstration project, SSA fully considers and assesses the
policy implications of its demonstration results and clearly
communicates SSA's assessment to Congress. Such processes should ensure
that SSA consults sufficiently with internal and external experts in
its review of demonstration results and that SSA issues a report to
Congress clearly identifying: major project outcomes; major project
limitations; total project costs; any policy options or
recommendations; expected costs and benefits of proposed options or
recommendations; and any further research or other actions needed to
clarify or support the project's results. Another key aspect of such
formal processes should be a requirement that SSA maintain a
comprehensive record of DI demonstration projects. This record would
help SSA in establishing an empirically based body of knowledge
regarding the possible return to work strategies and to derive the
full value of its substantial investments in demonstration projects.
Response:
We agree that a central source of information regarding the results and
policy implications of disability demonstrations needs to be
established. Our intent and plan is to fully analyze the results of
demonstration projects to infonn DI policy decisions, including the
implementation of the Commissioner's new approach to the disability
determination process. We also agree that we should communicate the
results to Congress.
[End of section]
Appendix III: GAO Contacts and Staff Acknowledgments:
GAO Contacts:
Shelia D. Drake, Assistant Director, (202) 512-7172 Mark Trapani,
Analyst-in-Charge, (202) 512-6513:
Staff Acknowledgments:
The following individuals also made important contributions to this
report: Jacquelyn D. Stewart, Erin M. Godtland, Corinna A. Nicolaou,
Daniel A. Schwimer, Ronald La Due Lake, Michele C. Fejfar.
FOOTNOTES
[1] GAO identifies federal programs and operations that are high-risk
because of their greater vulnerability to fraud, waste, abuse, and
mismanagement. GAO has also increasingly used its high-risk designation
to draw attention to the challenges faced by government programs and
operations in need of broad-based transformation. See GAO, High-Risk
Series: An Update, GAO-03-119 (Washington, D.C.: January 2003) and GAO,
Major Management Challenges and Program Risks: Social Security
Administration, GAO-03-117 (Washington, D.C.: January 2003).
[2] For 2004, SSA considers countable earnings above $810 a month to be
substantial gainful activity for persons who are not blind and $1,350 a
month for persons who are blind.
[3] The full retirement age is the earliest age at which an unreduced
retirement benefit is payable. The age for full retirement benefits is
scheduled to rise gradually from age 65 to 67.
[4] In 2002, the DI program also paid about $6 billion in cash benefits
to about 1.7 million spouses and children of disabled workers. DI cash
benefits are paid from the Federal Disability Insurance Trust Fund.
This trust fund is funded through payroll deductions paid by employers
and workers.
[5] For disabled and blind individuals who have low income and limited
resources, SSA provides cash benefits under the Supplemental Security
Income (SSI) program. Although both the DI and the SSI programs use the
same definition of disability, SSI has no prior work requirement. In
2002, SSA paid about $26 billion in SSI benefits to about 5.5 million
people with disabilities, of whom 3.9 million were working age adults
aged 18 to 64. The average monthly federal SSI cash benefit amounted to
about $398 per person. In 2002, about 1.2 million DI beneficiaries were
dually eligible for SSI benefits because of the low level of their
income and resources.
[6] This DI demonstration authority was provided under Section 505(a)
of the Social Security Disability Amendments of 1980 (Pub. L. No. 96-
265). Section 505(b) of this act amended Section 1110 of the Social
Security Act to provide SSA with similar demonstration authority for
the SSI program, including authority to waive SSI program rules.
However, unlike the DI demonstrations, SSI demonstration projects were
to be funded from congressional appropriations. Although SSA's DI and
SSI demonstration authorities are separate and somewhat distinct, the
agency's disability demonstration projects often involve both DI and
SSI beneficiaries and applicants. When a demonstration is conducted
under both the DI and SSI demonstration authorities, funding for the
project is also split between trust fund (i.e., DI) and appropriated
(i.e., SSI) sources.
[7] In a report discussing federal evaluation activities, GAO defined
demonstration projects as "those that aim to provide evidence of the
feasibility or effectiveness of a new approach or practice." See GAO,
Program Evaluation: Improving the Flow of Information to Congress, GAO-
PEMD-95-1 (Washington, D.C.: January 30, 1995). This definition is
consistent with the purpose of SSA's DI demonstration projects, as
spelled out in legislation.
[8] Although SSA has temporary authority for DI demonstrations, it has
permanent authority to conduct SSI demonstrations.
[9] In addition to extending the authority to initiate DI demonstration
projects through 2005, Pub. L. No. 108-203 would also allow DI
demonstrations that SSA has initiated on or before December 17, 2005,
to be completed thereafter.
[10] The results of a rigorous demonstration may lead SSA to recommend
adoption of a particular policy option or, alternatively, to recommend
that such an option not be pursued. Also, to the extent that the
results of a demonstration are not definitive enough to make such
policy recommendations, SSA may instead propose that additional
research be conducted to further assess policy alternatives.
[11] Experimental designs involve random assignment of study
participants to either a treatment group or a control group. The
treatment group is subjected to the new program or policy, and the
control group is not. The strength of the experimental design is in its
assurance that those who experience the treatment are like, in all
important ways, those who do not experience the treatment except for
the difference in receiving the treatment itself.
[12] Ticket to Work and Work Incentives Advisory Panel, Advice Report
to the Commissioner of Social Security: Statutory Requirements and
Design Issues Related to SSDI $1 for $2 Benefit Offset Research, p. 12
(Washington, D.C.: August 2002).
[13] For example, in a 1997 publication examining welfare reform
evaluations, Besharov and others state that with experimental designs
"Policymakers can then focus on the implications of findings, rather
than 'become entangled in a protracted and often inconclusive
scientific debate about whether the findings of a particular study are
statistically valid.'" See Douglas J. Besharov, Peter Germanis, and
Peter H. Rossi, Evaluating Welfare Reform, A Guide for Scholars and
Practitioners, the University of Maryland, School of Public Affairs,
1997, p. 42.
[14] Alternative methods involving quasi-experimental designs do not
involve the use of randomized control groups. Instead, such designs
rely on the nonrandom selection of a comparison group of
nonparticipants with characteristics similar to those of study
participants.
[15] Although experimental methods are generally preferred, they may
not always be feasible or may not necessarily provide the optimal
approach for conducting a demonstration because of various
implementation or ethical considerations. Quasi-experimental methods
may instead be preferable or may need to be used in conjunction with
experimental methods to produce the most rigorous results. For
instance, quasi-experimental methods may be preferred when there is a
chance that a randomized approach will not remain intact, such as when
participants in the control group are likely to receive elements of the
intervention.
[16] OPDR is one of several organizational components under SSA's
Office of Disability and Income Security Programs (ODISP), which has
responsibility for carrying out DI and SSI program functions. Prior to
OPDR's creation in November 2002, responsibility for disability
demonstration projects was shared by components within ODISP and SSA's
Office of Policy.
[17] While grants and cooperative agreements are closely related to
each other, cooperative agreements generally allow for greater federal
agency involvement in the funded activities. Contracts generally
provide federal agencies with more control over funded activities than
either grants or cooperative agreements.
[18] SSA also initiated three other projects under its DI demonstration
authority, which were subsequently terminated. In addition, SSA has
sometimes conducted demonstration projects involving DI beneficiaries
under other research authorities (rather than under its DI
demonstration authority). Although our discussion in this report
focuses only on those projects that were conducted under SSA's DI
demonstration authority, our review of these other demonstration
projects suggests that our findings would apply similarly to them.
[19] We were not able to obtain complete cost figures for these
projects because of the limited information maintained by SSA. Because
most of these projects involved both DI and SSI beneficiaries, they
were funded through the use of OASDI Trust Funds (under SSA's DI
demonstration authority) and general appropriations (as provided under
SSA's SSI demonstration authority).
[20] Under this program, beneficiaries are issued a "ticket," or
voucher, which they can use to obtain vocational rehabilitation,
employment, or other return-to-work services from an approved public or
private provider of their choice.
[21] These provisions permitted reimbursement only for rehabilitation
services which result in a beneficiary's performance of substantial
gainful activity for a continuous period of 9 months. The provisions
were identical to those in place for reimbursing state vocational
rehabilitation agencies.
[22] Social Security Advisory Board, Strengthening Social Security
Research: The Responsibilities of the Social Security Administration,
p. 6-7 (Washington, D.C.: January 1998).
[23] National Academy of Social Insurance, The Environment of
Disability Income Policy: Programs, People, History, and Context, p.
108-109 (Washington, D.C.: 1996).
[24] Our discussion in this section is based on our review of SSA's
three major DI demonstrations conducted since 1980--the Research
Demonstration Program, Project Network, and the State Partnership
Initiative. For the two other projects SSA has conducted under its DI
demonstration authority--the Pain Assessment Instruments Development
Project and Project RSVP--SSA was unable to provide, and we were unable
to otherwise identify, any documentation or information other than
relatively brief project descriptions provided in SSA reports or
notices. However, these project descriptions as well as information
obtained in discussions with SSA officials who were knowledgeable of
these projects indicate that these projects were not research efforts
designed to assess particular return-to-work policy options but instead
represented attempts by SSA to implement operational changes in the DI
program.
[25] Department of Health and Human Services, Office of Inspector
General, Impact of SSA's Disability Research Demonstration Program,
OEI-04-91-01660, pp. 5 (Washington, D.C.: September 1994).
[26] The researchers involved in Project Network described case
management as "a tool for facilitating employment-oriented
interventions customized for each individual, given that persons with
disabilities face a range of barriers to work." The four models for
providing case management services used in Project Network included one
that was operated by a private contractor and another operated by state
vocational rehabilitation agencies, as well as two other models that
were operated by SSA but differed in terms of the intensity of services
offered.
[27] Because participation in the demonstration was voluntary, the
project's results could not be generalized to the broader population of
beneficiaries, most of whom did not volunteer. For example, the
project's volunteers were generally healthier and had fewer work
limitations than those who did not volunteer.
[28] The SPI demonstration is still in process and is expected to be
completed in 2006.
[29] Because the SPI demonstration is not yet complete, it is possible
that SSA and its contractors will take additional steps to address
these limitations. However, information obtained through our
discussions with SSA and SPI contractor officials provided little
indication that such problems were likely to be addressed.
[30] The officials noted that being able to generalize to the larger DI
population does not necessarily mean that a policy tested through a
demonstration has to be applicable to all or most DI beneficiaries, but
could instead be applicable to a subgroup of beneficiaries. For
example, an SSA demonstration might evaluate the costs and benefits of
a policy option that is targeted toward beneficiaries with mental
impairments. In this case, SSA would want to be assured that the
project's results are generalizable to the larger population of
beneficiaries with mental illnesses, but it would not expect
generalizability to other DI beneficiaries (i.e., those with other
types of impairments).
[31] Although our focus in this report is on DI demonstrations, during
the course of our work, we had also requested information on SSI
demonstrations and were similarly unable to identify a historical
record at SSA.
[32] In its strategic plan for fiscal years 2003-2008, SSA said that
during this decade, over 28,000 of its federal employees will retire
and another 10,000 will leave the agency for other reasons, which
represents approximately 59 percent of the agency's current workforce.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
