Electronic Disability Claims Processing
SSA Needs to Address Risks Associated with Its Accelerated Systems Development Strategy Gao ID: GAO-04-466 March 26, 2004The Social Security Administration's (SSA) AeDib initiative is designed to provide SSA with a more efficient, paperless system that will enable its disability components to electronically view and share claims data and process claims electronically. Yet previous GAO reviews found that SSA's accelerated strategy to develop AeDib involved risks that could threaten a complete and successful transition to this capability. At the Subcommittee's request, GAO reviewed AeDib to assess (1) SSA's progress and strategy, (2) the adequacy of measures taken to avoid software development problems similar to those encountered in SSA's previous efforts, (3) the adequacy of cost/benefit analyses, and (4) SSA's consultation with stakeholders.
SSA is continuing its work on the AeDib initiative and is in various stages of completing its electronic disability system; however, its accelerated strategy continues to involve risks. Specifically, GAO found that the agency is relying on limited pilot testing to help guide business and technical decisions and ensure that technology supporting the electronic disability system will work as intended. Further, it is beginning its national rollout without ensuring that all critical problems identified in the pilot testing have been resolved and without conducting testing adequate to evaluate the performance of all system components collectively. Without resolution of critical problems and full testing, SSA cannot be assured that interrelated components will work together successfully. While SSA has procedures to guide its software development, it could provide no evidence that it was consistently applying them in this case. In addition, while SSA has identified AeDib system and security risks, it has not finalized mitigation strategies. Without these measures being in place, SSA stands at greater vulnerability to circumstances that could impede project success. The agency also has not validated its analysis to ensure the reasonableness of estimated AeDib costs and benefits. While indicating that it would use pilot test results to validate cost-benefit estimates, officials have not indicated when this will be accomplished. This leaves SSA without a validated cost-benefit analysis, and the assurance that its AeDib cost estimates are reliable and that anticipated benefits will therefore be realized. Finally, SSA reports that it has increased its communications with AeDib stakeholders and users; however, state officials dealing with disability determinations have varying perspectives. A national organization representing these state officials continues to voice concerns about SSA's approach. And while the Commissioner states that SSA is consulting with stakeholders and the medical community, the agency has not articulated a comprehensive plan for ensuring that the concerns of this population are addressed.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-04-466, Electronic Disability Claims Processing: SSA Needs to Address Risks Associated with Its Accelerated Systems Development Strategy
This is the accessible text file for GAO report number GAO-04-466
entitled 'Electronic Disability Claims Processing: SSA Needs to Address
Risks Associated with Its Accelerated Systems Development Strategy'
which was released on March 26, 2004.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Chairman, Subcommittee on Social Security, Committee on
Ways and Means, House of Representatives:
March 2004:
ELECTRONIC DISABILITY CLAIMS PROCESSING:
SSA Needs to Address Risks Associated with Its Accelerated Systems
Development Strategy:
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-466]:
GAO Highlights:
Highlights of GAO-04-466, a report to the Chairman, Subcommittee on
Social Security, Committee on Ways and Means, House of Representatives
Why GAO Did This Study:
The Social Security Administration‘s (SSA) AeDib initiative is
designed to provide SSA with a more efficient, paperless system that
will enable its disability components to electronically view and share
claims data and process claims electronically. Yet previous GAO
reviews found that SSA‘s accelerated strategy to develop AeDib
involved risks that could threaten a complete and successful
transition to this capability.
At the Subcommittee‘s request, GAO reviewed AeDib to assess (1) SSA‘s
progress and strategy, (2) the adequacy of measures taken to avoid
software development problems similar to those encountered in SSA‘s
previous efforts, (3) the adequacy of cost-benefit analyses, and (4)
SSA‘s consultation with stakeholders.
What GAO Found:
SSA is continuing its work on the AeDib initiative and is in various
stages of completing its electronic disability system; however, its
accelerated strategy continues to involve risks. Specifically, GAO
found that the agency is relying on limited pilot testing to help
guide business and technical decisions and ensure that technology
supporting the electronic disability system will work as intended
(see table). Further, it is beginning its national rollout without
ensuring that all critical problems identified in the pilot testing
have been resolved and without conducting testing adequate to evaluate
the performance of all system components collectively. Without
resolution of critical problems and full testing, SSA cannot be
assured that interrelated components will work together
successfully.
Pilot Tests are Limited in Scope, Involving Few Examiners:
State/pilot start date: North Carolina/July 2003;
Number/percentage of examiners participating: 21 of 254/8 percent.
State/pilot start date: Illinois/September 2003;
Number/percentage of examiners participating: 15 of 250/6 percent.
State/pilot start date: California/October 2003;
Number/percentage of examiners participating: 2 of 620/0.3 percent.
Source: GAO analysis of SSA data.
[End of table]
While SSA has procedures to guide its software development, it could
provide no evidence that it was consistently applying them in this
case. In addition, while SSA has identified AeDib system and security
risks, it has not finalized mitigation strategies. Without these
measures being in place, SSA stands at greater vulnerability to
circumstances that could impede project success.
The agency also has not validated its analysis to ensure the
reasonableness of estimated AeDib costs and benefits. While indicating
that it would use pilot test results to validate cost-benefit
estimates, officials have not indicated when this will be
accomplished. This leaves SSA without a validated cost-benefit
analysis, and the assurance that its AeDib cost estimates are reliable
and that anticipated benefits will therefore be realized.
Finally, SSA reports that it has increased its communications with
AeDib stakeholders and users; however, state officials dealing with
disability determinations have varying perspectives. A national
organization representing these state officials continues to voice
concerns about SSA‘s approach. And while the Commissioner states that
SSA is consulting with stakeholders and the medical community, the
agency has not articulated a comprehensive plan for ensuring that the
concerns of this population are addressed.
What GAO Recommends:
GAO recommends that before proceeding with national rollout of AeDib,
the Commissioner of Social Security, among other steps, ensure that
critical problems have been resolved and full testing completed,
expedite completion of risk-mitigation strategies, and validate cost-
benefit estimates. In commenting on a draft of this report, SSA stated
that it would conduct studies to help validate AeDib cost assumptions
but disagreed with GAO‘s other recommendations. In GAO‘s view, it is
essential that SSA fully address all recommendations.
www.gao.gov/cgi-bin/getrpt?GAO-04-466.
To view the full product, including the scope and methodology, click
on the link above. For more information, contact Linda D. Koontz at
(202) 512-6240 or koontzl@gao.gov.
[End of section]
Contents:
Letter:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendixes:
Appendix I: GAO's January 30, 2004, Briefing:
Appendix II: Comments from the Social Security Administration:
Letter March 26, 2004:
The Honorable E. Clay Shaw, Jr. Chairman,
Subcommittee on Social Security:
Committee on Ways and Means:
House of Representatives:
Dear Mr. Chairman:
As you know, the Social Security Administration (SSA) is currently
pursuing a major initiative to establish an electronic disability
claims processing capability. With this initiative--known as AeDib--SSA
aims to develop and implement a more efficient, paperless system that
will enable its disability processing components to electronically view
and share claims information, including large volumes of medical
images, files, and other documents currently maintained in paper
folders. SSA's pursuit of an electronic disability process represents a
positive and necessary step toward more efficient delivery of benefits
payments and services to an increasing beneficiary population. However,
as we have previously reported,[Footnote 1] the agency's accelerated
strategy to develop the electronic disability system involves risks
that could pose a threat to the complete and successful transition to
this capability.
At your request, we have been reviewing SSA's continued actions on the
AeDib initiative. Specifically, our objectives were to:
* assess SSA's progress toward and strategy for achieving AeDib,
* determine the adequacy of measures SSA is taking with AeDib to avoid
the kinds of software development problems that it encountered with its
prior disability system initiative and to identify and mitigate risks
to a successful development,
* assess the adequacy of SSA's analysis of AeDib costs and benefits,
and:
* assess SSA's consultation with and support from key stakeholders.
To address these objectives, we analyzed project management and
technical documentation describing SSA's plans, strategies, and
progress related to developing and implementing the electronic
disability system. We identified and evaluated measures--including
institutional software capability mechanisms--that the agency had taken
to guide its development of the AeDib software and mitigate program and
project risks. We also assessed the adequacy of SSA's analyses of AeDib
costs and benefits, including evaluating them against federal guidance
and industry best practices and analyzing the impact of critical pilot
evaluation data on costs and benefits. In addition, we analyzed SSA's
plans and actions for consulting with and identifying and resolving
issues of key disability stakeholders (i.e., state Disability
Determination Services [DDS] offices, medical providers, and SSA field
offices). We performed our work in accordance with generally accepted
government auditing standards, from August 2003 through January 2004.
On January 30, 2004, we provided your staff with a briefing on the
results of our study. The slides from that briefing are included as
appendix I to this report. The purpose of this report is to provide the
published briefing slides to you and to officially transmit our
recommendations to the Commissioner of Social Security.
In summary, our briefing made four main points:
* While the AeDib initiative is important to achieving more efficient
delivery of disability payments, SSA's accelerated strategy continues
to involve risks that threaten its ultimate success. SSA is relying on
limited pilot testing to help guide business and technical decisions
and ensure that technology supporting the electronic disability folder
will work as intended. Further, the agency is beginning its national
rollout without ensuring that all critical problems identified during
the pilot tests have been resolved and without conducting end-to-end
tests to evaluate the functionality and performance of all electronic
disability system components collectively. SSA has maintained that its
pilot tests of the components would be sufficient to evaluate the
system; however, without end-to-end testing, SSA lacks assurance that
the interrelated components will work together successfully.
* While SSA has established processes and procedures to guide its
software development activities, such as project management plans, it
could not provide evidence that it was consistently applying these
procedures to AeDib. In addition, while SSA has identified AeDib system
and security risks, it has not finalized mitigation strategies. Without
concurrence on software validation and system certifications, or risk
mitigation strategies, SSA lacks assurance that its system will be
acceptable to end users and it will not be positioned to effectively
prevent circumstances that could impede project success.
* SSA has not validated its analysis to ensure the reasonableness of
estimated AeDib costs and benefits. The agency stated that it would use
pilot test results to validate its cost-benefit estimates; however, its
officials have not indicated when this will be accomplished. Without a
validated cost-benefit analysis, SSA lacks assurance that its AeDib
cost estimates are reliable and that anticipated benefits will be
realized.
* While SSA has stated that it has increased its communications with
AeDib stakeholders and users and instituted mechanisms to facilitate
its communications, state DDSs have varying views regarding their
involvement. Specifically, while state disability offices
participating in the AeDib pilots agree that communications have
improved, a national organization representing these offices continues
to have concerns about SSA's approach. In addition, although the
Commissioner has stated that SSA is consulting with stakeholders and
the medical community, SSA has not articulated a comprehensive plan for
ensuring that all of these concerns are addressed.
In light of the risks associated with the AeDib initiative, SSA's
successful transition to an electronic disability claims process could
be jeopardized unless it addresses the weaknesses described above.
Recommendations for Executive Action:
To reduce the risks associated with SSA's strategy to develop an
electronic disability claims processing system, we recommend that the
Commissioner of Social Security, before continuing with the AeDib
national rollout,
* ensure that all critical problems identified in pilot testing of the
electronic disability system components are resolved and that end-to-
end testing of the interrelated systems is performed;
* ensure that users have approved the software developed and that
systems have been certified for production;
* establish a revised time frame for and expedite actions toward
finalizing AeDib risk mitigation strategies;
* validate all AeDib cost and benefit estimates; and:
* implement a communications plan that clearly and comprehensively
conveys SSA's approach for effectively addressing disability
stakeholders' and users' concerns and ensuring their full involvement
in the AeDib initiative.
Agency Comments and Our Evaluation:
In providing written comments on a draft of this report (reprinted in
app. II), the Commissioner of Social Security reiterated the
department's general disagreement with the findings discussed in our
briefing slides. The Commissioner added that, in considering our
recommendations, the agency had not been compelled to change its
direction on the AeDib initiative.
In particular, SSA continued to disagree with the need for end-to-end
testing, stating that to perform such testing would delay the agency's
ability to realize benefits from this initiative. SSA estimated that
following our recommendation would have delayed the project, and the
resulting benefits, by at least 3 years. Given the technological
complexity of the AeDib project, we remain convinced that end-to-end
testing is essential to ensuring that the multiple interrelated
components of the electronic disability system will operate as
intended. As our report noted, at the time that SSA began national
implementation of the electronic disability system, it had conducted
only limited pilot testing of the Document Management Architecture that
is fundamental to the electronic disability folder. It also had not
fully evaluated the results of its pilot tests or ensured that all
critical problems were resolved. Further, we disagree with SSA's
contention that end-to-end testing would delay the AeDib project by 3
years. In referring to this time frame, SSA has emphasized the
processing of its most comprehensive cases--those in which individuals
pursue their disability claims through all levels of agency appeal--
which reportedly take an average of 1,153 days to reach a final
decision. However, not every case that SSA processes requires this
level of appeal; thus, some cases can be processed in less time. In
addition, SSA has reported that, due to backlogs, cases that go through
all levels of appeal spend nearly 50 percent of that time in queue,
waiting for agency action. Given these factors, we believe SSA has an
opportunity to apply a more expedited approach to end-to-end testing,
focused on evaluating certain claims that require some, but not all
levels of agency appeal, and processing them outside the normal backlog
of cases. Without full awareness and resolution of critical problems
and end-to-end testing, SSA cannot ensure that the electronic
disability system components will work together effectively. Thus, SSA
could incur future unanticipated costs to correct the system, and
stands to increase its risk of not realizing the very benefits that it
seeks, by proceeding with AeDib without this critical testing.
In addition, SSA said it considered moot our recommendation that the
agency ensure that users have approved new software and that it certify
its systems for production, stating that the agency has always
performed these tasks. In addition, SSA said that our concern in this
area had pertained to the lack of wet signatures[Footnote 2] on its
project documents. In speaking to this issue, our primary concern was
not that SSA lacked wet signatures on its software development
documentation. Rather, during the course of our review, agency
officials were unable to provide readily verifiable evidence that users
had approved software developed for the electronic disability system,
or that systems had been certified for production. At the end of our
study, SSA provided certain documentation supporting its software
validation and certification efforts; however, the documents lacked
sufficient details for us to fully evaluate and reach conclusions about
the extent to which all software had been validated by the users and
certified for production. For any project, and especially one of
AeDib's size and complexity, having readily verifiable evidence that
systems were developed in accordance with specified requirements is
essential to ensuring that users' needs will be met by the software
developed and that systems can be certified for production.
Regarding AeDib program and project risks, SSA stated that it believed
the success of AeDib over the last 26 months had attested to the
agency's diligent and ongoing efforts to mitigate risks. Nonetheless,
as our report stresses, strategies for addressing program-level and
security-level risks are needed to help prevent circumstances that
could impede successful project outcomes. In the absence of explicit
mitigation strategies, SSA increases the probability of failing to meet
project commitments before they can manifest themselves in cost,
schedule, and performance shortfalls. Accordingly, it is essential that
SSA establish a time frame for and expedite actions toward finalizing
risk-mitigation strategies for the AeDib project.
Regarding our recommendation that SSA validate all AeDib cost and
benefit estimates, the agency reiterated its intent to conduct studies
to help validate planning assumptions, and noted that work supporting
this action was under way.
Finally, SSA stated that it continually reassesses its communications,
and that while it can always improve, it is satisfied that information
is being shared, and that stakeholder voices are being heard. Further,
SSA stated that the stakeholders' differing opinions about how to
design the AeDib system do not reflect an issue of communication, but
rather, the reality of bringing change into a very complex and diverse
environment. Our report noted that SSA had taken additional actions to
work collaboratively with stakeholders. Further, we recognize that
changing conditions can elicit varying opinions regarding the actions
being taken on a project. Nonetheless, stakeholders continue to voice
serious concerns regarding the manner in which SSA is communicating
with them about AeDib, necessitating that the agency remain diligent in
pursuing a mutually agreed-upon understanding with stakeholders on
systems development and implementation issues. SSA's success in fully
implementing AeDib depends heavily on resolving all outstanding issues
and concerns that could affect the ultimate use and outcome of the
intended electronic capability. By implementing a communications plan
that clearly and comprehensively conveys its approach for effectively
addressing disability stakeholders' and users' concerns and ensuring
their full involvement in the AeDib initiative, SSA could strengthen
the likelihood that it will achieve a successful project outcome that
effectively meets the needs of all critical players in the disability
claims process.
We are sending copes of this report to the Commissioner of Social
Security, and to the Director, Office of Management and Budget. Copies
will also be available at no charge on our Web site at [Hyperlink,
http://www.gao.gov.].
Should you have any question on matters contained in this report,
please contact me at (202) 512-6240, or Valerie Melvin, Assistant
Director, at (202) 512-6304. We can also be reached by e-mail at
[Hyperlink, koontzl@gao.gov] and [Hyperlink, melvinv@gao.gov],
respectively. Other key contributors to this report were Michael A.
Alexander, Tonia B. Brown, Harold J. Brumm, Mary J. Dorsey, and
Michael P. Fruitman.
Sincerely yours,
Signed by:
Linda D. Koontz:
Director, Information Management Issues:
[End of section]
Appendixes:
Appendix I: GAO's January 30, 2004, Briefing:
Update on the Social Security Administration's Accelerated Electronic
Disability Strategy:
Briefing for the Staff of the Subcommittee on Social Security: House
Ways and Means Committee:
January 30, 2004:
Purpose and Outline:
Purpose:
To provide an updated assessment of SSA's accelerated electronic
disability (AeDib) claims processing initiative:
Outline:
* Introduction:
* Objectives:
* Scope and Methodology:
* Results in Brief:
* Background:
* Objective 1: Strategy:
* Objective 2: Software Development and Risk Management:
* Objective 3: Cost-Benefit Analysis:
* Objective 4: Stakeholder/User Involvement:
* Conclusions:
* Recommendations:
* Agency Comments and Our Evaluation:
Introduction:
Over the years, SSA has faced increasing challenges in ensuring
acceptable levels of service to the millions of disabled individuals
and beneficiaries. A major barrier to improving efficiency in the
disability determination process has been SSA's continuing reliance on
paper folders to process and adjudicate claims.
As a result, SSA is currently pursuing a major systems-related
initiative to enhance its disability claims processing capability. A
key goal of this initiative, known as the Accelerated Electronic
Disability (AeDib) System, is to develop by late January 2004, a
disability claims process with the capability to electronically view
and share claims information among all processing components.
GAO previously reported [NOTE 1] on SSA's progress in developing the
AeDib initiative, noting that its strategy involved risks that could
jeopardize the agency's successful transition to an electronic
disability process.
Objectives:
As requested by the Chairman of the subcommittee, our objectives were
to:
* assess SSA's progress toward and strategy for achieving AeDib,
* determine the adequacy of measures SSA is taking with AeDib to avoid
the kinds of software development problems that it encountered with its
prior disability system initiative and to identify and mitigate risks
to a successful development effort,
* assess the adequacy of SSA's analysis of AeDib costs and benefits,
and:
* assess SSA's consultation with and support from key stakeholders.
Scope and Methodology:
Analyzed project management and technical documentation describing
SSA's plans, strategies, and progress related to developing and
implementing AeDib:
Evaluated the extent to which SSA has implemented measures-including
institutional software capability mechanisms-to avoid the types of
software development problems identified in prior reviews and
established strategies for mitigating program and project risks:
Assessed the adequacy of SSA's analyses of AeDib costs and benefits,
including evaluating them against federal guidance and industry best
practices and analyzing the impact of critical pilot evaluation data on
costs and benefits:
Assessed SSA's plans and actions to effectively consult with and
identify and resolve issues raised by key disability stakeholders
(i.e., state Disability Determination Services [DDS] offices, medical
providers, and SSA field offices):
Conducted site visits at the North Carolina DDS office and SSA field
offices in Raleigh and Durham to observe pilot tests supporting the
electronic folder capability, and teleconferences with officials in the
California and Illinois DDSs and the SSA field office in Hillside,
Illinois to discuss pilot test progress and results:
Interviewed officials in selected DDS offices that had been included in
our prior review[NOTE 2] (Delaware, Iowa, Mississippi, Nebraska, New
York, Virginia, and Wisconsin) to update our assessments of SSA's
involvement of stakeholders in the AeDib initiative:
Interviewed headquarters officials in SSA's Offices of Disability
Programs, Operations, Systems, Hearings and Appeals, and the Inspector
General to obtain their perspectives on the AeDib project:
Conducted our review in accordance with generally accepted government
auditing standards, from August 2003 through January 2004:
Results in Brief: Objective 1:
SSA's Progress and Strategy for Achieving AeDib:
AeDib represents an important step toward more efficient delivery of
disability payments and SSA is in various stages of completing key
components of the initiative. SSA's strategy, however, involves risks
that could threaten project success. Specifically, SSA is relying on
limited pilot tests to help guide business and technical decisions and
ensure that technology supporting the electronic disability folder will
work as intended. Further, SSA is beginning its national rollout
without ensuring that all critical problems identified in its pilot
testing have been resolved and without conducting end-to-end testing to
evaluate the functionality and performance of all electronic disability
system components collectively. SSA maintained that its pilot tests of
the electronic disability system components would be sufficient to
evaluate the system. However, in the absence of end-to-end testing, SSA
lacks assurance that the interrelated electronic disability system
components will work together successfully.
Results in Brief: Objective 2; Previous Problems and Mitigating Risks:
SSA has established processes and procedures to guide its software
development activities, including using plans to manage projects and
tracking and overseeing initiatives to measure progress. However, SSA
could not provide evidence that it is consistently applying these
procedures to the AeDib initiative. For example, our review found no
evidence of approved software validation plans and system
certifications to show that users' requirements were satisfied. In
addition, while SSA had identified AeDib system and security risks, it
had not finalized mitigation strategies needed to help ensure a
successful project outcome. Without concurrence on software validation
and system certifications, or risk mitigation strategies, SSA lacks
assurance that its system will be acceptable to end users and it will
not be positioned to effectively prevent circumstances that could
impede project success.
Results in Brief: Objective 3 Adequacy of AeDib Cost-Benefit Analysis:
SSA has not validated its cost-benefit analysis to ensure the
reasonableness of estimated AeDib costs and benefits. SSA stated that
it would use pilot test results to validate its cost-benefit estimates;
however, its officials have not indicated when this will be done.
Without a validated cost-benefit analysis, SSA lacks assurance that its
AeDib cost estimates are reliable and that anticipated benefits will be
realized.
Results in Brief: Objective 4 Consultation With/Support from
Stakeholders:
SSA stated that it has increased its communications with AeDib
stakeholders/users and has instituted some mechanisms to facilitate its
communications. However, state DDSs have varying views regarding their
involvement. Specifically, while state disability offices involved in
AeDib pilots agreed that communications have improved, a national
organization representing these offices continues to have concern about
SSA's approach. In addition, although the Commissioner has stated that
SSA is consulting with stakeholders and the medical community, SSA has
not articulated a comprehensive plan for ensuring that all of their
concerns are addressed:
Results in Brief: Objective 4; Recommendations for Improvement:
Given the importance of the AeDib initiative to SSA's future service
delivery capability, it is essential that the agency satisfy itself
that the electronic disability system will perform as intended with
minimal risk before it is deployed nationwide. Therefore, to help
improve the success of this initiative, we are making recommendations
to the Commissioner of Social Security on (1) resolving critical
problems identified and conducting end-to-end testing, (2) ensuring
user concurrence on software validation and systems certifications, (3)
finalizing AeDib risk mitigation strategies, (4) validating AeDib costs
and benefits, and (5) improving agency communications with stakeholders
and users.
SSA officials reviewed a draft of our briefing slides, and their
comments have been incorporated where appropriate. SSA took issue with
many of our findings. For example, it disagreed with the need for end-
to-end testing, and also disagreed with our finding that it had not
followed all agency processes and procedures for software development.
We discuss these comments and our response to them in the Agency
Comments and Our Evaluation section.
Background:
SSA's Disability Program:
The Disability Insurance and Supplemental Security Income programs are
the nation's largest providers of federal income assistance to disabled
individuals, with SSA making payments of approximately $86 billion to
about 10 million beneficiaries in 2002.
SSA's process of approving or denying disability benefits is complex
and involves multiple partners at both the state and federal levels in
determining a claimant's eligibility.
* SSA's 1,300 field offices and its Office of Hearings and Appeals,
along with 54 state DDSs, are the primary players in processing
disability claims.
* Physicians and other members of the medical community support
disability determinations by providing the medical evidence needed to
evaluate disability claims.
To address concerns regarding the disability program's efficiency, in
1992 SSA initiated a redesign of the disability claims process,
emphasizing the use of automation to achieve an electronic (paperless)
processing capability. However, SSA encountered problems with this
prior initiative, the Reengineered Disability System (RDS), and in 1999
suspended the project after approximately 7 years and about $71 million
reportedly spent on the initiative [NOTE 3].
SSA's Disability Program:
In August 2000 SSA renewed its plan to achieve paperless disability
decision making through the use of an electronic disability folder and
automated case processing systems for disability claims adjudication/
review by the end of 2005. In the spring of 2002, the Commissioner of
Social Security accelerated the strategy to more quickly automate the
disability claims process. Under the accelerated strategy-called AeDib-
SSA planned to begin implementing its electronic disability system by
January 2004.
Projects Supporting AeDib:
SSA is undertaking five key projects to support the AeDib strategy:
1. an Electronic Disability Collect System, giving SSA field offices
the capability to electronically capture and store in an electronic
folder information about a claimant's disability;
2. a Document Management Architecture (data repository and scanning and
imaging capabilities) to allow claimant information and medical
evidence to be captured, stored, indexed, and shared electronically
among disability processing components;
3. Internet applications to enable SSA to obtain disability claims and
medical information from claimants via the Internet;
4. a DDS systems migration and electronic folder interface to enhance
existing case processing systems and enable state DDS offices to
operate on a common platform and share information in the electronic
folder; and:
5. a Case Processing and Management System for the Office of Hearings
and Appeals that will interface with the electronic folder and enable
staff to track, manage, and complete case-related tasks electronically.
Projects Supporting AeDib:
The AeDib strategy focuses on developing the capability for claimant
information and large volumes of medical images, files, and other
documents that are currently maintained in paper folders to be stored
in electronic folders, and then accessed, viewed, and shared by the
disability processing offices.
According to SSA, the Electronic Disability Collect System and the
Document Management Architecture are the two fundamental elements
needed to achieve the electronic disability folder.
SSA is using an incremental, phased approach to implementing AeDib.
The agency planned to rely on pilot tests and evaluations to help guide
business and technical decisions about the electronic disability
folder, including critical decisions regarding the Document Management
Architecture.
SSA had planned to complete its pilot tests of the Document Management
Architecture in December 2003, and begin its national implementation of
the electronic disability folder in late January 2004, with scanning
and imaging capabilities and interface software being rolled out over
18 months.
SSA's Approach:
AeDib is a major capital investment project, and its February 2003
cost-benefit analysis estimates 10-year life-cycle costs of about $900
million and benefits of approximately $1.3 billion.
SSA reported actual AeDib costs of approximately $91 million for fiscal
year 2000 through fiscal year 2003.
SSA identified two AeDib funding sources for IT hardware, software, and
services:
* information technology systems/automation investment fund, and:
* limitation on administrative expenses budget.
July 2003 Findings:
In July 2003 testimony [NOTE 4] we noted that SSA:
* had performed important tasks for the initial electronic capability,
but had substantial work to accomplish in order to begin implementing
the electronic folder by late January 2004;
* was not expected to complete pilot tests that were to provide
critical information about the electronic folder's performance until
late December 2003 and had not developed a strategy for conducting end-
to-end testing to demonstrate that the individual components would work
together reliably;
* had not comprehensively assessed AeDib project risks or developed
risk mitigation strategies; and:
* had taken some steps to involve key stakeholders in the system's
development, but had not resolved all outstanding concerns that could
affect the use and ultimate success of the intended electronic
capability.
September 2003 Findings:
In September 2003 [NOTE 5] we reported that SSA:
may have underestimated the accelerated electronic disability system's
costs and overstated corresponding benefits in its AeDib cost-benefit
analysis, and:
had applied software development practices that could help the agency
avoid software development problems experienced with its prior
electronic disability initiative-the Reengineered Disability System.
Objective 1: Strategy; Progress Update:
Since July, SSA has continued its work toward completing the AeDib
initiative, and is in various stages of development.
* SSA has implemented all planned releases of the Electronic Disability
Collect System (as of January 20, 2004).
* SSA has completed and placed into production three Internet
applications to aid claimants in filing for disability benefits and
services online.
* SSA has enhanced the DDS legacy systems by installing hardware
upgrades and moving systems to a common platform; it has also migrated
legacy systems claims processing software in all but 3 of the 54 state
DDS offices. However, it has yet to complete the legacy system
electronic folder interface software that will enable existing SSA and
DDS systems to interface with the electronic folder. SSA reported that
it plans to complete its implementation of the legacy system electronic
folder interface software by October 2004.
* The Case Processing Management System is being pilot tested in a
standalone environment at five Office of Hearings and Appeals sites.
Objective 1: Strategy; Testing:
Consistent with generally accepted best practices, SSA should ensure
that the system it delivers successfully meets key business and
technical requirements for reliably exchanging data among disability
processing components and is protected from errors and vulnerabilities
that can disrupt service.
Accomplishing this requires SSA to conduct complete and thorough
testing to provide reasonable assurances that systems perform as
intended. This includes tests and evaluations of pilot projects to
obtain data on the system's functional performance, and end-to-end
tests to ensure that the interrelated systems will operate together
effectively.
SSA planned to rely on the pilot tests and evaluations to guide
critical decisions about its document management technology and verify
that this system component works in the actual user environment. For
example, SSA stated that Document Management Architecture pilots would
be used to test electronic folder interface and DDS site configurations
for the electronic disability system's national implementation. SSA and
DDS offices stated that they intended to expand the size and scope of
the pilot tests to analyze the system's performance.
SSA had planned to complete these pilot tests in December 2003.
Since last July, SSA has been pilot testing the Document Management
Architecture in three locations-North Carolina, Illinois, and
California. SSA has been incrementally providing increased electronic
folder functionality.
These pilot tests have demonstrated that the DDSs can view medical
evidence captured in the Document Management Architecture; however, the
pilots also identified numerous technical/operational problems, which
SSA is in the process of resolving:
* DDS pilot systems experienced software glitches (i.e., screen lock-
ups, inability to save highlighted text, and inability to enhance
images) after new releases.
* Scanning quality for some documents, such as graphical images, was
poor.
Response times for retrieving medical evidence stored in the data
repository was slow (California/Midas).
* 2-D barcode resolution diminished with multiple faxing and copying,
preventing effective identification of the case file and the assigned
DDS.
* Existing 19-inch monitors proved too small for DDS claims examiners
to simultaneously view two different documents (e.g., forms and medical
evidence) on split screens.
Although the pilot tests are proceeding, they are limited in scope,
involving a relatively small number of examiners and disability cases:
North Carolina (pilot started July 29, 2003):
* 21 of 254 examiners participating (8 percent):
* Has closed 561 of 1,246 cases received for the pilot:
* Closes about 147,000 cases annually:
Illinois (pilot started Sept. 8, 2003):
* 15 of 250 examiners participating (6 percent):
* Has closed 282 of 1,165 cases received for the pilot:
* Closes about 164,000 disability cases annually:
California (pilot started Oct. 15, 2003):
* 2 of 620 examiners participating (0.3 percent):
* Has closed 47 of 160 cases received for the pilot:
* Closes about 400,000 disability cases annually:
Although SSA had planned to expand the pilot tests to additional users,
the technical/operational problems encountered during the pilots (e.g.,
software glitches experienced after new releases) have limited the
expansion, and consequently, the opportunity to analyze system
performance under increasingly heavy workloads (e.g., higher
transaction rates.):
The pilot tests, originally scheduled to be completed in December 2003,
are also behind schedule. Further, it is not clear when SSA plans to
complete the pilot tests. On January 13, SSA's Deputy Commissioner for
Systems stated that, beginning on January 20, SSA planned to release to
the three pilot states, the interface software and enhancements that
would provide disability users the capability to electronically process
an entire disability case.
The Deputy Commissioner added that, barring significant problems, SSA
planned to begin the national rollout of this functionality in
Mississippi on January 26, 2004. Thus, SSA would begin its national
rollout before fully evaluating pilot test results and ensuring that
all critical problems are resolved.
Without fully evaluating pilot test results and ensuring the resolution
of all critical problems before national rollout of the electronic
disability system, SSA not only risks being unable to ensure that the
electronic folder will perform as intended, but also risks being unable
to rely on the new system to sustain current disability workloads.
Given AeDib's technological complexity and multiple interrelated
components, coupled with limited pilot testing of the Document
Management Architecture and the numerous problems encountered during
that testing, end-to-end testing remains essential to helping SSA
reduce system risks prior to national roll-out.
Our prior work has noted the need for such testing to ensure that
interrelated systems that collectively support a core business area or
function will work as intended in a true operational environment. [NOTE
6] End-to-end testing evaluates both the functionality and performance
of all systems components, enhancing an organization's ability to trust
the system's reliability.
In addition to its limited pilot testing of the Document Management
Architecture, performance and other problems that SSA has encountered
while developing and implementing other critical AeDib components
further underscore the need for end-to-end testing. For example:
* since implementing the Electronic Disability Collect System, SSA has
encountered software performance issues, such as slow response times
when moving from screen to screen, which have contributed to longer
waits in reception areas, scheduling fewer claimant appointments, and
backlogs in post-entitlement work; and:
* edits for claimant information provided on Internet applications
generate additional edits after the data are input to the Electronic
Disability Collect System. Having to respond to the additional edits
has contributed to increased SSA field office claims representatives'
workloads.
Resolving all critical problems identified in its testing of the AeDib
components and then assessing how these components function in an
integrated environment before proceeding to national roll-out are
essential to gauging the system's overall successful operation.
Nonetheless, SSA does not plan to conduct end-to-end testing of the
electronic disability system prior to national implementation. SSA has
maintained that its pilot tests of each component of the electronic
disability system would be sufficient to evaluate the system's
functionality and performance.
Without resolution of all critical problems and end-to-end testing, SSA
cannot make fully informed decisions about the system it delivers,
risks having a system that will not work effectively, and could incur
future unanticipated costs for systems adjustments or corrections.
Objective 2: Software Development/Risk Management Previous Problems:
Our prior reports [NOTE 7] have noted that effective software
development processes and procedures are essential to producing high-
quality software products and ensuring that the software works as
intended.
In response to problems encountered during its prior effort with the
Reengineered Disability System, SSA established key software process
improvement procedures to guide its software development activities.
These procedures include:
* using plans to manage projects,
* tracking and overseeing initiatives to measure progress,
* performing quality assurance reviews to ensure compliance with
policies and procedures, and:
* conducting software validation and systems certifications to ensure
the effectiveness of and users' satisfaction with the software.
Implementation of these processes should help the agency ensure that
the system being developed is ready for production and will meet users'
requirements.
SSA is adhering to some of its software development processes and
procedures for the AeDib initiative. For example, it is using plans to
monitor software development activities and is performing quality
assurance reviews. However, the agency has not consistently implemented
certain key procedures. For example, we found no evidence that:
* users approved software validation plans to show that they agreed
with what was being produced, and:
* all systems currently in production were certified for release, to
indicate that the systems were developed in accordance with specified
requirements.
Without software validation and systems certification, SSA lacks
assurance that the system is ready for production and will be
acceptable to its end users.
Objective 2: Software Development/Risk Management Mitigating Risks:
Best practices and federal guidance advocate risk management to
identify facts and circumstances that decrease the probability of
failing to meet project commitments before they manifest themselves as
cost, schedule, and performance shortfalls.
SSA has developed a risk management plan to guide the identification
and mitigation of AeDib program and project risks. The plan requires
SSA to ensure that the necessary risk assessments and mitigation
strategies are developed and implemented to reduce risks and achieve
schedule and performance goals.
SSA has identified system and security risks for the five AeDib
projects. For example, it noted:
* the need for Document Management Architecture interfaces with other
AeDib components to be properly defined, tested, integrated, and
installed to avoid jeopardizing AeDib success, and:
* the need for completion and approval of systems security plans for
all five projects.
However, SSA has not yet finalized the mitigation strategies needed to
help prevent circumstances that could impede successful project
outcomes. SSA had previously stated that its mitigation strategies
would be completed by December 15, 2003. However, according to agency
officials, the draft strategies are currently under review by the
Office of the Chief Information Officer and a date for finalizing them
has not been established. Until the strategies are finalized, SSA will
not be in a position to cost-effectively plan for and prevent
circumstances that could impede project success.
Objective 3: Cost-Benefit Analysis Adequacy:
Office of Management and Budget guidance [NOTE 8] states that the
purpose of a cost-benefit analysis is to promote efficient resource
allocation through well-informed decision making when initiating,
renewing, or expanding programs or projects that would result in a
series of measurable benefits or costs extending for 3 or more years.
SSA's guidance states that a cost-benefit analysis is to be conducted
when there is a major capital investment, potential major human
resources impact, and the investment is in competition with other
projects for funding and resources.
We previously reported that SSA's estimates of costs and benefits were
based on critical assumptions that were not substantiated:
* Critical infrastructure costs, such as outsourced scanning and
imaging (keying and indexing), telecommunications, and disaster
recovery, were not included in the analysis.
* The estimated electronic receipt of 30 percent of medical evidence by
2004 was not based on sound data or tested in pilots.
Objective 3: Cost-Benefit Analysis Adequacy:
Validating the estimates is necessary to review assumptions that were
made, and to continuously update and ensure that costs and benefits
reflect the current situation and are accurate and realistic.
Our review determined that SSA's analysis had not been validated to
ensure the reasonableness of estimated AeDib costs and benefits. SSA
agreed with our assessment and stated that it planned to use the
results of pilot tests to validate cost and performance data included
in the analysis. However, agency officials could not state when the
validation of costs and benefits would be performed.
Without a validated cost-benefit analysis, SSA lacks assurance that its
AeDib cost estimates are reliable and that anticipated benefits will be
realized.
Objective 4: Stakeholder/User Involvement Consultation and Support:
As partners in the disability determination process, stakeholders can
offer valuable insight regarding existing work processes and
information technology needs, and their involvement in the systems
development initiative is essential for ensuring AeDib's acceptance and
use.
Our work on cross-organizational collaborative projects identified
effective communication and outreach as a key practice that contributes
to success. [NOTE 9] The research indicated that to achieve this, a
specific outreach plan specifying tasks and mechanisms may be needed.
In July we reported on the need for SSA to resolve stakeholder concerns
to ensure AeDib's acceptance and use, and to take additional steps to
consult with the medical community. [NOTE 10] SSA acknowledged the
importance of ensuring sound relations with stakeholders and the need
to take additional actions toward addressing stakeholders' concerns.
Objective 4: Stakeholder/User Involvement Consultation and Support:
SSA stated that it has increased its communications about AeDib with
state DDSs and medical providers, and the agency has established some
specific communication mechanisms (e.g., ad hoc meetings, Internet
television broadcasts, and an AeDib intranet site).
State DDSs, however, have varying views and concerns about their
involvement with the AeDib initiative.
DDSs associated with the Document Management Architecture pilot (North
Carolina, Illinois, and California) agreed that SSA has increased
communications and is working more closely with them.
Four states that are not participating in the pilot (Iowa, Nebraska,
New York, and Virginia) cited no improvement in SSA's overall
communications with them about the development and implementation of
the electronic disability system.
The National Council of Disability Determination Directors, which
represents the DDSs, continues to have concerns about SSA's
communications approach, stating that the AeDib steering committee is
only being used as a vehicle for reporting decisions already made, and
that improved communication with DDSs is still needed.
Objective 4: Stakeholder/User Involvement Consultation and Support:
While the Commissioner previously stated that SSA was consulting with
stakeholders and the medical community, agency officials could not
articulate a comprehensive plan for effectively involving all
stakeholders and consulting with the medical community.
Without a clear and comprehensive plan for communicating with its
stakeholders and the medical community, SSA risks not obtaining vital
end-user buy-in and acceptance of the electronic disability system and
the medical community's support in providing electronic medical
evidence that is crucial to achieving anticipated benefits.
Conclusions:
SSA is beginning its implementation of an electronic disability system
without fully evaluating pilot test results or ensuring that all
critical problems have been addressed, and with no plans for end-to-end
testing to guide its system implementation. Thus, SSA lacks assurance
that the system will operate as intended in a national environment-
making the agency highly susceptible to failure in its current effort
to achieve an electronic disability claims processing capability.
While SSA has established software process improvement procedures,
critical user concurrence for all software validations and systems
certifications are not evident. Without such concurrence, SSA lacks
assurance that it is building a system that will meet users'
requirements. Further, although SSA had planned to complete strategies
for mitigating program and project risks by last December, it has yet
to finalize them.
SSA is beginning its national rollout of the electronic disability
system without a validated cost-benefit analysis. Lacking a reliable
analysis based on validated data, the agency is assuming the risk that
its anticipated costs may be inconsistent with planned spending for the
initiative and that anticipated measurable benefits may not be
realized.
Conclusions:
Finally, SSA has taken some steps to improve its communications with
key AeDib stakeholders and users; however, the agency has not yet
articulated a comprehensive plan for ensuring that all stakeholders and
users are fully involved in the initiative and that their concerns are
adequately addressed. State DDSs currently have mixed views regarding
their involvement, with DDS representatives continuing to see a need
for improved communications with SSA about critical AeDib decisions.
Recommendations:
Given the technological complexity, scope, and size of the AeDib
initiative, it is essential that SSA take the necessary steps to avoid
the substantial risks of moving into production with a system that does
not perform as intended. Accordingly, we recommend that the
Commissioner of Social Security, before proceeding with the AeDib
national rollout,
* ensure that all critical problems identified in pilot testing of the
electronic disability system components are resolved and that end-to-
end testing of the interrelated systems is performed;
* ensure that users have approved of software developed and that
systems have been certified for production;
* establish a revised time frame for and expedite actions toward
finalizing AeDib risk mitigation strategies,
* validate all AeDib cost and benefit estimates, and:
* implement a communications plan that clearly and comprehensively
conveys SSA's approach for effectively addressing disability
stakeholders' and users' concerns and ensuring their full involvement
in the AeDib initiative.
Agency Comments and Our Evaluation:
SSA's Deputy Commissioner for Finance, Assessment and Management
provided an e-mail message containing the agency's comments on a draft
of the briefing slides. In its comments, SSA took issue with many of
our findings.
SSA disagreed with the need for end-to-end testing, stating that such
testing would have been too slow for its accelerated strategy. In
addition, it stated that examples discussed in our slides did not
support our position, noting, for example, that the Electronic
Disability Collect System performance issues highlighted in our slides
had not appeared until the software was in full production across the
country.
We maintain that end-to-end testing is essential for AeDib. The
acceleration of such a technologically complex initiative, with its
multiple interrelated components, elevates the need for end-to-end
testing to help SSA reduce system risks, many of which were identified
during pilot tests of each component. The fact that performance issues
did not appear until the Electronic Disability Collect System software
was in production, and that high edit rates were discovered when data
from Internet applications were input into that system, is further
evidence of the need for end-to-end testing to provide SSA with the
opportunity to evaluate both the functionality and performance of all
systems components prior to national rollout.
Regarding examples of technical problems that we noted for the Document
Management Architecture pilots, SSA stated that (1) it had completed a
technical evaluation of the Document Management Architecture in three
pilot states and (2) that four of five examples identified in our
slides were resolved before SSA made a final decision to roll out to
Mississippi. In addition, SSA stated that our finding regarding the 19-
inch monitor size prejudged a conclusion that the screens were too
small.
While SSA stated that it had completed technical evaluations of the
Document Management Architecture pilot efforts in the three pilot
offices, it did not provide evidence of its complete evaluations during
our review. Specifically, SSA provided evaluation results covering
tests performed in the North Carolina and Illinois DDSs only during the
July through September 2003 time frame. Further, on January 20, SSA
provided a software release supporting the electronic folder capability
to the pilot offices, leaving little time to ensure that the software
was fully tested and evaluated and that all critical problems were
resolved prior to beginning its national rollout.
In addition, while we acknowledge that SSA may have resolved four of
the five problems noted in our slides, the examples that we cited
constituted only a subset of the numerous problems that were noted in
the Document Management Architecture evaluations provided for our
review. Our analysis of the North Carolina and Illinois pilot
evaluations and SSA's problem tracking system report, and our
discussions with officials of all three state DDSs participating in the
pilots, revealed that, as certain software performance issues were
resolved, other problems appeared, and that DDS staff participating in
the pilots were overwhelmed as workloads and backlogs increased.
Finally, it was not our judgment or conclusion that SSA's monitor size
was too small. Rather, DDS claims examiners informed us that the 19-
inch monitors were too small for viewing two documents on split screen.
For example, during our visit to the North Carolina pilot site, claims
examiners participating in tests of the Document Management
Architecture and Electronic Disability Collect System stated that the
small size of the monitors had affected their ability to effectively
adjudicate a claim. In discussing this matter with SSA's Deputy
Associate Commissioner for Systems and Associate Commissioner for
Disability Programs, both stated that the agency was aware of DDS
staffs' concerns about the size of the monitors and that SSA planned to
send an ergonomics team to those sites to investigate this and other
issues affecting staff using the system. At the time of review, the
study had not been completed.
In discussing the October 2004 completion date of its legacy systems
migration, SSA stated that, while technically correct, our discussion
did not acknowledge that DDS legacy software migration had been
completed in all but 3 of the 54 state DDS offices and that those three
offices would be completed in February 2004.
We have modified our slides to reflect that SSA has completed legacy
software migration in all but three DDS offices.
SSA also stated that the DDSs have or will shortly have the Electronic
Disability Collect System electronic folder interface (EFI). It noted
that its more expansive EFI software is currently running in 11 states,
and expansion of that will occur in concert with the Document
Management Architecture rollout.
SSA's recent documentation stated that it has made the Electronic
Disability Collect System electronic folder interface software
available to 15 DDS states, including the three Document Management
Architecture pilot offices. However, SSA has yet to complete the
necessary DDS legacy systems software-including its critical Document
Management Architecture interface software-to enable these components
to process disability claims. Until this is accomplished, existing
disability processing components will not have the capability to
interface with or share information captured in the full electronic
folder.
SSA stated that it has followed processes and procedures for software
development and disagreed with our findings that it did not have
approved software validation plans and had not properly certified all
software releases.
While we agree that SSA has established key software process
improvement procedures to guide its software development activities, we
found that SSA was not consistently adhering to its established
procedures. At the time of our review, SSA could not provide the
necessary documentation to demonstrate that users had approved of the
AeDib software's being produced. SSA's Inspector General had reported
the same findings in July 2003, and in discussions with us last
December, reiterated this continuing concern.
SSA stated that its security risk assessment documentation is no longer
in draft and can be made available to us for review.
SSA's actions toward finalizing its security level risk assessments for
AeDib are commendable, and we look forward to reviewing this
documentation. However, our recent reviews of AeDib found that the
agency had not finalized mitigation strategies for its program-level or
security-level risk assessments. SSA officials informed us that the
draft mitigation strategies were under review by the Office of the
Chief Information Officer and that no date for finalizing them had been
established. We continue to believe that these strategies are needed to
help prevent circumstances that could impede successful project
outcomes.
SSA believes that its 30-percent electronic medical evidence is a
viable goal, and continued to disagree with our view that its cost-
benefit analysis did not consider proper infrastructure costs. However,
SSA did agree that validation of its cost and benefit assumptions is
important and stated that it is actively planning that work.
We are encouraged that SSA is planning to validate its cost and benefit
assumptions. In noting SSA's 30-percent electronic medical evidence
goal, our intent was to highlight the importance of SSA's validating
its cost-benefit analysis to ensure the reasonableness of its estimated
costs and benefits. The assumption that we highlighted was one of many
included in SSA's cost-benefit analysis that had not been validated.
Further, it continues to be our position that all relevant costs need
to be reflected in SSA's cost-benefit analysis. Office of Management
and Budget guidance states that cost-benefit analyses should include
comprehensive estimates of all direct and indirect costs associated
with a project. Without a validated cost-benefit analysis, and the
inclusion of missing critical cost elements, SSA lacks assurance that
its AeDib cost estimates are reliable and that anticipated benefits
will be realized.
Finally, regarding our findings on communication and consultation with
disability stakeholders, SSA stated that AeDib information is pervasive
and available to all stakeholders/users, and that the agency continues
to work collaboratively with them. It added that outreach to external
groups will ramp up in relation to national rollout.
We acknowledge that SSA has made additional efforts to work
collaboratively with its stakeholders regarding AeDib. However, we
found during our analyses of documentation and in discussions with
disability stakeholders that there are still varying views and concerns
about SSA's approach for implementing AeDib. Due to the complexity of
the AeDib initiative, and the many concerns that continue to be voiced
about SSA's approach, we continue to believe that having a clear and
comprehensive plan before national rollout is vital to ensuring end-
user buy-in and acceptance of the electronic disability system, and to
realizing anticipated benefits.
NOTES:
[1] U.S. General Accounting Office, Electronic Disability Claims
Processing: Social Security Administration's Accelerated Strategy
Faces Significant Risks, GAO-03-984T (Washington, D.C.: July 24, 2003)
and U.S. General Accounting Office, Social Security Administration:
Subcommittee Questions Concerning Efforts to Automate the Disability
Claims Process, GAO-03-1113R (Washington, D.C.: Sept. 5, 2003).
[2] GAO-03-984T.
[3] U.S. General Accounting Office, Social Security Administration:
Technical and Performance Challenges Threaten the Progress of
Modernization, GAO/AIMD-98-136 (Washington, D.C.: June 19, 1998).
[4] GAO-03-984T.
[5] GAO-03-1113R.
[6] U.S. General Accounting Office, Year2000 Computing Crisis: A
Testing Guide, GAO/AIMD-10.1.21 (Washington, D.C.: November 1998).
[7] U.S. General Accounting Office, Customs Service Modernization:
Serious Management and Technical Weaknesses Must Be Corrected, GAO/
AIMD-99-41 (Washington, D.C., Feb. 26,1999); Customs Service
Modernization: Ineffective Software Development Processes Increase
Customs System Development Risks, GAO/AIMD-99-35 (Washington, D.C.,
February 11, 1999); and Air Traffic Control: Immature Software
Acquisition Processes Increase FAA System Acquisition Risks, GAO-AIMD-
97-47 (Washington, D.C., March 21, 1997).
[8] Office of Management and Budget Circular A-94 (Revised Transmittal
Memo No. 64), October 29, 1992.
[9] U.S. General Accounting Office, Electronic Government. Potential
Exists for Enhancing Collaboration on Four Initiatives, GAO-04-6
(Washington, D.C.: Oct. 10, 2003).
[10] GAO-03-1113R.
[End of section]
Appendix II: Comments from the Social Security Administration:
SOCIAL SECURITY:
The Commissioner
March 15, 2004:
Ms. Linda D. Koontz:
Director, Information Management Issues
U.S. General Accounting Office:
441 G Street, NW Washington, D.C. 20548:
Dear Ms. Koontz:
Thank you for the opportunity to review and comment on the draft report
"Electronic Disability Claims Processing: SSA Needs to Address Risks
Associated with Its Accelerated Systems Development Strategy" (GAO-04-
466). Our comments on the report are enclosed.
If you have any questions, please have your staff contact Candace
Skurnik, Director, Audit Management and Liaison Staff at (410) 965-
4636.
Sincerely,
Signed by:
Jo Anne B. Barnhart:
Enclosures (2):
SOCIAL SECURITY ADMINISTRATION BALTIMORE MD 21235-0001:
COMMENTS ON THE GENERAL ACCOUNTING OFFICE (GAO) DRAFT REPORT
"ELECTRONIC DISABILITY CLAIMS PROCESSING: SSA NEEDS TO ADDRESS RISKS
ASSOCIATED WITH ITS ACCELERATED SYSTEMS DEVELOPMENT STRATEGY" (GAO-04-
466):
Thank you for sharing the draft report with us.
We note that this draft contains essentially the same slides you
provided us to comment on earlier this year. Given that, and since you
already presented this material to the sub-committee staff in January,
we are not going to re-state our disagreements with your findings. In
the interest of efficiency, we request that you include this note, plus
our prior comments from Mr. Sopper (copy attached) in your published
report.
Considering GAO's long audit presence on the eDib project, I know your
interest in our progress is keen and I am sure that you will be pleased
to hear that things continue to move forward. Our first full
functionality Disability Determination Service (DDS) site in
Mississippi recently expanded to 24 examiners using the new system.
South Carolina is scheduled to go live on March 22, and then Tennessee
on April 5. All told, we have used the system to receipt in over 5,700
cases in DDS sites, and have processed over 1,900 cases.
Although it is too early to draw conclusions about processing time,
comparing 2003 to 2002 overall disability processing time, we saw a 7-
day improvement. We believe 5 days of this are attributable to the
rollout of our Electronic Disability Collect System. In addition, early
indicators for North Carolina show that, over the last six months, as
they have piloted the electronic folder, they processed their claims 6
days faster than the national average. Further, we have over 30,000
documents in our electronic evidence repository, and the percent of
evidence that bypasses scanning and comes in electronically currently
stands at 38 percent in Mississippi.
Turn-key pilots of the Case Processing Management System (CPMS) in five
hearing offices are running smoothly, and we will begin a five month
national rollout of CPMS in April. We will also begin installing full
electronic folder capability in hearing offices this summer.
You have made five recommendations in this draft and, as with all GAO
views, we have given them careful consideration. However, as noted
below, none of your recommendations compel us to change direction.
1. Systems Testing: We continue to disagree with you on end-to-end
testing. If we had followed your recommendation, we estimate that it
would have delayed the project by at least three years-and of course,
that is three years of benefits that would have:
been missed. We do continue to aggressively troubleshoot the technical
bugs that are inevitable with any complex new IT system, and so far our
track-record has been strong.
2. System Development Process: Your recommendations that we ensure
users have approved new software, and that we certify systems for
production are moot. We have always done both of these things, and will
continue to do so. We believe your concern tracks back to the lack of
"vet" signatures on some of the hundreds of project documents we have
given you. If that is the case, and especially since this project is
all about movement to an electronic environment, we are puzzled that
you would be uncomfortable reviewing documents from our artifact files
that have been electronically signed.
3. Risk: Every large IT project presents a wide array of risks. We
believe the success of this project over the last 26 months attests to
our diligent and ongoing efforts to mitigate them.
4. Validating Planning Assumptions: As stated in our prior comments, we
will conduct special studies to help validate key eDib planning
assumptions. This work is underway. In the mean time, technical data on
systems performance, operational management information, and progress
reports from sites all support continued momentum on rollout.
5. Communication: Finally, we continually reassess how we are doing
with communication. While we can always improve, I am satisfied that
information is being shared, and that stakeholder voices are being
heard. There have been differing opinions about how we should design
this system, and we have given thorough consideration to a wide variety
of views, but we have had to make the difficult choices and move on.
This is not an issue of communication, but rather the reality of
bringing change into a very complex and diverse environment.
Because of the intelligence, creativity, hard work, and tremendous
dedication of literally thousands of SSA and DDS employees across the
country, our massive eDib project is going very well. After two years,
we are essentially on schedule, staying within our project budget, and
delivering the core functionality as originally conceived.
In the near term, eDib will help us to process claims and appeals
faster and more efficiently. Beyond that, eDib will become the enabling
platform we will use to implement more strategic improvements to the
disability determination process. We have never lost sight that the
real benefactors of this new system will be the American people who
seek government services at extremely vulnerable periods in their
lives. I am proud of what we have accomplished so far, and I have
confidence in our ability to fully implement the system in the next
year and half.
From: Sopper, Dale:
Sent: Wednesday, January 28, 2004 2:31 PM To: 'Koontzl@gao.gov':
Cc: 'MelvinV@gao.gov':
Subject: Comments on GAO draft briefing slides systems.doc:
Thank you for the opportunity to review the latest GAO draft talking
points on AeDib. We take issue with many of your findings.
We continue to disagree with you on the need for end-to-end testing.
This project is all about accelerating strategically important IT.
Simply put, your notion of testing would have been too slow. In
addition, the examples you cite don't support your broader position.
Our performance issues with EDCS didn't appear until the software was
in full production across the country. And the high edit rate for
Internet self-help claims was not a surprise because of deliberate
user-centered design choices to minimize online edits for first-time
filers.
You cite five examples of technical problems SSA has experienced with
the DDS pilots. It's important to note that 1) we did complete a
technical evaluation of DMA in the three pilot States; and 2) the first
four of your examples were resolved before we made a final decision to
begin roll-out of DMA in
Mississippi. Your fifth example on monitor size pre judges a conclusion
that 19" screens are too small. We do not agree that this conclusion is
warranted at this time.
* You state that SSA will not complete our DDS legacy software migration
till October, 2004. While technically true because of unique challenges
posed by two States which have independent systems, you should
acknowledge that all but 3 of the DDS's that use a standard legacy
system have already been migrated, and that those three will be
completed in February. In addition, all of the DDS's that use a
standard legacy system have (or will shortly have) EDCS EFI. Our more
expansive EFI software is currently running in 11 States, and expansion
of that will occur in concert with DMA roll-out.
We have followed agency processes and procedures for software
development and disagree with your qualifier that we are only following
some steps. The two examples you cite are not correct. The staff groups
that represent users in our process have approved the software
validation plans. In addition, all of our software releases have been
properly certified.
* Our security risk assessment documentation is no longer in draft and
can be made available to GAO for review.
We continue to believe that our 30 percent EME target is a viable goal.
And we continue to disagree with your view that our CBA did not
consider proper infrastructure costs. However, we do agree that an
evaluation of AeDib which extends beyond technical performance and can
help validate CBA assumptions is important and we are actively planning
that work.
* Finally, in regard to your findings on communication and consultation,
AeDib information is pervasive and available to all SSA and DDS
employees, including
employees who are not involved in current pilots. We continue to work
collaboratively with SSA and DDS stakeholders, and all opinions and
ideas are considered in decision-making. Outreach to external groups
will ramp up in relation to national roll-out.
In conclusion, we think the acceleration of this very complex and
strategically vital project has gone remarkably well. We fully expect
many more technical and management challenges to come up during roll-
out, but our significant accomplishments over the last 22 months have
positioned us to overcome them, and to successfully implement e-Dib.
[End of section]
FOOTNOTES
[1] U.S. General Accounting Office, Electronic Disability Claims
Processing: Social Security Administration's Accelerated Strategy
Faces Significant Risks, GAO-03-984T (Washington, D.C.: July 24, 2003)
and Social Security Administration: Subcommittee Questions Concerning
Efforts to Automate the Disability Claims Process, GAO-03-1113R
(Washington, D.C.: Sept. 5, 2003).
[2] A wet signature refers to an original signature written on a piece
of paper, as opposed to a fax copy or an agreement offered verbally or
electronically.
GAO's Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW,
Room LM Washington,
D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.
General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.
20548:
