Social Security Administration
Subcommittee Questions Concerning Efforts to Automate the Disability Claims Process Gao ID: GAO-03-1113R September 5, 2003This letter responds to a Congressional request on August 12, 2003 that we provide answers to questions relating to our July 24, 2003, testimony. In that testimony, we discussed the risks that the Social Security Administration (SSA) faces in its efforts to automate its disability claims process. The questions concerned money saved by implementing an electronic disability folder, SSA's development of a risk management plan, areas which SSA could improve, and difficulties in completing the project.
Our work to date on SSA's February 2003 cost-benefit analysis raises concerns that SSA may have underestimated its accelerated electronic disability (AeDib) system costs. Because SSA has not yet fully estimated these costs, we are unclear about their magnitude. A risk management plan provides guidance to project management teams and requires them to proactively identify facts and circumstances that could increase the probability of failing to meet project commitments, and take steps to prevent this from occurring. A comprehensive assessment of risks, which is completed according to the risk management plan, is the process of identifying risks with a high probability and cost of failure, and developing strategies for mitigating those risks. SSA has acknowledged our concerns relative to the areas of improvement we identified in our testimony, and has taken some action to address them. However, more work remains to fully address these issues. Software development is one of the riskiest areas of systems development. We have reported that SSA's software development efforts have been problematic and plagued with delays because SSA has not consistently followed sound practices in developing systems designed to automate its disability claims process; thus, it has experienced numerous software development problems over the past 11 years.
GAO-03-1113R, Social Security Administration: Subcommittee Questions Concerning Efforts to Automate the Disability Claims Process
This is the accessible text file for GAO report number GAO-03-1113R
entitled 'Social Security Administration: Subcommittee Questions
Concerning Efforts to Automate the Disability Claims Process' which was
released on September 05, 2003.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
September 5, 2003:
The Honorable E. Clay Shaw, Jr. Chairman, Subcommittee on Social
Security Committee on Ways and Means House of Representatives:
Subject: Social Security Administration: Subcommittee Questions
Concerning Efforts to Automate the Disability Claims Process:
Dear Mr. Chairman:
This letter responds to your August 12, 2003, request that we provide
answers to questions relating to our July 24, 2003, testimony.[Footnote
1] In that testimony, we discussed the risks that the Social Security
Administration (SSA) faces in its efforts to automate its disability
claims process. Your questions, along with our responses, follow.
1. The Social Security Administration (SSA) has indicated that the
agency could potentially save $1 billion, at an estimated cost of
approximately $900 million, by implementing an electronic disability
folder. In your opinion, does the $900 million cost of this project
appear to be a reasonable estimate? Is the $1 billion in savings a
reasonable estimate? Is there a possibility that the cost of the
project could balloon?
Our work to date on SSA's February 2003 cost-benefit analysis raises
concerns that SSA may have underestimated its accelerated electronic
disability (AeDib) system costs. For example, the cost-benefit analysis
did not fully consider the costs associated with certain critical
information technology infrastructure elements supporting the
nationwide rollout, such as scanning and imaging by the outsourced
vendor, telecommunications, disaster recovery, and on-site retention
and destruction of source documents, such as medical records. Because
SSA has not yet fully estimated these costs, we are unclear about their
magnitude.
We are also concerned that the corresponding benefits cited in SSA's
cost-benefit analysis may be overstated. Specifically, our review found
that certain assumptions used in the analysis could be too optimistic.
For example, SSA estimated benefits based on an assumption that state
Disability Determination Services (DDS) would receive 30 percent of all
medical evidence in electronic form by 2004. However, state DDS
officials with whom we spoke contend that their offices currently
receive about 11 percent or less of medical evidence electronically,
and disagree with the 30-percent assumption.
As to whether the costs of AeDib could balloon, the possibility of cost
increases exists for any project the size and magnitude of AeDib. As
mentioned previously, the existing estimates do not include certain
costs, such as some costs for outsourced scanning and imaging and the
cost of disaster recovery. Including these could add to SSA's overall
cost estimate. Our work analyzing the costs and benefits of AeDib is
ongoing; the final results will be included in our report to be issued
to you later this year.
2. According to your testimony, the SSA has developed a risk management
plan, but is still without a comprehensive assessment of risk that
could affect the electronic disability folder. Can you explain the
difference between the risk management plan and a comprehensive
assessment of risks? How does their risk management plan fall short of
what GAO recommends? Why is this plan so important?
A risk management plan provides guidance to project management teams
and requires them to proactively identify facts and circumstances that
could increase the probability of failing to meet project commitments,
and take steps to prevent this from occurring.[Footnote 2] A
comprehensive assessment of risks, which is completed according to the
risk management plan, is the process of identifying risks with a high
probability and cost of failure, and developing strategies for
mitigating those risks.[Footnote 3]
Based on our work to date, we do not have concerns about SSA's risk
management plan, which was developed in accordance with our own and
Office of Management and Budget (OMB) guidance. However, as mentioned
in our testimony, we were concerned that SSA had not completed a
subsequent comprehensive risk assessment. SSA officials agreed with the
need for a comprehensive risk assessment, and since our testimony, the
agency has provided us with detailed risk assessments for four of the
five AeDib projects--the Electronic Disability Collect System, the
Document Management Architecture, a DDS systems migration and
electronic folder interface, and Internet disability applications. SSA
has not yet provided us with a time frame for completing the remaining
risk assessment for its Hearings and Appeals case processing management
system.
A risk management plan is an essential tool used to guide the
development of a comprehensive assessment of risks and mitigation
strategies. A comprehensive risk assessment is equally critical. Both
tools can help SSA avoid potential problems before they manifest
themselves in cost, schedule, and performance shortfalls.
3. The GAO has identified a number of areas where the SSA could
improve. For instance, the agency needs to develop risk assessment
tools and could take additional measures to involve key stakeholders in
the systems development process. Is there any indication that the SSA
is acting on your recommendations?
SSA has acknowledged our concerns relative to the areas of improvement
we identified in our testimony, and has taken some action to address
them. However, more work remains to fully address these issues.
Specifically, we expressed the need for SSA to perform a comprehensive
risk assessment to identify project risks and establish mitigation
strategies for them. As noted in our response to question 2, SSA
recently provided us with detailed risk assessments for four of the
five AeDib projects, but has not yet provided a detailed assessment of
risks for its remaining project.
We also commented on the need for SSA to perform end-to-end testing
prior to implementation, to ensure that the system it is developing
will perform as intended. To date, SSA has not yet finalized its test
and evaluation strategy. Therefore, it is unclear whether SSA intends
to include end-to-end testing.
Finally, we commented on the need for SSA to resolve stakeholder
concerns to ensure program acceptance and use, and take additional
steps to consult with the medical community. SSA acknowledged the
importance of ensuring sound relations with stakeholders and the need
to take additional actions, where necessary, to ensure that all
stakeholder concerns have been adequately addressed. SSA also stated
that additional steps would be taken to keep stakeholders involved in
the initiative, and that plans were being made for a meeting with state
DDS representatives to discuss the AeDib initiative. However, SSA has
not yet provided us with a copy of its communications plan for dealing
with stakeholder issues, including its plans for consultation with the
medical community.
As part of our ongoing work, we will continue to monitor SSA's progress
in addressing these issues.
4. The SSA has worked 11 years now to implement an electronic
disability folder. The automation project started in 1992 as the
Modernized Disability System (later renamed the Reengineered Disability
System). After this project failed in 1999, the SSA immediately began
work on the accelerated electronic disability initiative. Why is this
project so difficult for the agency to complete? Has the SSA learned
from its mistakes along the way?
Software development is one of the riskiest areas of systems
development.[Footnote 4] We have reported that SSA's software
development efforts have been problematic and plagued with delays
because SSA has not consistently followed sound practices in developing
systems designed to automate its disability claims process; thus, it
has experienced numerous software development problems over the past 11
years. For example, in September 1996 we reported that software
development problems had delayed the scheduled implementation of the
Reengineered Disability System (RDS) by more than 2 years.[Footnote 5]
An assessment of the development activity revealed a number of factors
as having contributed to that delay, including (1) using programmers
with insufficient experience, (2) using software development tools that
did not perform effectively, and (3) establishing initial software
development schedules that were too optimistic. We reported again, in
June 1998, that SSA had encountered performance problems during its RDS
pilot tests.[Footnote 6] In response to these performance problems, SSA
delayed its plans for expanding the pilot to other offices, and
obtained a contractor to independently evaluate and recommend options
for proceeding with the initiative.
SSA's contractor reported that RDS software had defects that would
diminish the case-processing rate at DDS sites, and that SSA had not
been timely in addressing software defects. For example, 90 software
problems identified by SSA staff remained unresolved for more than 120
days. As a result, the contractor recommended that SSA discontinue the
RDS initiative and focus on an alternative solution involving the use
of an electronic folder to replace the paper-based case folder in the
disability determination process.
In another example, we reported in August 2001 on weaknesses regarding
SSA's adherence to key software development procedures for several
projects, including its electronic disability system.[Footnote 7] We
noted that SSA did not consistently adhere to its software development
procedures in the areas of requirements management, software project
planning, software quality assurance, and software configuration
management.
In our ongoing review, we have found that SSA has taken important steps
to mitigate past software development problems. It has generally
addressed its contractor's recommendations aimed at automating its
disability claims process, and has generally been applying key software
process improvement practices to its development of AeDib projects,
such as developing plans to manage the projects, tracking and
overseeing the initiatives to measure progress, performing quality
assurance reviews to determine that the project is complying with its
policies and procedures, and performing configuration management
activities. While continually applying these software development
practices is no guarantee of AeDib success, these practices
nevertheless should improve SSA's capability to develop high-quality
software in support of AeDib, thereby avoiding mistakes experienced in
the past.
Agency Comments and Our Evaluation:
The Commissioner of Social Security provided comments on a draft of
this correspondence, which are reproduced in full as the attachment. In
her comments, the Commissioner offered clarifications to our responses
to questions 1 through 3.
Regarding question 1, the Commissioner said that we were incorrect in
stating that SSA's cost-benefit analysis had not considered all costs
of certain critical IT infrastructure elements supporting the
nationwide AeDib rollout, and that, in fact, SSA had included the costs
of scanning and imaging by the outside vendor in its estimates.
We have revised our response to clarify that SSA's cost-benefit
analysis considered some, but not all, of the key cost elements that
could affect the initiative. SSA's February 2003 cost-benefit analysis
noted, for example, that the agency had not considered as part of its
scanning and imaging functions, the keying in of indexing information
(for case folder identification) by the outsourced scanning vendor,
although this is deemed critical to the implementation of SSA's
document management capability. Further, during our review, SSA
officials told us that certain costs associated with the scanning and
imaging functions were not expected to be identified until the agency
performed its pilot tests (now ongoing) for its document management
architecture.
The Commissioner added that other AeDib-related costs associated with
telecommunications, disaster recovery, and on-site retention and
destruction of source documents should more appropriately be accounted
for as part of the agency's ongoing operations, and therefore were
covered in the agency's regular infrastructure costs, rather than the
AeDib cost estimates. However, we disagree. OMB guidance states that
cost-benefit analyses should include comprehensive estimates of all
direct and indirect costs associated with a project.[Footnote 8] As
such, a sound cost-benefit analysis for AeDib will depend on SSA's
fully considering the project-related costs for these critical elements
supporting the development, operation, and maintenance of the
electronic disability system.
Further, the Commissioner expressed concern about the example that we
provided in noting that SSA's estimate of AeDib benefits could be
overstated. Based on our interviews with SSA and state DDS
representatives, our response highlighted the possibility that AeDib
benefits could be overstated because of differences in SSA's and the
DDSs' assumptions about the extent that medical evidence may be
received electronically. We are continuing to assess SSA's cost-benefit
analysis as part of our ongoing review.
In commenting on our response to question 2, the Commissioner stated
that SSA had expected to receive draft risk assessments by June 30,
2003, and either planned to or already had shared some of its risk
assessments with us. We have revised our response to reflect that SSA
recently provided us with detailed risk assessments for four of the
five AeDib projects.
Finally, regarding question 3, the Commissioner stated that SSA had
provided us with information concerning stakeholder issues and the
agency's plans for consulting with the medical community. She further
noted that outreach to the medical community was occurring. We are
encouraged that SSA is taking steps to ensure productive communications
with its key stakeholders and the medical community, and look forward
to reviewing documented evidence of the agency's actions. To date,
however, we have not received the additional information on SSA's plans
for addressing stakeholder issues or consulting with the medical
community, that the Commissioner mentions in her letter.
:
In responding to these questions, we relied on past work and our
ongoing review of SSA's efforts to automate its disability claims
process. We discussed our assessment of the cost-benefit analysis with
SSA's Office of Disability Programs, Office of Systems, and Office of
the Chief Information Officer, and with the supporting contractor and
key stakeholders. We reviewed and analyzed the most recent agency
documents associated with SSA's risk management efforts. We also
discussed with officials in the Office of Disability Programs SSA's
efforts to develop a strategy for resolving stakeholder concerns, as
well as a more aggressive approach for consultation with the medical
community. Finally, we reviewed and analyzed SSA's software process
improvement documentation, as well as past assessments of the agency's
failed attempts to automate its disability claims process to determine
lessons learned, and whether SSA is avoiding past software development
problems. We conducted our work in accordance with generally accepted
government auditing standards, during August 2003.
We are sending copies of this letter to the Honorable Jo Anne B.
Barnhart, Commissioner of Social Security, and other interested
parties. Copies will also be available at our Web site at [Hyperlink,
www.gao.gov.] w [Hyperlink, http://www.gao.gov] ww.gao.gov.
If you have any questions on matters discussed in this letter, please
contact me at (202) 512-6240 or Valerie Melvin, Assistant Director, at
(202) 512-6304. We can also be reached by E-mail at [Hyperlink,
koontzl@gao.gov] k [Hyperlink, koontzl@gao.gov] oontzl@gao.gov and
[Hyperlink, melvinv@gao.gov] m [Hyperlink, melvinv@gao.gov]
elvinv@gao.gov, respectively. Key contributors to this correspondence
include Michael A. Alexander, Tonia B. Brown, and Mary J. Dorsey.
Sincerely yours,
Linda D. Koontz
Director, Information Management Issues:
Signed by Linda D. Koontz:
[End of section]
Enclosure: Comments from Social Security:
SOCIAL SECURITY:
The Commissioner:
September 3, 2003:
Ms. Linda D. Koontz:
Director, Information Management Issues United States General
Accounting Office 441 G Street N. W.
Washington, D.C 20548:
Dear Ms. Koontz:
This responds to your letter dated August 27, 2003, requesting our
comments on your proposed correspondence entitled Social Security
Administration: Subcommittee Questions Concerning Efforts to Automate
the Disability Claims Process (GAO-03-1113R). I appreciate the
opportunity to review and comment on the correspondence before it is
issued. We have the following comments and clarifications on your
proposed responses to questions 1 through 3.
Question 1:
In your draft note to Congressman Shaw you indicate that in the
February 2003 cost-benefit analysis (CBA) the Social Security
Administration (SSA) "...did not consider all costs of certain critical
information technology infrastructure elements supporting the
nationwide rollout, such as scanning and imaging by the outsource
vendor, telecommunications, disaster recovery and on-site retention and
destruction of source documents, such as medical records." This
statement is incorrect, since we did in fact include the costs of
scanning and imaging by the outsource vendor in the estimated costs for
the accelerated electronic disability system (AeDib). Pages 8 and 9 of
the CBA (Version 3.1 dated February 3, 2003) list these costs among the
significant cost impacts for AeDib, with a pie chart showing outsource
scanning costs as 14 percent of the total life cycle costs. We consider
the other costs to be part of our ongoing operations, rather than
specifically related to AeDib, and these costs are more appropriately
covered in our regular infrastructure costs.
Your draft response also indicates that the benefits of AeDib may have
been overestimated because the State Disability Determination Services
(DDS) you contacted disagree with the estimate that 30 percent of all
medical evidence will be received in electronic form by 2004. During
the fall of 2002, we held several discussions with the DDS Systems
Committee and the National Council of Disability Determination
Directors leadership relative to business process and technological
issues. One of the central topics of these discussions was how to
gather
information that would assist in building the electronic medical
evidence (EME) estimates. We derived the 30 percent estimate based on
these discussions and information received from the DDS community. This
is a national estimate for fiscal year 2004, and we would expect that
this level would vary from location to location. Included in the 30
percent estimate is medical evidence received through both direct
electronic submissions as well as through fax submittals.
Question 2:
As stated in your draft note, we have provided the comprehensive risk
assessments for two of the five parts of AeDib. We had advised General
Accounting Office (GAO) staff that we were to receive the draft risk
assessments from our contractor by June 30, 2003. We then needed to
conduct an internal review of these documents. We shared two additional
risk assessments with you on August 28, 2003. The last one will be sent
after we complete our internal review.
Question 3:
Our comments on Question 2 also apply to the response to Question 3.
Additionally, we recently provided GAO staff with information regarding
stakeholder issues and our plans for consultation with the medical
community. Our senior executive leadership meets weekly to address
AeDib issues, including those raised by all entities, internal as well
as external.
Additionally, our outreach to the medical community is handled through
our public affairs and professional relations personnel in the DDSs and
in SSA's central office, Regional Offices, and Field Offices. The
issues of EME and processing disability claims in an electronic
environment are being institutionalized throughout SSA. They have been
apart of, and will continue to be included in, our contacts and
communication vehicles that are already in place, including
communication plans. Furthermore, I have undertaken special efforts to
communicate with the major medical associations by holding and
personally chairing a series of meetings to address their issues. In
addition, my staff has had a series of productive meetings with staff
at the Department of Health and Human Services on these issues.
If I may be of further assistance, please do not hesitate to contact me
or have your staff contact Mr. Robert M. Wilson, Deputy Commissioner
for Legislation and Congressional Affairs, at (202) 358-6030.
Sincerely,
Jo Anne B. Barnhart:
Signed by Jo Anne B. Barnhart:
[End of section]
(310386):
FOOTNOTES
[1] U.S. General Accounting Office, Electronic Disability Claims
Processing: Social Security Administration's Accelerated Strategy
Faces Significant Risks, GAO-03-984T (Washington, D.C.: July 24, 2003).
[2] U.S. General Accounting Office, Information Technology: Greater Use
of Best Practices Can Reduce Risks in Acquiring Defense Health Care
System, GAO-02-345 (Washington, D.C.: September 26, 2002).
[3] Social Security Administration, SSA AeDib Risk Management Plan,
Version 2.01 (Baltimore, MD, January 21, 2003).
[4] U.S. General Accounting Office, Social Security Administration:
Information Technology Challenges Facing the Commissioner, GAO/T-AIMD-
98-109 (Washington, D.C.: March 12, 1998).
[5] U.S. General Accounting Office, Social Security Administration:
Effective Leadership Needed to Meet Daunting Challenges, GAO/HEHS-96-
196 (Washington, D.C.: September 12, 1996).
[6] U.S. General Accounting Office, Social Security Administration:
Technical and Performance Challenges Threaten Progress of
Modernization, GAO/AIMD-98-136 (Washington, D.C.: June 19, 1998).
[7] U.S. General Accounting Office, Information Technology Management:
Social Security Administration Practices Can Be Improved, GAO-01-961
(Washington, D.C.: August 21, 2001).
[8] Office of Management and Budget Circular No. A-94 Revised
(Transmittal Memo No. 64), Guidelines and Discount Rates for Benefit-
Cost Analysis of Federal Programs, October 29, 1992.
