Office of Personnel Management
Improvements Needed to Ensure Successful Retirement Systems Modernization Gao ID: GAO-08-345 January 31, 2008Through its Retirement Systems Modernization (RSM) program, the Office of Personnel Management (OPM) is modernizing the paper intensive processes and antiquated information systems it uses to support the retirement of civilian federal employees. RSM is intended to deploy new or modified systems beginning in February 2008 to improve the efficiency and effectiveness of the agency's retirement program. GAO was asked to (1) determine whether OPM is effectively managing the RSM program to ensure that system components perform as intended and (2) evaluate the risks, cost, and progress of the RSM program. To meet these objectives, GAO analyzed program documentation against relevant plans, policies, and practices.
In executing the RSM program, OPM has improved its management processes for selecting contractors, defining system and security requirements, managing risks, planning organizational change, and providing program executive oversight. The agency also recently established performance targets for the improvements to retirement processing accuracy and timeliness that it expects the program to achieve and established an independent verification and validation capability. However, the agency's management of RSM in areas that are important to successful deployment of new systems has not ensured that system components will perform as intended. Specifically, initial test results do not provide assurance that a major system component will perform as intended. In addition, OPM's system testing schedule has been compressed, and upcoming tests are to be conducted concurrently, increasing the risk that the agency will not have sufficient resources or time for testing. Further, trends in identifying and resolving system defects indicate a growing backlog of problems to be resolved prior to deployment. Although OPM has established a risk management process that has identified program risks, the agency has not reliably estimated RSM's cost or reported progress. In particular, the reliability of the program's revised life-cycle cost estimate of $421.6 million is questionable because the agency could not support the estimate with a description of the system to be developed and a description of the methodology used to produce the estimate. Also, the agency's reporting of RSM progress--based on the satisfaction of established program goals and the calculation of variances from the planned cost and schedule--has not reflected the state of the program. With respect to goals, the agency reported that it had met its fiscal year 2007 goals, including completing the imaging of paper-based retirement records and beginning training. While OPM's reporting that it satisfied program goals provided a favorable view of progress, this view did not include program areas for which the agency had not established goals. With respect to program cost and schedule variances, OPM reported in October 2007 that the program was progressing almost exactly as planned. However, the agency's reported favorable view of program progress was not consistent with the state of the program.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-08-345, Office of Personnel Management: Improvements Needed to Ensure Successful Retirement Systems Modernization
This is the accessible text file for GAO report number GAO-08-345
entitled 'Office Of Personnel Management: Improvements Needed to Ensure
Successful Retirement Systems Modernization' which was released on
January 31, 2008.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
United States Government Accountability Office:
GAO:
Report to the Subcommittee on Financial Services and General
Government, Committee on Appropriations, U.S. Senate:
January 2008:
Office Of Personnel Management:
Improvements Needed to Ensure Successful Retirement Systems
Modernization:
GAO-08-345:
GAO Highlights:
Highlights of GAO-08-345, a report to the Subcommittee on Financial
Services and General Government, Committee on Appropriations, U.S.
Senate.
Why GAO Did This Study:
Through its Retirement Systems Modernization (RSM) program, the Office
of Personnel Management (OPM) is modernizing the paper intensive
processes and antiquated information systems it uses to support the
retirement of civilian federal employees. RSM is intended to deploy new
or modified systems beginning in February 2008 to improve the
efficiency and effectiveness of the agency‘s retirement program. GAO
was asked to (1) determine whether OPM is effectively managing the RSM
program to ensure that system components perform as intended and (2)
evaluate the risks, cost, and progress of the RSM program. To meet
these objectives, GAO analyzed program documentation against relevant
plans, policies, and practices.
What GAO Found:
In executing the RSM program, OPM has improved its management processes
for selecting contractors, defining system and security requirements,
managing risks, planning organizational change, and providing program
executive oversight. The agency also recently established performance
targets for the improvements to retirement processing accuracy and
timeliness that it expects the program to achieve and established an
independent verification and validation capability. However, the
agency‘s management of RSM in areas that are important to successful
deployment of new systems has not ensured that system components will
perform as intended. Specifically, initial test results do not provide
assurance that a major system component will perform as intended. In
addition, OPM‘s system testing schedule has been compressed, and
upcoming tests are to be conducted concurrently, increasing the risk
that the agency will not have sufficient resources or time for testing
(see figure below). Further, trends in identifying and resolving system
defects indicate a growing backlog of problems to be resolved prior to
deployment.
Although OPM has established a risk management process that has
identified program risks, the agency has not reliably estimated RSM‘s
cost or reported progress. In particular, the reliability of the
program‘s revised life-cycle cost estimate of $421.6 million is
questionable because the agency could not support the estimate with a
description of the system to be developed and a description of the
methodology used to produce the estimate. Also, the agency‘s reporting
of RSM progress”based on the satisfaction of established program goals
and the calculation of variances from the planned cost and schedule”has
not reflected the state of the program. With respect to goals, the
agency reported that it had met its fiscal year 2007 goals, including
completing the imaging of paper-based retirement records and beginning
training. While OPM‘s reporting that it satisfied program goals
provided a favorable view of progress, this view did not include
program areas for which the agency had not established goals. With
respect to program cost and schedule variances, OPM reported in October
2007 that the program was progressing almost exactly as planned.
However, the agency‘s reported favorable view of program progress was
not consistent with the state of the program.
RSM Original versus Revised Test Schedule:
Original dates:
Integrated product test: August, September, 2007;
Performance test: October, 2007;
Parallel/business capacity release test: November, December, 2007;
System deployment: early February, 2008.
Revised dates:
Parallel test: late December, 2007;
Parallel test: mid-January to mid-February, 2008;
Integrated product test: late January to mid-February, 2008;
Business capability release test: late January to late February, 2008;
System deployment: late February, 2008.
Source: GAO based on OPM data.
What GAO Recommends:
To improve the management of RSM and reduce the risks to successful
system deployment, GAO is making recommendations to the Director of the
Office of Personnel Management, including conducting effective system
tests, resolving system defects, and improving cost estimating and
earned value reporting. In written comments on a draft of this report,
the Director of OPM expressed her appreciation of GAO‘s insightful
recommendations and stated that the agency is taking steps to address
them.
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.GAO-08-345]. For more information, contact
Valerie C. Melvin at (202) 512-6304 or melvinv@gao.gov
[End of section]
Contents:
Letter:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Briefing to the Staff of the Subcommittee on Financial
Services and General Government, Committee on Appropriations, U.S.
Senate:
Appendix II: Comments from the Office of Personnel Management:
Appendix III: GAO Contact and Staff Acknowledgments:
Abbreviations:
CIS: Center for Information Services:
CSRS: Civil Service Retirement System:
DBTS: defined benefits technology solution:
EVM: earned value management:
FERS: Federal Employees Retirement System:
IV&V: independent verification and validation:
OPM: Office of Personnel Management:
PMB: performance measurement baseline:
RSM: Retirement Systems Modernization:
UAT: user acceptance test:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
January 31, 2008:
The Honorable Richard Durbin:
Chairman:
The Honorable Sam Brownback:
Ranking Member:
Subcommittee on Financial Services and General Government:
Committee on Appropriations:
United States Senate:
The Office of Personnel Management (OPM) is modernizing the paper
intensive processes and antiquated systems it uses to support the
retirement of civilian federal employees. According to OPM, existing
processes and systems make providing timely and accurate benefit
payments to retirees and their families increasingly difficult and must
be improved before an expected increase in the number of retirements
occurs. The Retirement Systems Modernization (RSM) program is intended
to remedy this situation with the deployment of new or modified systems
beginning in February 2008, and full deployment planned by the end of
2009. The agency expects RSM to improve the efficiency and
effectiveness of its retirement program, which serves civilian federal
employees who are eligible to receive benefits in the future, employees
who are already retired, and their survivors and beneficiaries. OPM has
estimated the RSM life-cycle cost to be $421.6 million.
At your request, we reviewed OPM's management of the RSM program.
Specifically, our objectives were to:
* determine whether OPM is effectively managing the RSM program to
ensure that system components perform as intended, and:
* evaluate the risks, cost, and progress of the program.
On November 30, 2007, we provided your offices with briefing slides
that outlined the results of our study. On December 3, 2007, we met
with your staff to discuss our findings, conclusions, and
recommendations. The purpose of this report is to provide the published
briefing slides to you and to officially transmit our recommendations
to the Director of the Office of Personnel Management. The slides,
which discuss our scope and methodology, are included as appendix I. We
performed our work from March 2007 to January 2008 in accordance with
generally accepted government auditing standards.
In summary, our study highlighted two key issues:
First, OPM has improved its management processes for selecting
contractors, defining system and security requirements, managing risks,
planning organizational change, and providing program executive
oversight. Additionally, the agency recently established performance
targets for the improvements to retirement processing accuracy and
timeliness that it expects RSM to achieve. However, the agency's
management of the program in other areas that are important to the
successful deployment of new systems has not ensured that system
components will perform as intended. A recently established independent
verification and validation capability should help the agency identify
and make program management improvements. Nevertheless, management
improvements are needed in key areas:
* Initial test results do not provide assurance that a major system
component, the defined benefits technology solution, will perform as
intended. OPM officials acknowledged that test results had not met
established quality goals and stated that they expected future test
results to indicate improved quality. Nevertheless, until actual test
results indicate improved system quality, the agency faces increased
risk that it will deploy technology that does not work as expected in
February 2008.
* The system testing schedule has been compressed and upcoming tests
are to be conducted concurrently in about half the time originally
planned. The agency identified a shortage of testing resources and the
need for further system development as contributing to the delay and
concurrency of planned tests. This high degree of concurrent testing
increases the risk that the agency will not have sufficient resources
or time to verify that the technology it plans to deploy in February
2008 will work as expected.
* Trends in identifying and resolving system defects indicate a growing
backlog of problems to be resolved prior to deployment. Until defect
trends indicate resolution of the backlog of urgent and high priority
defects, OPM faces increased risk that it will not have sufficient time
to resolve significant problems before its planned February 2008
deployment.
Second, OPM has established a risk management process that has been
effective in identifying program risks, but it has not reliably
estimated RSM's cost or reported progress. Several examples follow:
* The agency has established and used a risk management process that
has resulted in the identification of risks to the successful
completion of the RSM program. For example, the agency has identified
risks associated with the need to plan activities, conduct training,
design and build interfaces, modify legacy systems, and execute tests
prior to system deployment in February 2008. As a result of identifying
these and other risks, the program should be positioned to reduce the
probability of their occurrence and to reduce the impact if they occur.
* In 2007, OPM revised the program life-cycle cost estimate from $371.2
million to $421.6 million. However, the reliability of this estimate is
questionable because the agency could not support the estimate with a
description of the system to be developed and a description of the
methodology used to produce the estimate. Without a reliable cost
estimate, the agency does not have a sound basis for formulating future
RSM program budgets or for monitoring and predicting program
performance.
* OPM's reporting of RSM progress has not reflected the state of the
program. Specifically, the agency reported two views of program
progress: by describing the satisfaction of established program goals
and by using earned value management (EVM).[Footnote 1] With respect to
goals, the agency reported that it had met its fiscal year 2007 goals,
including completing the imaging of paper-based retirement records and
beginning training. While OPM's reporting that it satisfied program
goals provided a favorable view of progress, this view did not include
program areas for which the agency had not established goals. Using
EVM, which is intended to provide a view of progress on the program as
a whole, the agency reported in October 2007 that the program was
progressing almost exactly as planned. However, the agency's reported
favorable view of program progress was not consistent with the state of
the program. As a result of this approach, whereby OPM frequently
revised its performance measurement baseline in lieu of establishing
and controlling a valid baseline, the agency's EVM reporting did not
reliably reflect program progress.
Conclusions:
To its credit, OPM has undertaken the RSM program to expedite
retirement processing for civilian federal employees and the agency
reported that it has met key program goals. Further, the agency has
improved its management processes for selecting contractors, defining
system and security requirements, managing risks, planning
organizational change, and providing program executive oversight.
Nevertheless, much remains to be accomplished before the program is
effectively positioned to deploy its first planned increment of new
technology in February 2008.
Although OPM has developed performance targets necessary to gauge the
success of the new system and has entered into a contract to obtain an
independent review of its management of the RSM program, the agency has
not ensured that system components will perform as intended. In
particular, initial test results indicate that the defined benefits
technology solution that is a major component of the new system has not
performed as intended, the backlog of system defects to be addressed
before deployment is growing, and future system tests are to be
conducted concurrently in about half the time originally planned.
Further, OPM recognized the importance of risk management and has
established a risk management process and identified program risks.
However, the agency has not yet developed the capability to reliably
analyze and report program progress. Such progress reporting should be
grounded in a reliable cost estimate that is in part the basis for
reliable earned value management. Without a reliable cost estimate, the
agency does not have a firm foundation for the RSM program budget or
for reliable earned value management reporting.
Until OPM makes improvements to the RSM program in the areas discussed
above, the agency risks not achieving successful program outcomes,
including the planned deployment of new technology beginning in
February 2008.
Recommendations for Executive Action:
To address the risks to OPM's deployment of new retirement system
technology and improve the agency's ability to reliably report progress
of the RSM program, we are recommending that the Director of the Office
of Personnel Management direct the RSM Executive Director to take the
following actions:
* Ensure that sufficient resources are provided to fully test
functionality, actions for mitigating the risks inherent in concurrent
testing are identified, test results verify that all system components
perform as expected, and test activities and results are subjected to
independent verification and validation.
* Monitor and review defined benefits technology solution defects to
ensure all urgent and high priority defects are resolved prior to
system deployment and that the resolution of urgent and high priority
defects is subjected to independent verification and validation.
* Develop a revised RSM cost estimate that addresses the weaknesses
identified in this briefing and task an independent verification and
validation contractor with reviewing the process used to develop the
estimate and assessing the reliability of the resulting estimate.
* Establish a basis for effective use of earned value management by
validating the RSM performance measurement baseline through a program
level integrated baseline review and task an independent verification
and validation contractor with reviewing the process used to develop
the baseline and assessing the reliability of the performance
measurement baseline.
Agency Comments and Our Evaluation:
In written comments on a draft of this report, the Director of OPM
expressed her appreciation of GAO's insightful recommendations and
stated that the agency is taking steps to address them. The Director
reiterated the agency's intention to begin deploying the new retirement
system in February 2008, and stated that when fully deployed, the new
system will provide a high level of customer service, enhanced
retirement planning tools, and prompt, complete annuity payments. To
this end, the Director stated that the agency has, among other things,
dedicated additional resources to monitor, evaluate, and troubleshoot
system development and testing. OPM's actions, if effectively
implemented and monitored by agency leadership, should facilitate
deployment of the new retirement system. The comments are reprinted in
appendix II.
We are sending copies of this report to the Director of the Office of
Personnel Management and other appropriate congressional committees. We
will make copies available to other interested parties upon request.
Copies of this report will also be made available at no charge on GAO's
Web site at [hyperlink, http://www.gao.gov].
If you or your staff have any questions about this report, please
contact me at (202) 512-6304 or melvinv@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. GAO staff who made major contributions to
this report are listed in appendix III.
Signed by:
Valerie C. Melvin:
Director, Human Capital and Management:
Information Systems Issues:
[End of section]
Appendix I: Briefing to the Staff of the Subcommittee on Financial
Services and General Government, Committee on Appropriations, U.S.
Senate:
Office of Personnel Management: Improvements Needed to Ensure
Successful Retirement Systems Modernization:
Briefing for the Subcommittee on Financial Services and General
Government:
Committee on Appropriations:
United States Senate:
November 30, 2007:
Table of Contents:
Introduction:
Objectives:
Scope and Methodology:
Results in Brief:
Background:
Results:
Conclusions:
Recommendations:
Agency Comments and Our Evaluation:
[End of section]
Introduction:
The Office of Personnel Management (OPM) is modernizing the paper
intensive processes and antiquated systems it uses to support the
retirement of civilian federal employees. According to OPM, these
existing processes and systems make providing timely and accurate
benefit payments to retirees and their families increasingly difficult
and must be improved before an expected increase in the number of
retirements occurs. The Retirement Systems Modernization (RSM) program
is intended to remedy this situation with the deployment of new or
modified systems beginning in February 2008, with full deployment
planned by the end of 2009. OPM has estimated the RSM life-cycle cost
to be $421.6 million.
RSM is intended to improve the efficiency and effectiveness of OPM‘s
retirement program, which serves civilian federal employees who are
eligible to receive benefits in the future, employees who are already
retired and their survivors and beneficiaries.
Objectives:
The Chairman and the Ranking Member of the Senate Subcommittee on
Financial Services and General Government requested that we:
* determine whether OPM is effectively managing the RSM program to
ensure that system components perform as intended, and;
* evaluate the risks, cost, and progress of the RSM program.
Scope and Methodology:
To determine whether OPM is effectively managing the RSM program to
ensure that system components perform as intended, we:
* analyzed RSM program objectives and current retirement performance
measures to determine what target goals OPM established for the
modernized system, and evaluated metrics developed for the new system
and analyzed the extent to which those metrics included target goals
for performance relative to program objectives;
* evaluated planned tests and dates and compared them to actual tests
completed to determine to what extent tests were delayed or compressed,
and analyzed system test results and compared them to expected quality
goals to determine whether test results provided assurance that the
system would perform as intended;
* analyzed defect data of a major system component to determine the
number of defects identified and resolved each week since the beginning
of development, and analyzed trends in defects opened and identified
the number of defects to be resolved prior to deployment;
* evaluated independent verification and validation (IV&V) plans and
analyzed what program activities were subject to IV&V to determine the
scope of these activities; and;
* interviewed OPM officials, RSM program management, and contractors
about performance measures, testing activities, defect management, and
IV&V plans.
To evaluate RSM risks, cost, and progress, we:
* reviewed the RSM risk management program to determine to what extent
the program had been implemented, and analyzed risk logs and attended
monthly program management review meetings at which program risks were
identified, tracked, mitigated, and reported;
* analyzed the current RSM life-cycle cost and compared the process OPM
used to develop the cost estimate with best practices identified in
GAO‘s Cost Assessment Guide, and reviewed cost estimation documentation
to determine the sufficiency of support for the estimate as outlined in
the guide;
* evaluated RSM progress as reported through program goals and earned
value management; reviewed fiscal year 2007 program goals to determine
progress toward achievement of these goals; evaluated RSM progress as
reported in EVM performance reports to determine variances in cost and
schedule, and analyzed inputs to performance reports to evaluate to
what extent the EVM approach aligned with best practices identified in
the GAO Cost Assessment Guide; and;
* interviewed RSM program officials and contractors at OPM headquarters
about risk management, cost estimating, and program progress.
Our work was performed at OPM headquarters in Washington, D.C., from
March 2007 to November 2007 in accordance with generally accepted
government auditing standards.
Results in Brief:
To its credit, OPM has undertaken the RSM program to speed retirement
processing for civilian federal employees and has reported meeting key
agency goals. Further, the agency has improved its management processes
for selecting contractors, defining system and security requirements,
managing risks, planning organizational change, and providing program
executive oversight. Additionally, OPM recently established performance
targets for the improvements to retirement processing accuracy and
timeliness that it expects RSM to achieve. However, the agency has not
ensured that system components will perform as intended when they are
planned to be deployed in February 2008. A recently established
independent verification and validation function should help the agency
identify and make program management improvements. Nevertheless,
management improvements are needed in key areas:
* Initial test results do not provide assurance that a major system
component will perform as intended. OPM officials acknowledged that
test results had not met established quality goals and stated that they
expected future test results to indicate improved quality.
Nevertheless, until actual test results indicate improved system
quality, OPM faces increased risk that it will deploy technology that
does not work as expected (e.g., does not accurately calculate
retirement benefits) in February 2008.
* OPM‘s system testing schedule has been compressed and upcoming tests
are to be conducted concurrently. OPM identified a shortage of testing
resources and the need for further system development to occur as
contributing to the delay and increased concurrency of planned tests.
This high degree of concurrent testing that OPM now plans increases the
risk that OPM will not have sufficient resources or time to verify that
the technology it plans to deploy in February 2008 will work as
expected.
* Trends in identifying and resolving system defects indicate a growing
backlog of problems to be resolved prior to deployment. Until defect
trends indicate resolution of the backlog of urgent and high priority
defects, OPM faces increased risk that it will not have sufficient time
to resolve significant problems before its planned February 2008
deployment.
OPM has established a risk management process that has been effective
in identifying program risks, but has not reliably estimated RSM‘s cost
or reported progress:
* OPM has established and used a risk management process, which has
resulted in the identification of risks to successful completion of the
RSM program. For example, the agency has identified risks associated
with the need to plan activities, conduct training, design and build
interfaces, modify legacy systems, and execute tests prior to system
deployment in February 2008. As a result of identifying these and other
risks, the RSM program should be positioned to reduce the probability
of their occurrence and to reduce the impact if the risks do occur.
* In 2007, OPM revised its RSM life-cycle cost estimate from $371.2
million to $421.6 million. However, the reliability of this estimate is
questionable because the agency could not support the estimate with a
description of the system to be developed and a description of the
methodology used to produce the estimate. Without a reliable cost
estimate, OPM does not have a sound basis for formulating future RSM
program budgets or for monitoring and predicting program performance.
* OPM‘s reporting of RSM progress has not reflected the state of the
program. Specifically, the agency reported two views of RSM progress:
by describing satisfaction of established program goals and by using
earned value management (EVM). [Footnote 2] With respect to goals, the
agency reported that it had met its fiscal year 2007 goals, including
completing the imaging of paper-based retirement records and beginning
RSM training. While OPM‘s reporting that it satisfied program goals
provided a favorable view of progress, this view did not include
program areas for which the agency had not established goals. Using
EVM, which is intended to provide a view of progress on the program as
a whole, OPM reported in October 2007 that the program was progressing
almost exactly as planned. However, the agency‘s reported favorable
view of program progress was not consistent with the state of the
program.
To improve OPM‘s management of RSM, we are making recommendations aimed
at reducing the risks to successful system deployment by conducting
effective system tests, resolving system defects, and improving cost
estimating and earned value reporting (see pages 53-54).
In oral comments on a draft of this briefing, OPM officials including
the Director and the RSM Executive Director generally agreed with our
recommendations and provided additional information and written
technical comments related to program activities, which we incorporated
in the briefing as appropriate.
Background: OPM Mission and Retirement Plans:
OPM‘s mission is to ensure that the federal government has an effective
civilian workforce. [Footnote 3] In this regard, one of its major human
resources tasks is to manage and administer the federal retirement
program for federal employees.
OPM‘s Center for Retirement and Insurance Services administers two
defined-benefit retirement plans that provide retirement, disability,
and survivor benefits for federal employees: [Footnote 4]
* Civil Service Retirement System (CSRS)”a pension system that covers
most employees hired before 1984.
* Federal Employees Retirement System (FERS)”a plan that also includes
Social Security and a defined contribution system; it covers most
employees hired in 1984 and subsequent years. [Footnote 5]
Background: Retirement Services:
According to OPM, there are approximately 2.5 million active federal
employees and 2.4 million retired federal employees. The agency
estimates that between 2007 and 2017,60 percent of active federal
employees will be eligible to retire and 40 percent will most likely
retire.
OPM reports that the current retirement process takes approximately 60
to 90 days from the submission of a retirement application until
initial benefit payments are made to a federal retiree.
OPM has identified factors that could limit its ability to provide high-
quality retirement services, including:
* an increasing number of FERS retirement claims, which are more
difficult to process than CSRS claims because of the complexity of FERS
calculations;
* high costs, limited capabilities, and other problems with existing
OPM information systems and processes; and;
* the inability to attract personnel to work with antiquated retirement
processes.
Background: RSM History:
OPM began planning RSM in 1997 and originally intended to structure the
program as an acquisition of commercially available hardware and
software that the agency would modify to meet its needs. From 1997 to
2001, OPM developed plans and analyses and began developing business
and security requirements for RSM.
In 2001, OPM decided to change the direction of RSM. Specifically, the
agency examined the possibility of increasing the role of private
sector vendors by contracting for key system components.
In 2002, the agency issued a request for information to vendors
regarding contracting for key retirement program functions. The
agency‘s analysis of responses showed that contracting was a viable
alternative that would be cost efficient, less risky, and more likely
to be completed on time and on budget. As a result, the agency decided
to contract for use of a commercially available pension benefits
solution.
In 2003, OPM conducted an assessment of alternative contracting options
before issuing a request for proposals in 2004. During 2005 and early
2006, OPM evaluated proposals and selected vendors.
Background: RSM Objectives:
OPM‘s January 2007 RSM Program Management Plan defined two program
objectives to address current retirement processing performance issues:
* Timely, accurate retirement benefit payments”streamlined electronic
processing of retirement applications for the establishment of accurate
automatic benefit payments for retirees.
* More efficient and flexible processes”electronic processes to record
and retrieve retirement information to reduce delays associated with
the storage and management of paper records, and reduce processing
times with fewer errors.
With RSM, OPM expects the retirement process will take less time and
result in fewer errors.
Background: RSM Program:
The RSM program is expected to improve retirement services to active
and retired federal employees by implementing new technology and
business processes. Modernizing the current paper-based manual
processing of retirement applications and claims involves:
* modifying a commercial pension benefits solution, called the Defined
Benefits Technology Solution (DBTS), to meet federal requirements and
policies;
* capturing and converting federal employee paper records to electronic
files;
* redesigning retirement processes to align with supporting technology;
* developing interfaces to receive data from and send data to external
and remaining legacy systems;
* decommissioning or modifying OPM‘s 86 legacy systems that currently
support retirement processing; and;
* training OPM employees to use the modernized technology and
processes.
The cornerstone technology of the RSM program is DBTS. In addition to
calculating retirement benefit amounts, the technology is intended to
provide active and already retired federal employees self-service
Internet-based tools for accessing accounts, updating retirement
records, submitting transactions, monitoring the status of claims, and
forecasting retirement income. This technology is also expected to
provide electronic tools for accessing retirement information to OPM
staff and agency customer service representatives.
In addition to DBTS, RSM includes modifying OPM legacy systems and
developing interfaces to external and internal systems. Figure 1 shows
the planned process and technology for the program.
Figure 1: Simplified View of Planned RSM Process and Technology:
* Payroll Processing Centers (e.g. GDA):
* OPM Interface;
* Defined Benefits Technology Solution (e.g. Benefit estimation,
calculation, and payment, claims processing, customer service;
- Active employees, benefit officers, retirees, and OPM administrators;
* OPM Interface;
* OPM Legacy Systems (e.g. financial management, actuarial analysis);
* External Systems (e.g. SSA, Treasury).
Source: GAO analysis of OPM data.
[End of figure]
Background: RSM Contracts:
OPM has overall responsibility for RSM, including retirement process
redesign, legacy system modification, interface development,
integrating system components, system testing, training employees, and
program management.Additionally, the RSM program is supported by
contractors in key areas, as shown in the following table.
Table 1: RSM Program Contracts:
Contract Name: DBTS;
Award Date: May 2006;
Contract Value: $290 million;
Awarded to: Hewitt Associates;
Deliverables: Modification of commercial solution to meet federal
requirements and policies.
Contract Name: Paper Data Capture and Conversion;
Award Date: September 2006;
Contract Value: $30.7 million
Awarded to: Integic;
Deliverables: Capture and conversion of federal paper records to
electronic files.
Contract Name: Business Transformation and Information Technology;
Award Date: May 2006;
Contract Value: $40 million;
Awarded to: Accenture Ltd.
Deliverables: Assist in redesigning the retirement processes and
developing supporting technology.
Contract Name: Program Management;
Award Date: September 2004;
Contract Value: $10.7 million;
Awarded to: Booz Allen Hamilton;
Deliverables: Program management support.
Contract Name: IV&V;
Award Date: August 2007;
Contract Value: $0.2 million;
Awarded to: Bearing Point;
Deliverables: Requirements assessment, test validation and results
evaluation.
[End of table]
Background: RSM Implementation Approach:
OPM plans to implement its DBTS-based retirement system through a
series of five increments. Each increment consists of deploying the new
retirement system to support specific populations of federal agency
employees according to the following schedule:
* Increment 1 – active employees serviced by the General Services
Administration payroll processing center starting in February 2008.
* Increment 2 – active United States Postal Service employees starting
in May 2008.
* Increment 3 – already retired federal employees, active employees
serviced by the National Business Center payroll processing center, and
other independent agencies starting in August 2008.
* Increment 4 – active employees serviced by the National Finance
Center payroll processing center starting in November 2008.
* Increment 5 – active employees serviced by the Department of Defense
payroll processing center in February 2009.
Background: RSM Governance:
OPM has established governance committees and senior management
positions to lead RSM, as table 2 shows.
Table 2: RSM Governance:
Title: Executive Steering Committee;
Description: Chaired by the OPM Director, the Executive Steering
Committee makes decisions related to the RSM program that have cross-
OPM impact and resolves cross-OPM issues.
Title: Associate Director of Human Resources Products and Services;
Description: As the current business owner, the associate director is
responsible for the transition to operations and assumes ownership of
the RSM program when complete.
Title: Executive Director;
Description: As the program manager, the executive director is
responsible for daily operations and progress of the program.
Title: Chief Information Officer;
Description: As the deputy associate director for the Center for
Information Services (CIS), the chief information officer is
responsible for providing support and oversight for acquisition,
systems, contract, and security management. CIS supports the transition
and modification of legacy systems, which are owned by this unit.
[End of table]
Background: GAO Prior Review:
In February 2005, we reported that OPM lacked processes for RSM
acquisition activities such as determining requirements, developing
acquisition strategies, and implementing a risk program. Furthermore,
the agency also had not yet established effective security management,
change management, and program executive oversight. [Footnote 6]
Accordingly, we recommended that the director of OPM ensure that the
RSM program office expeditiously establish processes for effective
oversight of RSM. We made nine recommendations in the areas of system
acquisition management, information security, organizational change
management and IT investment management.
Between 2005 and 2007, OPM made progress towards establishing
management processes for RSM and demonstrated the completion of
activities with respect to each of the nine recommendations.
Specifically, the agency developed key system acquisition processes for
determining requirements, developed acquisition strategies, and
implemented a risk management program.
Additionally, OPM developed information security plans and
requirements, updated organizational change management plans, and
instituted processes for guiding the executive investment oversight
committee. For example, OPM:
* developed both system specific and programwide security plans along
with a set of security requirements for RSM;
* updated its change management plans to include tasks and milestones
that represented the current acquisition approach; and;
* developed and established processes to guide the executive steering
committee‘s activities through a charter and program management plans.
As a result of these actions, OPM improved its ability to select
contractors, define system and security requirements, manage risks,
plan organizational change, and provide RSM program executive
oversight.
Objective 1: Management Effectiveness:
OPM‘s Management of the RSM Program Has Not Ensured that System
Components Will Perform as Intended, but IV&V Capability Could Result
in Improvements.
OPM recently established performance targets for the improvements to
retirement processing accuracy and timeliness that it expects RSM to
achieve. However, the agency‘s management of RSM in other areas that
are important to successful deployment of new systems has not ensured
that system components will perform as intended. Key areas of program
management weaknesses are system testing and defect management. The
agency recently established an IV&V capability, which should identify
and recommend areas for improvement.
Objective 1: Performance Measures:
Performance targets for expected improvements to retirement processing
were recently developed.
The Government Performance and Results Act of 1993 provides, among
other things, that federal agencies establish program performance
measures, including the assessment of relevant outputs and outcomes of
program activities. By analyzing the gap between target measurements
and actual levels of performance, management can focus on those
processes that are most in need of improvement, set improvement goals,
and identify appropriate process improvements or other actions.
[Footnote 7]
Recognizing the importance of performance measures, OPM has established
goals for measuring the accuracy and timeliness of current retirement
claims processing. OPM also has defined targets that relate to
measuring progress towards these goals and reports actual performance.
Specifically, in 2006 the agency reported targets for current
retirement claims processing, including:
* 93 percent accuracy (actual accuracy was 89 percent) and;
* 30-day claims processing (actual performance was 41 days).
According to the RSM Program Management Plan, OPM expects RSM to
achieve more accurate and faster retirement claims processing.
In November 2007, the agency completed its development of new
performance targets for timeliness and accuracy. Specifically, the
agency developed new targets for future retirement claims processing,
including:
* 99 percent accuracy and;
* 30-day processing for 99 percent of claims.
According to the OPM Director, the agency plans to use these targets
beginning in February 2008. By establishing these targets, OPM should
be positioned to determine whether the technology that it plans to
deploy in February 2008 and beyond will result in the timeliness and
accuracy improvements in retirement processing that it has asserted RSM
will achieve.
Objective 1: Test Results:
Test results do not provide assurance that a major system component
will perform as intended.
Effective testing is an essential component of any program that
includes system development. Generally, the purpose of testing is to
identify defects or problems in meeting defined system requirements or
satisfying system user needs. [Footnote 8]
Recognizing the importance of testing, OPM plans to perform user
acceptance tests (UATs) that are intended to verify that DBTS meets
requirements such as accurately calculating retirement benefits. The
agency plans six UATs to be conducted incrementally as the system is
developed and established goals, expressed as the percentage of test
scenarios passed, for each UAT.
Although the results of UAT 1 slightly exceeded the goal, the results
of UATs 2, 3, and 4 fell far short of meeting the established goals and
indicate that DBTS is not performing as intended.
Table 3: Test Results in Terms of Scenarios:
Test: UAT1;
Scenarios Tested: 44;
Scenarios Passed: 34;
Scenarios Failed: 20;
Passed Goal (percent): 75;
Passed Actual (percent): 77.
Test: UAT2;
Scenarios Tested: 179;
Scenarios Passed: 8;
Scenarios Failed: 171;
Passed Goal (percent): 80;
Passed Actual (percent): 4.
Test: UAT3;
Scenarios Tested: 193;
Scenarios Passed: 42;
Scenarios Failed: 151;
Passed Goal (percent): 85;
Passed Actual (percent): 22.
Test: UAT4;
Scenarios Tested: 302;
Scenarios Passed: 117;
Scenarios Failed: 185;
Passed Goal (percent): 90;
Passed Actual (percent): 39.
[End of table]
OPM officials acknowledged that these test results showed that DBTS had
not performed as intended and stated that they expected the UAT 5 test
results to indicate continued quality improvement.
Until actual test results indicate improvement in the quality of DBTS,
OPM faces increased risk that it will deploy technology that does not
work as expected (e.g., does not accurately calculate retirement
benefits) in February 2008.
Objective 1: Test Schedule:
Testing schedule is compressed and concurrency of tests is increased.
In addition to UATs, OPM planned tests that we and others recognize as
important to ensuring that system components and the new system as a
whole perform as intended. [Footnote 9] These tests are intended to
verify that DBTS and other RSM components work together as intended
when they are combined and that the complete system resulting from the
RSM program satisfies all requirements (e.g., functional and
performance) and is acceptable to end users. Specifically, OPM plans
the following tests:
* integrated product test to confirm that DBTS, modified legacy
systems, and associated interfaces meet functional requirements (e.g.,
accurately calculate benefits).
* performance test to confirm that the new system meets performance
requirements (e.g., processing volume and execution time).
* parallel test to verify that the new system produces the same results
as existing systems.
* business capability test to confirm the operational readiness of the
new system for end users.
OPM planned to perform these tests in sequence over about 5 months from
late July 2007 through December 2007.
In November 2007 OPM revised its test schedule. According to the
revised schedule, the integrated product test was delayed from late
July 2007 until late January 2008. In order to maintain its schedule to
deploy technology in late February 2008, the agency now plans to reduce
the time necessary to conduct the integrated product, performance,
parallel, and business capability tests by performing them concurrently
within about a 2-1/2 month time period beginning in mid-December 2007
and concluding in late February 2008.
OPM‘s original test schedule is compared to the actual or revised
schedule in figure 2.
Figure 2: RSM Original Versus Actual or Revised Test Schedule:
[See PDF for image]
Original dates:
UAT1: Mid-March, 2007;
UAT2: Late May, 2007;
UAT3: Early August, 2007;
UAT4: Late September, 2007;
Integrated test product: August and September, 2007;
Performance test: October, 2007;
Parallel/business capability release test: November and December, 2007;
UAT5: Late December, 2007;
Increment 1 deployment: Early February, 2008.
Revised or Actual dates:
UAT1: Mid-March, 2007;
UAT2 part 1: Late May, 2007;
UAT2 part 2: Early June, 2007;
UAT3 part 1: Early August, 2007;
UAT3 part 2: Mid-September, 2007;
UAT4: Mid-October, 2007;
UAT5: Mid-December, 2007;
Parallel test: Mid-to-late December, 2007;
UAT6: Mid-January, 2008;
Parallel test: Mid-January to Mid-February, 2008;
Performance test: Early to Mid-February, 2008;
Integrated product test: Early to Mid-February, 2008;
Business capability release test: February, 2008;
Increment 1 deployment: Late February, 2008.
Source: GAO based on OPM data.
[End of figure]
OPM identified the lack of testing resources, including the
availability of subject matter experts, and the need for further system
development as contributing to the delay of planned tests and the need
for concurrent testing.
As a result of test concurrency, OPM is faced with performing a
significant volume of concurrent test activities at the same time that
critical resources, particularly key staff, are also engaged in
activities such as completing the building of interfaces and
modification of legacy systems. As we have previously reported,
concurrent tests can require additional time if critical defects are
found that necessitate stopping all affected tests, fixing the baseline
configuration, and then restarting the tests. [Footnote 10] The high
degree of concurrent testing that OPM now plans increases the risk that
OPM will not have sufficient resources or time to verify that the
technology it plans to deploy in February 2008 works as expected.
Objective 1: Defect Management:
Trends in defects indicate a growing backlog of problems to be resolved
before deployment.
In addition to test results, a measure of system maturity and quality
is trends in defects. Defects are system problems that require a
resolution and can be due to a failure to meet the system
specifications. Defects are often identified prior to and during system
tests. As we have previously reported, having current and accurate
defect information is necessary to adequately understand system
maturity and to make informed decisions about how to best allocate
limited resources to meet competing priorities. [Footnote 11]
The contractor that is providing DBTS identified defects during its
work to modify the system to meet OPM‘s requirements. Each defect is
documented and assigned a unique identifier, status, and priority.
According to the contractor, defects are categorized according to the
following priority categories:
* Urgent priority defects prevent progress of the solution in the
current system phase.
* High priority defects are items that need to be addressed in the
current system phase.
* Medium priority defects are items to be addressed in an upcoming
system phase.
* Low priority defects are items that can be addressed during
transition to deployment.
Our analysis of the contractor-identified defects showed a pattern of
defect identification that is consistent with the axiom that defects
are often identified prior to and during system tests. Because DBTS
functionality remains to be developed, integrated, and tested, the
pattern of defect identification is likely to continue. Further, the
RSM executive director stated that OPM expects the number of defects
identified to reach a peak in December 2007.
Figure 3 shows the total number of defects that the DBTS contractor
identified on a weekly basis.
Figure 3: New Defects Identified per Week:
[See PDF for image]
This figure is a multiple line graph depicting the number of new
defects identified per week. The vertical axis of the graph represents
number of defects from 0 to 200. The horizontal axis of the graph
represents weekly dates from January 6, 2007 to October 19, 2007. Lines
represent defects in three categories: high, urgent, and total. Also
depicted are the following start dates, with defect totals approximated
from the graph:
UAT1 Start:
Date: 3/13/2007;
High: less than 5;
Urgent: less than 5;
Total: less than 10.
UAT2 Start:
Date: 5/21/2007;
High: approximately 30;
Urgent: approximately 20;
Total: approximately 70;
UAT3 Start:
Date: 8/6/2007;
High: approximately 25;
Urgent: approximately 10;
Total: approximately 45;
UAT4 Start:
Date: 10/15/2007;
High: approximately 20;
Urgent: approximately 10;
Total: approximately 30.
Source: GAO analysis of OPM data.
[End of figure]
At the end of October 2007, a total of 367 defects remained open. Nine
of these defects were assigned urgent priority and 129 were high
priority. Among these defects, one identified as urgent related to a
system weakness that could result in the generation of different
identifications for the same Social Security number. A high priority
defect was identified as a result of DBTS indicating that a user
already had an active session when logging back in to the system hours
after having logged out.
Figure 4 shows the cumulative number of defects that remained open at
the end of each week.
Figure 4: Cumulative Total, Urgent, and High Open Defects:
[See PDF for image]
This figure is a multiple line graph depicting the cumulative total of
urgent, and high open defects. The vertical axis of the graph
represents number of defects from 0 to 400. The horizontal axis of the
graph represents weekly dates from January 6, 2007 to October 19, 2007.
Line represent defects in three categories: high, urgent, and total.
The cumulative totals, as approximated from the graph at the end of the
tracking on 10/19/2007 appear to be as follows:
High: approximately 20;
Urgent: approximately 100;
Total: approximately 350.
Source: GAO analysis of OPM data.
[End of figure]
The increasing numbers of defects that remain open (shown in figure 4)
indicate a growing backlog of defects. Because two additional user
acceptance tests as well as the integrated product test, parallel test,
and performance/business capability test remain to be conducted,
additional defects could be identified. Until defect trends indicate
resolution of the backlog of urgent and high priority defects, which by
definition are to be resolved prior to deployment, OPM faces increased
risk that it will not have sufficient time to resolve significant
problems before its planned February 2008 deployment.
Objective 1: Independent Verification and Validation:
OPM recently engaged an IV&V contractor, which could result in
improvements. The purpose of independent verification and validation is
to provide an independent review of system processes and products to
ensure that quality standards are being met. As we have previously
reported, the use of IV&V is a recognized best practice for large and
complex system development and acquisition projects such as RSM and
involves an independent organization conducting unbiased reviews of
processes, products, and results to verify and validate that they meet
stated requirements and standards. [Footnote 12]
OPM‘s February 2007 RSM IV&V Approach emphasized the importance of IV&V
for identifying and correcting problems as well as for providing
visibility into the system in order to deliver it on schedule and
within budget.
In August 2007, OPM awarded a contract for IV&V of the RSM program.
According to the contract statement of work, the scope of the
contractor‘s work includes reviewing project approaches, plans,
analyses, methods, processes, and deliverables. Further, according to
the November 2007 IV&V plan, the contractor is expected to recommend
approaches for resolving issues regarding design, development, testing,
and any potential problem area.
As a result of instituting IV&V, OPM should be better positioned to
identify and correct deficiencies in the RSM program and deploy
technology that performs as expected.
Objective 2: Risks, Cost, and Progress:
OPM Has Identified Program Risks but Has Not Reliably Estimated RSM‘s
Cost or Reported Progress.
OPM has established and used a risk management process to identify RSM
program risks. However, the agency‘s current RSM life-cycle cost
estimate of $421.6 million is not supported by necessary documentation
and is thus of questionable reliability. OPM‘s two methods of reporting
program progress, by reporting achievement of goals and using EVM,
provide a favorable view of progress. However, the agency‘s EVM
reporting was unreliable and neither of the two progress reporting
methods reflected OPM‘s decision to delay deployment of a portion of
the technology originally planned for February 2008.
Objective 2: Risks:
OPM has established a risk management process and identified risks.Risk
management is vital to the success of a program such as RSM. Relevant
best practice guidance advocates proactively identifying facts and
circumstances that can increase the probability of a program failing to
meet cost, schedule, and performance commitments and then taking steps
to reduce the probability of their occurrence and impact. [Footnote 13]
OPM has established a risk management program that includes, among
other things, written policies and procedures, roles and
responsibilities, and guidance for identifying, prioritizing and
mitigating risks. Additionally, the agency has implemented the program
and uses a database to help track risks. Risks are summarized and
reported at monthly program management review meetings.
At the October 2007 monthly program management review, OPM reported the
following risks:
* A clear action plan for work related to deployment of increment 1
must be defined, executed, and tracked.
* Training employees for increment 1 must be developed and executed to
ensure the transition.
* A significant volume of testing must be executed in a short time with
constrained resources.??Significant planning and coordination for
preparing records for increment 1 is required.
* Interfaces and legacy system modifications required for increment 1
must be designed, built, and tested in a limited time frame.
As a result of identifying these and other risks, OPM should be
positioned to reduce the probability of their occurrence and to reduce
the impact if the risks do occur.
Objective 2: Cost:
OPM has estimated RSM cost, but reliability of the estimate is
questionable.
A cost estimate is the summation of individual program cost elements,
using established methods and valid data to estimate future costs. The
establishment of a reliable cost estimate is important for developing a
program budget and having a sound basis for measuring performance,
including comparing the actual and planned costs of program activities.
Credible cost estimates are produced by following rigorous steps and
are accompanied by detailed documentation, including descriptions of
the system under development, estimation methodology, ground rules and
assumptions, and sensitivity and uncertainty analyses. [Footnote 14]
In August 2007, OPM revised its RSM life-cycle cost estimate from
$371.2 million to $421.6 million based in part on the estimated costs
of major program activities such as contracting for the capture and
conversion of paper files, development of the defined benefits
technology solution, and program support.
However, OPM‘s revised estimate was not supported by the documentation
that is fundamental to a reliable cost estimate.
Specifically, OPM did not document:
* a technical baseline description;
* a cost estimating methodology;
* ground rules and assumptions, and;
* sensitivity and uncertainty analyses.
OPM officials asserted that the revised cost estimate is adequately
supported by the firm fixed priced contracts upon which the estimate is
largely based. However, while the RSM contracts partially support the
cost estimate, they do not provide sufficient documentation of the
technical baseline description, estimation methodology, ground rules
and assumptions, and sensitivity and uncertainty analyses. Without such
supporting documentation, the reliability of the RSM cost estimate is
questionable.
Without a reliable cost estimate, OPM does not have a sound basis for
formulating future RSM program budgets or for developing the program-
level baseline that is necessary for measuring and predicting program
performance.
Objective 2: Progress:
OPM reported that it met its fiscal year 2007 RSM program progress
goals, but progress reporting using EVM was unreliable.
OPM reported two views of RSM progress. Specifically, it described
satisfaction of established program goals and used EVM as a progress
measurement and reporting tool.
With respect to goals, OPM established five fiscal year 2007 goals to
support development and testing of the RSM program components necessary
to deploying initial functionality in February 2008. As of October
2007, OPM reported that it met these goals.
Table 4: RSM FY 2007 Program Goals:
Operational Goal: Make data element dictionary available for Government
Shared Service Centers;
Date Completed: January 2007;
Description: Published updated list of electronic human resources
information needed for federal agencies to send records to OPM.
Operational Goal: Complete development of licensed technology –
employee/client application;
Date Completed: April 2007;
Description: Delivered licensed technology solution to OPM for testing
and development of interfaces.
Operational Goal: Begin RSM training;
Date Completed: May 2007;
Description: Finalized training approach and started courses.
Operational Goal: Develop licensed technology for GSA active employees;
Date Completed: July 2007;
Description: Delivered increment 1 data for the licensed technology
solution.
Operational Goal: Complete Active Employee Folder imaging in the
Retirement Operations Center;
Date Completed: September 2007;
Description: Captured paper records stored at the Retirement Operations
Center.
[End of table]
OPM also uses EVM to measure and report program progress. EVM is a tool
for measuring program progress by comparing the value of work
accomplished with the amount of work expected to be accomplished. Such
a comparison permits actual performance to be evaluated, based on
variances from the planned cost and schedule, and future performance to
be forecasted. Identification of significant variances, which OPM
defines as plus or minus 10 percent from planned cost and schedule, and
analysis of their causes helps program managers determine the need for
corrective actions.
Fundamental to reliable EVM is the development of a performance
measurement baseline (PMB), which represents the cumulative value of
planned work and is the baseline against which variances are
calculated. To establish a meaningful PMB, programs must fully:
* define the work in a work breakdown structure;
* develop a complete integrated master schedule, and;
* formulate budgets for all planned work. [Footnote 15]
After these inputs are integrated to develop the PMB, it should be
validated through an integrated baseline review during which
stakeholders reach agreement on the baseline. Once validated, the PMB
is closely controlled and generally not subject to change unless events
beyond the program manager‘s control occur (e.g. changes to program
scope).
OPM used EVM to measure and report monthly performance. The agency‘s
October 2007 monthly report identified a cumulative actual cost of
$110.1 million for the work performed on RSM through September 2007.
OPM‘s comparison of this actual cost with the budgeted cost resulted in
a positive cost variance of about $0.5 million or less than 1 percent
(i.e., the program created less than 1 percent more value than planned
for the money spent) and a negative schedule variance of $1.3 million
or about 1 percent (i.e., the program created about 1 percent less
value than planned for the time spent).
These reported results provided a favorable view of program performance
over time because the variances indicated the program was progressing
almost exactly as planned. Further, OPM‘s earned value reporting
indicated that such favorable performance had been sustained as shown
by the agency‘s reporting of cumulative variances less than plus or
minus 2 percent since September 2006.
However, this view of program performance is not reliable because it is
not consistent with established EVM practices:
* The baseline upon which the results were based was derived from a
work breakdown structure and an integrated master schedule that did not
reflect the full scope of the program. For example, the full scope of
system integration activities such as legacy system modification and
the development of interfaces critical to deploying initial
functionality in February 2008 was not included.
* An integrated baseline review had not been conducted to validate the
baseline.
* The baseline was unstable. For example, the baseline used to measure
performance was $137 million in September 2006 and changed ten times
between then and the $175.1 million baseline reported in September
2007.
Further, OPM‘s recently reported EVM-based view of favorable program
progress was not consistent with the state of the program.
Specifically, in September 2007, the agency decided to delay deployment
of new retirement system technology for already retired federal
employees, originally planned for February 2008, to August 2008 because
the agency did not have sufficient time and resources to modify its
legacy systems and develop interfaces necessary to fully support the
originally planned deployment.
OPM officials acknowledged that their approach to instituting EVM
reflected that they did not have a level of confidence in their
definition of the program scope that was sufficient to establish a PMB
that could be validated in an integrated baseline review. In September
2007, the agency reported that it had established a PMB that it plans
to validate in an integrated baseline review in December 2007. At the
time we concluded our review in November 2007, the agency had not
provided a PMB for our analysis.
This EVM approach, whereby OPM frequently revised its baseline in lieu
of establishing and controlling a valid PMB, in effect ensured that
material program level variances from planned performance over time
(i.e., above the 10 percent threshold) would not be identified and that
the state of the program would not be reliably reported.
Conclusions:
To its credit, OPM has undertaken the RSM program to expedite
retirement processing for civilian federal employees and the agency
reported that it has met key program goals. Further, OPM has improved
its management processes for selecting contractors, defining system and
security requirements, managing risks, planning organizational change,
and providing program executive oversight. Nevertheless, much remains
to be accomplished before the program is effectively positioned to
deploy its first planned increment of new technology in February 2008.
OPM developed performance targets necessary to gauge the success of the
new system with respect to the major objectives of increasing the
timeliness and accuracy of retirement claims processing. The agency
also recognized the value that an independent review of the RSM program
could provide and entered into a contract to obtain such a review,
which should help the agency identify and address weaknesses in its
management of the RSM program. However, the agency‘s program management
has not ensured that system components will perform as intended. In
particular, initial test results indicate that the defined benefits
technology solution that is a major component of the new system has not
performed as intended and future system tests are to be conducted
concurrently in about half the time originally planned. Compounding
this already risky scenario, the contractor that is providing the
defined benefits technology solution continues to identify system
defects faster than they can be resolved, thus building a backlog of
defects that will need to be resolved and tested within the same time
period that OPM is concurrently conducting important system tests.
OPM recognized the importance of risk management and has established a
risk management process and identified program risks. However, the
agency has not yet developed the capability to reliably analyze and
report RSM progress. Such progress reporting should be grounded in a
reliable cost estimate that is in part the basis for reliable earned
value management. Without a reliable cost estimate, OPM does not have a
firm foundation for the RSM program budget or for reliable earned value
management reporting.
Until OPM makes improvements to the RSM program in the areas discussed
above, the agency risks not achieving successful program outcomes,
including the planned deployment of new technology beginning in
February 2008.
GAO Recommendations:
To address the risks to OPM‘s deployment of new retirement system
technology and improve the agency‘s ability to reliably report progress
of the RSM program, we are recommending that the Director of the Office
of Personnel Management direct the RSM Executive Director to take the
following actions:
* Ensure that sufficient resources are provided to fully test
functionality, actions for mitigating the risks inherent in concurrent
testing are identified, test results verify that all system components
perform as expected, and test activities and results are subjected to
independent verification and validation.
* Monitor and review DBTS defects to ensure all urgent and high
priority defects are resolved prior to system deployment and that the
resolution of urgent and high priority defects is subjected to
independent verification and validation.
* Develop a revised RSM cost estimate that addresses the weaknesses
identified in this briefing and task an independent verification and
validation contractor with reviewing the process used to develop the
estimate and assessing the reliability of the resulting estimate.
* Establish a basis for effective use of earned value management by
validating the RSM performance measurement baseline through a program
level integrated baseline review and task an independent verification
and validation contractor with reviewing the process used to develop
the baseline and assessing the reliability of the performance
measurement baseline.
Agency Comments and Our Evaluation:
In oral comments on a draft of this briefing, OPM officials including
the Director and the RSM Executive Director generally agreed with our
recommendations and provided additional information and written
technical comments related to program activities, which we incorporated
in the briefing as appropriate.
In the comments, the Director stated that the agency:
* had contracted in October 2007 for additional resources and expertise
to help modify and test DBTS;
* continues to actively track the identification and resolution of
system defects and is in the process of determining which open defects
must be resolved prior to deployment of new retirement system
technology in February 2008; and;
* plans to review guidance on preparing reliable cost estimates.
[End of section]
Appendix II: Comments from the Office of Personnel Management:
The Director:
United States Office Of Personnel Management:
Washington, DC 20415:
[hyperlink, http://www.opm.gov]:
[hyperlink, http://www.usajobs.gov]:
Our mission is to ensure the Federal Government has an effective
civilian workforce.
January 15, 2008:
The Honorable David M. Walker:
Comptroller General:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Mr. Walker:
Thank you for the opportunity to provide comments in response to the
Government Accountability Office's (GAO) draft report entitled Office
of Personnel Management: Improvements Needed to Ensure Successful
Retirement Systems Modernization (GAO-08-345SU). The Office of
Personnel Management (OPM) appreciates GAO's recognition of the process
improvements made for managing and executing the agency's Retirement
Systems Modernization (RSM) program. We also acknowledge the importance
of the four recommendations outlined in the draft report and are
already taking steps to address them.
Developing and implementing a retirement system that provides Federal
employees and annuitants with a modem and reliable information
technology platform is one of my top priorities. This historic
modernization is one of the largest projects of its type and is long
overdue. Unlike past initiatives to address this major deficiency in a
core OPM function, this project will have a successful outcome. When
fully live, retirement administration processes resulting from the
Retirement Systems Modernization effort will provide high quality
customer service, enhanced modeling and planning tools, and prompt,
complete payment on annuity commencement.
To accomplish our objectives, OPM has invested a tremendous amount of
time, energy, and resources to enable deployment of the new system for
the first wave of employees in late February 2008. Project management
includes the highest level of agency executives. Such accepted
practices as Earned Value Management and monthly performance management
review meetings where status, risks, and other issues of the
modernization effort are addressed have been in place since project
inception. We also dedicated additional resources with subject matter
expertise in the Defined Benefit Technology Solution to monitor,
evaluate, and troubleshoot building and testing of the solution. OPM
has reviewed and verified the components of the existing cost estimate
and conducted sensitivity analysis on the resulting program costs. We
believe our comprehensive strategy, which includes these and other
actions, will address GAO's concerns outlined in the draft report and
thereby enable the system components to perform as intended.
I am also pleased to report that the results of our latest User
Acceptance Test (UAT 5) yielded positive trends for identifying and
resolving system defects. Specifically, the results of UAT 5 indicate
that the backlog of urgent and high priority defects has been reduced
from UAT 4 levels.
This supports the anticipated pattern we communicated to GAO, namely
that testing scores will improve while the defect rate declines as we
approach the February "Go Live" milestone.
We appreciate GAO's insightful recommendations outlined in the draft
report. OPM is taking the necessary steps to ensure that the Federal
Government has a state-of-the-art retirement system for annuitants and
employees. We look forward to giving GAO an update after "Go Live" for
the first wave of employees.
Sincerely,
Signed by:
Linda M. Springer:
Director:
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Valerie C. Melvin (202) 512-6304 or melvinv@gao.gov:
Staff Acknowledgments:
In addition to the contact named above, key contributions to this
report were made by Mark T. Bird, Assistant Director; Neil J. Doherty;
David A. Hong; Jacqueline K. Mai; Teresa M. Neven; B. Scott Pettis;
Margaret E. Poston; and Amos A. Tevelow.
Footnotes:
[1] EVM is a management tool used for measuring program performance by
comparing the value of work accomplished with the amount of work
expected to be accomplished. Such a comparison permits performance to
be evaluated based on variances from the planned cost and schedule.
[2] EVM is a management tool used for measuring program performance by
comparing the value of work accomplished with the amount of work
expected to be accomplished. Such a comparison permits performance to
be evaluated based on variances from the planned cost and schedule.
[3] Office of Personnel Management, OPM Strategic and Operational Plan
2006-2010, (Washington, D.C.: March 2006).
[4] Defined benefit plans calculate benefit amounts in advance of
retirement based on factors such as salary level and years of service
and defined contribution plans calculate benefit amounts based on how
the amount is invested by the employee and employer.
[5] The Social Security Administration is responsible for administering
Social Security and the Federal Thrift Investment Board administers the
defined contribution system known as the Thrift Savings Plan.
[6] GAO, Office of Personnel Management: Retirement Systems
Modernization Faces Numerous Challenges, GAO-05-237(Washington, D.C.:
February 28, 2005).
[7] GAO, Performance Measurement and Evaluation, GAO-05-739SP
(Washington, D.C.: May 2005).
[8] GAO,Year 2000 Computing Crisis: A Testing Guide, GAO/AIMD-10.1.21
(Washington, D.C.: November 1998); Information Technology: Customs
Automated Commercial Environment Progressing, but Need for Management
Improvements Continues, GAO-05-267 (Washington, D.C.: March 14, 2005);
and Homeland Security: Visitor and Immigrant Status Program Operating,
but Management Improvements Are Still Needed, GAO-06-318T (Washington,
D.C.: January 25, 2006).
[9] GAO/AIMD-10.1.21.
[10] GAO,2000 Census: New Data Capture System Progress and Risks,
GAO/AIMD-00-61 (Washington, D.C.: February 4, 2000).
[11] GAO,Customs Service Modernization: Automated Commercial
Environment Progressing, but Further Acquisition Management
Improvements Needed, GAO-03-406 (Washington, D.C.: February 28, 2003);
Homeland Security: Visitor and Immigrant Status Program Operating, but
Management Improvements Are Still Needed, GAO-06-318T (Washington,
D.C.: January 25, 2006); and GAO/AIMD-00-61.
[12] GAO,U.S. Customs Service: Observations on Selected Operations and
Program Issues, GAO-01-968T (Washington, D.C.: July 17, 2001) and
Homeland Security: First Phase of Visitor and Immigration Status
Program Operating, but Improvements Needed, GAO-04-586 (Washington,
D.C.: May 11, 2004).
[13] Software Engineering Institute, Software Acquisition Capability
Maturity Model® version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA:
March 2002).
[14] GAO, Cost Assessment Guide: Best Practices for Estimating and
Managing Program Costs, Exposure Draft, GAO-07-1134SP (Washington,
D.C.: July 2007).
[15] GAO-07-1134SP.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "Subscribe to Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Gloria Jarmon, Managing Director, JarmonG@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
