Comments on the Office of Personnel Management's February 20, 2008 Report to Congress Regarding the Retirement Systems Modernization
Gao ID: GAO-08-576R March 28, 2008The Office of Personnel Management (OPM) is modernizing the paper-intensive processes and antiquated information systems it uses to support the retirement of civilian federal employees through the Retirement Systems Modernization (RSM) program. In January 2008, we reported on the agency's management of this program, in which we noted concerns and made recommendations for improvement in four key areas: (1) system testing, (2) system defect resolution, (3) program cost estimation, and (4) program earned value management. The explanatory statement of the House Appropriations Committee regarding the fiscal year 2008 Consolidated Appropriations Act directed OPM to submit to Cpngressional Committees and to GAO not later than February 20, 2008, a report of its actions on the four areas of concern that we identified. Further, the explanatory statement directed that GAO provide to Congressional Committees and to OPM our comments on the agency's report.
Our study determined that OPM initial RSM system test results did not provide assurance that a major system component--the Defined Benefits Technology Solution (DBTS)--would perform as intended, and that the agency's compressed and concurrent testing schedule increased the risk that it would not have sufficient resources or time to verify that the system will work as intended. We recommended that OPM ensure that sufficient resources are provided to fully test functionality, actions for mitigating the risks inherent in concurrent testing are identified, test results verify that all system components perform as expected, and test activities and results are subjected to independent verification and validation. OPM's report did not address the results of other critical tests that the agency had planned to conduct starting in December 2007 and ending in February 2008. Specifically, the report did not discuss the results of the following tests: (1) parallel test to verify that the new system produces the same results as existing systems; (2) integrated product test to confirm that system components meet functional requirements; (3) performance test to confirm that the new system meets performance requirements (e.g., processing volume and execution time); (4) business capability test to confirm the operational readiness of the new system for end users. Our study of RSM determined that trends in identifying and resolving system defects indicated a growing backlog of problems to be resolved prior to deployment of the new system. We recommended that the agency monitor and review DBTS defects to ensure that all urgent and high priority defects are resolved prior to system deployment and that the resolution of urgent and high priority defects is subjected to independent verification and validation. OPM reported progress toward resolving defects between October 2007 and January 2008. However, the agency did not report actual defect resolution data for February 2008; instead it reported the projected defects it expected to remain at the time of system deployment. Our study determined that the reliability of OPM's $421.6 million RSM life-cycle cost estimate was questionable because the agency could not support the estimate with a description of the system to be developed and a description of the methodology used to produce the estimate. We recommended that the agency develop a revised RSM cost estimate that addresses the weaknesses identified and task an independent verification and validation contractor with reviewing the process used to develop the estimate and assessing the reliability of the resulting estimate. The report did not provide new information or describe the progress the agency has made to address the weaknesses in the RSM cost estimate. OPM's report did not address conducting independent verification and validation of its cost estimate and the process used to develop the estimate. Our study determined that OPM's reporting of program progress using Earned Value Management (EVM) was unreliable because the agency did not establish and validate a meaningful performance measurement baseline. We recommended that the agency establish a basis for effective use of earned value management by validating the RSM performance measurement baseline through a program level integrated baseline review, and that it task an independent verification and validation contractor with reviewing the process used to develop the baseline and assessing the reliability of the performance measurement baseline. OPM's report did not provide new information or describe the progress the agency has made toward addressing the three specific weaknesses that we identified in its use of EVM. Without addressing these weaknesses, OPM's use of EVM will continue to be unreliable.
GAO-08-576R, Comments on the Office of Personnel Management's February 20, 2008 Report to Congress Regarding the Retirement Systems Modernization
This is the accessible text file for GAO report number GAO-08-576R
entitled 'Comments on the Office of Personnel Management's February 20,
2008 Report to Congress Regarding the Retirement Systems
Modernization,' which was released on March 28, 2008.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
GAO-08-576R:
United States Government Accountability Office:
Washington, DC 20548:
March 28, 2008:
The Honorable Richard Durbin:
Chairman:
The Honorable Sam Brownback:
Ranking Member:
Subcommittee on Financial Services and General Government:
Committee on Appropriations:
United States Senate:
The Honorable José E. Serrano:
Chairman:
The Honorable Ralph Regula:
Ranking Member:
Subcommittee on Financial Services and General Government:
Committee on Appropriations:
House of Representatives:
Subject: Comments on the Office of Personnel Management's February 20,
2008 Report to Congress Regarding the Retirement Systems Modernization:
The Office of Personnel Management (OPM) is modernizing the paper-
intensive processes and antiquated information systems it uses to
support the retirement of civilian federal employees through the
Retirement Systems Modernization (RSM) program. In January 2008, we
reported on the agency's management of this program, in which we noted
concerns and made recommendations for improvement in four key areas:
(1) system testing, (2) system defect resolution, (3) program cost
estimation, and (4) program earned value management.[Footnote 1]
The explanatory statement of the House Appropriations Committee
regarding the fiscal year 2008 Consolidated Appropriations Act directed
OPM to submit to your Committees and to GAO not later than February 20,
2008, a report of its actions on the four areas of concern that we
identified.[Footnote 2] Further, the explanatory statement directed
that GAO provide to your Committees and to OPM our comments on the
agency's report. In response to this direction, our comments (including
a summarization of our report findings and recommendations and OPM's
reported actions for each area of concern) follow.
System Testing:
GAO Finding and Recommendation: Effective testing is an essential
component of any program that includes system development. Generally,
the purpose of testing is to identify defects or problems in meeting
defined system requirements or satisfying system user needs[Footnote
3]. Our study determined that OPM's initial RSM system test results did
not provide assurance that a major system component--the Defined
Benefits Technology Solution (DBTS)--would perform as intended, and
that the agency's compressed and concurrent testing schedule increased
the risk that it would not have sufficient resources or time to verify
that the system will work as intended. Accordingly, we recommended that
OPM ensure that sufficient resources are provided to fully test
functionality, actions for mitigating the risks inherent in concurrent
testing are identified, test results verify that all system components
perform as expected, and test activities and results are subjected to
independent verification and validation.
OPM Report to Congress: OPM's report summarized the results of certain
user acceptance tests (UAT) that the agency has conducted as part of
verifying that DBTS meets requirements such as accurately calculating
retirement benefits. The agency reported test results as shown in table
1.
Table 1: OPM Reported Test Results in Terms of Scenarios:
Scenarios Tested:
Scenarios Passed:
Percent Passed:
Test: UAT 5;
Scenarios Tested: 192;
Scenarios Passed: 142;
Percent Passed: 74.
Test: UAT 6;
"Discontinued and reconstituted as UAT 6.5 to limit focus to
high frequency functionality for Go Live".
Test: UAT 6.5;
Scenarios Tested: 57;
Scenarios Passed: 51;
Percent Passed: 89.
Source: OPM.
[End of table]
GAO Comments: OPM limited its discussion of system testing to the UAT
results. In this regard, the results shown for UAT 5, indicating that
74 percent of scenarios had passed testing, falls short of the agency's
stated goal of 95 percent of scenarios passed. In addition, the
statement that UAT 6 was "discontinued and reconstituted as UAT 6.5 to
limit focus to high frequency functionality for Go Live" indicates that
the scope of UAT 6.5 was reduced from the scope planned for UAT 6.
Further, OPM's report did not address the results of other critical
tests that the agency had planned to conduct starting in December 2007
and ending in February 2008 as illustrated in figure 1.
Figure 1: RSM Test Schedule:
[See PDF for image]
This figure is a timeline of the RSM test schedule. The following items
are displayed on the timeline:
Test: UAT 1:
Date: Mid-March 2007.
Test: UAT 2 part 1;
Date: Late May 2007.
Test: UAT 2 part 2;
Date: Early June 2007.
Test: UAT 3 part 1;
Date: Early August 2007.
Test: UAT 3 part 2;
Date: Mid-September 2007.
Test: UAT 4;
Date: Mid-October 2007.
Test: UAT 5;
Date: Mid-December 2007;
Test: Parallel test;
Date: Mid-to-late December 2007.
Test: UAT 6;
Date: Mid-January 2008.
Test: Parallel test;
Date: Mid-January to Mid-February 2008.
Test: Integrated product test;
Date: Late January to Mid-February 2008.
Test: Performance test;
Date: Late January to Mid-February 2008.
Test: Business capacity release test;
Date: February 2008.
Increment 1 deployment:
Date: End of February 2008.
Source: GAO, based on OPM data.
[End of figure]
Specifically, the report did not discuss the results of the following
tests:
* Parallel test to verify that the new system produces the same results
as existing systems.
* Integrated product test to confirm that system components meet
functional requirements (e.g., accurately calculate benefits).
* Performance test to confirm that the new system meets performance
requirements (e.g., processing volume and execution time).
* Business capability test to confirm the operational readiness of the
new system for end users.
As our report noted, these tests are intended to verify that DBTS and
other system components work together as expected when they are
combined and that the complete system resulting from the RSM program
satisfies all requirements (e.g., functional and performance) and is
acceptable to end users.
Finally, the report did not address the conduct of independent
verification and validation of test activities and results. The purpose
of independent verification and validation is to provide an independent
review of system processes and products to ensure that quality
standards are being met.
System Defect Resolution:
GAO Finding and Recommendation: Defects are system problems that
require a resolution and can be due to a failure to meet the system
specifications. Defects are often identified prior to and during system
tests. As we have previously reported, having current and accurate
defect information is necessary to adequately understand system
maturity and to make informed decisions about how to best allocate
limited resources to meet competing priorities.[Footnote 4] Our study
of RSM determined that trends in identifying and resolving system
defects indicated a growing backlog of problems to be resolved prior to
deployment of the new system. These include urgent and high priority
defects, which by OPM's definition are defects that either prevent
progress or defects that need to be addressed in the current system
phase prior to deployment. As a result, we recommended that the agency
monitor and review DBTS defects to ensure that all urgent and high
priority defects are resolved prior to system deployment and that the
resolution of urgent and high priority defects is subjected to
independent verification and validation.
OPM Report to Congress: OPM's report summarized system defect
resolution as shown in table 2.
Table 2: OPM Reported Defects Status:
Defect Severity Priority: Urgent;
Number of defects, October 2007: 9;
Number of defects, January 2008: 14;
Projected number of defects at February 2008 deployment: 1.
Defect Severity Priority: High;
Number of defects, October 2007: 129;
Number of defects, January 2008: 70;
Projected number of defects at February 2008 deployment: 12.
Defect Severity Priority: Medium/low;
Number of defects, October 2007: 229;
Number of defects, January 2008: 85;
Projected number of defects at February 2008 deployment: 2.
Defect Severity Priority: Total;
Number of defects, October 2007: 367;
Number of defects, January 2008: 168;
Projected number of defects at February 2008 deployment: 15.
Source: OPM.
[End of table]
In its report, OPM stated that the decreasing pattern of defects was
consistent with the agency's expectation for the identification and
resolution of defects. The report also stated that the one urgent
priority defect that was projected to remain at the time of the
February 2008 deployment was related to functionality referred to as
"unpaid re-deposit" and is expected to be infrequent and easily
identifiable.
GAO Comments: With the exception of an increase in urgent priority
defects from 9 to 14, OPM reported progress toward resolving defects
between October 2007 and January 2008. However, the agency did not
report actual defect resolution data for February 2008; instead it
reported the projected defects it expected to remain at the time of
system deployment, scheduled for February 25, 2008. If this projection
was realized, it would represent continued defect resolution progress.
Nevertheless, OPM's plan to deploy its system with a total of 13
unresolved urgent and high priority system defects increased risk and
contradicted earlier assertions that the agency would resolve urgent
and high priority defects prior to deployment. In addition, the report
did not address conducting independent verification and validation of
system defect resolution results.
Cost Estimating:
GAO Finding and Recommendation: A cost estimate is the summation of
individual program cost elements, using established methods and valid
data to estimate future costs. Credible cost estimates are produced by
following rigorous steps and are accompanied by detailed documentation,
including descriptions of the system under development, estimation
methodology, ground rules and assumptions, and sensitivity and
uncertainty analyses.[Footnote 5] Our study determined that the
reliability of OPM's $421.6 million RSM life-cycle cost estimate was
questionable because the agency could not support the estimate with a
description of the system to be developed and a description of the
methodology used to produce the estimate. Thus, we recommended that the
agency develop a revised RSM cost estimate that addresses the
weaknesses identified and task an independent verification and
validation contractor with reviewing the process used to develop the
estimate and assessing the reliability of the resulting estimate.
OPM Report to Congress: Relative to this area of concern, OPM reported
that 86 percent of program costs are associated with fixed price
contracts and that remaining costs are predominantly for OPM staff and
project management activities. Further, the agency asserted that most
future program costs are known and do not require cost estimating.
Additionally, the agency provided information on factors that
contributed to an increase in the RSM life-cycle cost estimate from
$371.2 million to $421.6 million.
GAO Comments: The report did not provide new information or describe
the progress the agency has made to address the weaknesses in the RSM
cost estimate. Specifically, the agency did not provide documentation
about the system under development (i.e., a technical baseline
description), a cost estimating methodology, estimating ground rules
and assumptions, and sensitivity and uncertainty analyses. Without such
documentation, an assessment of the reliability of OPM's estimate
cannot be made. Further, the extent to which OPM's estimate reflects
changes to the program, including delayed deployment of certain
functionality from February 2008 to August 2008, is not readily
apparent. Finally, OPM's report did not address conducting independent
verification and validation of its cost estimate and the process used
to develop the estimate.
Earned Value Management:
GAO Finding and Recommendation: Earned value management (EVM) is a tool
for measuring program progress by comparing the value of work
accomplished with the amount of work expected to be accomplished. Such
a comparison permits actual performance to be evaluated, based on
variances from the planned cost and schedule, and future performance to
be forecasted. Our study determined that OPM's reporting of program
progress using EVM was unreliable because the agency did not establish
and validate a meaningful performance measurement baseline.
Accordingly, we recommended that the agency establish a basis for
effective use of earned value management by validating the RSM
performance measurement baseline through a program level integrated
baseline review, and that it task an independent verification and
validation contractor with reviewing the process used to develop the
baseline and assessing the reliability of the performance measurement
baseline.
OPM Report to Congress: OPM stated in its report that the agency had
established a performance measurement baseline, and had implemented EVM
practices. Specifically, the agency stated that it had measured and
reported performance on a monthly basis, used an agency-standard EVM
tool set, and worked with its contractors to improve EVM reporting.
Further, OPM characterized GAO's concerns about its use of EVM as being
related to the proper inclusion of work to be performed by government
personnel and of work related to a delay in deployment of planned
functionality.
GAO Comments: OPM's report did not provide new information or describe
the progress the agency has made toward addressing the three specific
weaknesses that we identified in its use of EVM. First, the report did
not state whether OPM has established a performance measurement
baseline that reflects the full scope of the RSM program. Second, the
report did not say whether the agency has validated its performance
measurement baseline in an integrated baseline review. Lastly, the
report did not address whether the baseline against which the agency
has measured and reported has been stabilized. In addition to these
three weaknesses, the report also did not address conducting
independent verification and validation of the performance measurement
baseline and the process used to develop the baseline. Without
addressing these weaknesses, OPM's use of EVM will continue to be
unreliable.
Agency Comments:
In written comments on a draft of this letter, the Director of OPM
stated that the agency successfully initiated rollout of its new
retirement system to the approximately 26,000 employees in agencies
serviced by the General Services Administration payroll processing
center on February 25, 2008. Further, the Director stated that OPM's
forthcoming formal response to GAO's January 2008 report on RSM will
provide detailed information on the agency's progress in developing and
implementing the new system. Finally, the Director commented that the
short time between its February 20, 2008 report to Congress and system
deployment necessitated that the report be brief and that its contents
(which are summarized in this letter) be limited to official use only.
However, in subsequent comments e-mailed to us, OPM's liaison to GAO
stated that the report to Congress did not contain sensitive
information. The agency's comments are reprinted in the enclosure.
We are sending copies of this letter to the Director of the Office of
Personnel Management and other appropriate congressional committees. We
will make copies available to other interested parties upon request and
will made it available at no charge on GAO's Web site at [hyperlink,
http://www.gao.gov].
If you or your staff have any questions about this correspondence,
please contact me at (202) 512-6304 or melvinv@gao.gov. Key
contributors to this letter were Mark T. Bird, Rebecca E. LaPaze, and
Teresa M. Neven.
Signed by:
Valerie C. Melvin:
Director, Human Capital and Management:
Information Systems Issues:
[End of correspondence]
Enclosure:
Comments from the Office of Personnel Management:
The Director:
United States Office Of Personnel Management:
Washington, DC 20415:
[hyperlink, http://www.opm.gov]:
[hyperlink, http://www.usaJobs.gov]:
"Our mission is to ensure the Federal Government has an effective
civilian workforce."
March 18, 2008:
Mr. Gene L. Dodaro:
Acting Comptroller General:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Mr. Dodaro:
Thank you for the opportunity to comment on the Government
Accountability Office's (GAO) draft response to the Office of Personnel
Management's (OPM) February 20, 2008, Retirement Systems Modernization
(RSM) Report to Congress.
On February 25, 2008, OPM successfully initiated the rollout of its
landmark strategic program that has modernized the business processes
and supporting technology used to administer the Federal Government's
defined retirement benefit plans. Approximately 26,000 employees in
agencies serviced by the General Services Administration (GSA) payroll
processing center are now eligible to retire under the new system,
which is being called RetireEZ. These agencies include OPM, GSA, the
National Archives and Records Administration, and the Railroad
Retirement Board. Subsequent rollouts will cover the remainder of the
Executive Branch, the U.S. Postal Service and the Legislative and
Judicial Branches.
OPM was tasked with responding to the Congressional request for a
status report just three days before the launch of RetireEZ. The
challenging timing of this request necessitated that our responses were
briefer than we would have preferred. We believe the brevity of this
report supports our position that this information ” while accurate ”
should be limited to official use only. We believe the public deserves
the benefit of a more comprehensive report.
OPM is currently preparing our sixty day response to GAO's report
entitled Office of Personnel Management: Improvements Needed to Ensure
Successful Retirement Systems Modernization (GAO-08-345). This
information, to be submitted to the United States House of
Representatives and Senate Committees on Appropriations no later than
March 31, 2008, will provide greater detail on OPM's progress in
developing and implementing the new system, in addition to addressing
the four recommendations in this particular GAO report.
Thank you again for the opportunity to provide GAO with these comments.
OPM continues to take the necessary steps to ensure that the Federal
Government has a state-of-the-art retirement administration system for
annuitants and employees.
Sincerely,
Signed by:
Linda M. Springer:
Director:
[End of enclosure]
Footnotes:
[1] GAO, Office of Personnel Management: Improvements Needed to Ensure
Successful Retirement Systems Modernization, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-08-345] (Washington,
D.C.: January 31, 2008).
[2] Consolidated Appropriations Act, 2008, P.L. 110-161 (Dec. 26,
2007); 153 Cong. Rec. H15741, H16056 (daily ed. Dec. 17, 2007).
[3] GAO, Year 2000 Computing Crisis: A Testing Guide, GAO/AIMD-10.1.21
(Washington, D.C.: November 1998); Information Technology: Customs
Automated Commercial Environment Progressing, but Need for Management
Improvements Continues, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-05-267]
(Washington, D.C.: March 14, 2005); and Homeland Security: Visitor
and Immigrant Status Program Operating, but Management Improvements
Are Still Needed, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-318T]
(Washington, D.C.: January 25, 2006).
[4] GAO, Customs Service Modernization: Automated Commercial
Environment Progressing, but Further Acquisition Management
Improvements Needed, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-03-406]
(Washington, D.C.: February 28, 2003) and Homeland Security:
Visitor and Immigrant Status Program Operating, but Management
Improvements Are Still Needed, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-06-318T]
(Washington, D.C.: January 25, 2006).
[5] GAO, Cost Assessment Guide: Best Practices for Estimating and
Managing Program Costs, Exposure Draft, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP] (Washington,
D.C.: July 2007).
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
