Veterans Benefits Administration
Progress Made in Long-Term Effort to Replace Benefits Payment System, but Challenges Persist Gao ID: GAO-07-614 April 27, 2007Since 1996, the Veterans Benefit Administration (VBA) has been working on an initiative to replace its aging system for paying compensation and pension benefits. In 2005, concerned about the slow pace of development, VBA contracted with the Software Engineering Institute (SEI) for an independent evaluation of the project, known as the Veterans Service Network (VETSNET). SEI advised VBA to continue working on the project at a reduced pace while addressing management and organization weaknesses that it determined had hampered the project's progress. GAO was requested to determine to what extent the VETSNET project has followed the course of action recommended by SEI and describe the project's current status. To perform its review, GAO analyzed project documentation, conducted site visits, and interviewed key program officials.
VBA is generally following the course of action recommended by SEI by continuing to work on the replacement initiative at a reduced pace, while taking action to address identified weaknesses in overall management and software development processes. For example, VBA established a new governance structure for the initiative that included senior management and involved all stakeholders, and it incorporated all critical areas of system development in an integrated master schedule. However, not all of SEI's management concerns have been addressed. For example, SEI advised VBA to ensure that stakeholders take ownership responsibility for the project, including the total system and process operating costs; however, although VBA is tracking costs incurred by contractors, it is not yet tracking and reporting in-house costs incurred by the project. Further, although the project has improved its management processes, such as establishing a process to manage and stabilize system requirements, it has not yet developed processes for capacity planning and management. This will be important for ensuring that further VETSNET development does not lead to delays and slowdowns in processing of benefits. In addition, although the project has established certain performance measures, it has not yet established results-oriented measures for productivity and user satisfaction, both of which will be important for measuring progress. Finally, the process improvements that VBA has incorporated in the replacement initiative remain in draft and have not been established through documented policies and procedures. If VBA does not institutionalize these improvements, it increases the risk that they may not be maintained through the life of the project or be available for application to other development initiatives. After more than 10 years of effort, including the recent management, organizational, and process improvements, VBA has developed critical functionalities needed to process and pay certain original compensation claims using the replacement system, but it remains far from completing the project. According to VBA officials, all five of the major software applications that make up the new system are now being used to establish and process new compensation claims for veterans. In total, the replacement system is currently providing monthly compensation payments to almost 50,000 veterans (out of about 3 million veterans who receive such payments); the system was used to process about 83 percent of all new compensation claims completed in March 2007. Nonetheless, the system requires further development before it can be used to process claims for the full range of compensation and pension benefits available to veterans and their dependents.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-07-614, Veterans Benefits Administration: Progress Made in Long-Term Effort to Replace Benefits Payment System, but Challenges Persist
This is the accessible text file for GAO report number GAO-07-614
entitled 'Veterans Benefits Administration: Progress Made in Long-Term
Effort to Replace Benefits Payment System, but Challenges Persist'
which was released on April 27, 2007.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Chairman, Committee on Veterans' Affairs, House of
Representatives:
United States Government Accountability Office:
GAO:
April 2007:
Veterans Benefits Administration:
Progress Made in Long-Term Effort to Replace Benefits Payment System,
but Challenges Persist:
GAO-07-614:
GAO Highlights:
Highlights of GAO-07-614, a report to the Chairman, Committee on
Veterans‘ Affairs, House of Representatives
Why GAO Did This Study:
Since 1996, the Veterans Benefit Administration (VBA) has been working
on an initiative to replace its aging system for paying compensation
and pension benefits. In 2005, concerned about the slow pace of
development, VBA contracted with the Software Engineering Institute
(SEI) for an independent evaluation of the project, known as the
Veterans Service Network (VETSNET). SEI advised VBA to continue working
on the project at a reduced pace while addressing management and
organization weaknesses that it determined had hampered the project‘s
progress.
GAO was requested to determine to what extent the VETSNET project has
followed the course of action recommended by SEI and describe the
project‘s current status.
To perform its review, GAO analyzed project documentation, conducted
site visits, and interviewed key program officials.
What GAO Found:
VBA is generally following the course of action recommended by SEI by
continuing to work on the replacement initiative at a reduced pace,
while taking action to address identified weaknesses in overall
management and software development processes. For example, VBA
established a new governance structure for the initiative that included
senior management and involved all stakeholders, and it incorporated
all critical areas of system development in an integrated master
schedule. However, not all of SEI‘s management concerns have been
addressed. For example, SEI advised VBA to ensure that stakeholders
take ownership responsibility for the project, including the total
system and process operating costs; however, although VBA is tracking
costs incurred by contractors, it is not yet tracking and reporting in-
house costs incurred by the project. Further, although the project has
improved its management processes, such as establishing a process to
manage and stabilize system requirements, it has not yet developed
processes for capacity planning and management. This will be important
for ensuring that further VETSNET development does not lead to delays
and slowdowns in processing of benefits. In addition, although the
project has established certain performance measures, it has not yet
established results-oriented measures for productivity and user
satisfaction, both of which will be important for measuring progress.
Finally, the process improvements that VBA has incorporated in the
replacement initiative remain in draft and have not been established
through documented policies and procedures. If VBA does not
institutionalize these improvements, it increases the risk that they
may not be maintained through the life of the project or be available
for application to other development initiatives.
After more than 10 years of effort, including the recent management,
organizational, and process improvements, VBA has developed critical
functionalities needed to process and pay certain original compensation
claims using the replacement system, but it remains far from completing
the project. According to VBA officials, all five of the major software
applications that make up the new system are now being used to
establish and process new compensation claims for veterans. In total,
the replacement system is currently providing monthly compensation
payments to almost 50,000 veterans (out of about 3 million veterans who
receive such payments); the system was used to process about 83 percent
of all new compensation claims completed in March 2007. Nonetheless,
the system requires further development before it can be used to
process claims for the full range of compensation and pension benefits
available to veterans and their dependents.
What GAO Recommends:
To sustain the improved management and software development processes
currently being used by VETSNET project management, GAO is making
recommendations to the Secretary of Veterans Affairs in several areas,
including cost tracking, capacity planning and management, and
performance measures. The Secretary agreed with GAO‘s recommendations
and described actions planned in response.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-614].
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Valerie Melvin at (202)
512-6304 or melvinv@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
VBA Is Following Course Recommended by SEI, but Weaknesses Remain to Be
Addressed:
VETSNET Is Currently Processing a Portion of Compensation Claims, but
Much Work Remains:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Comments from the Department of Veterans Affairs:
Appendix III: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Roles and Responsibilities of VETSNET Organizational Elements:
Table 2: VETSNET Component Applications and Status of Development:
Table 3: Software Releases Planned to Complete the VETSNET System:
Table 4: Conversion Phases:
Figure:
Figure 1: VETSNET System Life Cycle and Review Concept:
Abbreviations:
BDN: Benefits Delivery Network:
CIO: Chief Information Officer:
FAS: Finance and Accounting System:
IT: Information Technology:
MAP-D: Modern Award Processing-Development:
RBA 2000: Rating Board Automation 2000:
SEI: Carnegie Mellon Software Engineering Institute:
VA: Department of Veterans Affairs:
VBA: Veterans Benefits Administration:
VETSNET: Veterans Service Network:
VRE: Vocational Rehabilitation and Employment:
United States Government Accountability Office:
Washington, DC 20548:
April 27, 2007:
The Honorable Bob Filner:
Chairman, Committee on Veterans' Affairs:
House of Representatives:
Dear Mr. Chairman:
Since 1996, the Veterans Benefits Administration (VBA), a major
component of the Department of Veterans Affairs (VA), has been
undertaking a project to replace its compensation and pension benefits
payment system. VBA's existing system, the Benefits Delivery Network
(BDN), is essential to ensuring the accurate processing of compensation
and pension benefits payments to over 3.5 million veterans and their
dependents each month. However, this system, which has been in
operation for more than 40 years, is based on antiquated software
programs, which have become increasingly difficult and costly to
maintain. VBA is in the process of replacing the aging BDN with a
faster, more flexible, and higher capacity system. When it began this
project in 1996, which it generally refers to as the Veterans Service
Network (VETSNET),[Footnote 1] VBA had planned to complete the
replacement system in May 1998 at an estimated cost of $8 million.
Over the years, we have reported on numerous problems that VBA has
encountered in completing the replacement system.[Footnote 2] Our prior
work found, for example, that the project was begun before VBA had
fully developed its business requirements, resulting in confusion over
the requirements to be addressed; in addition, VBA's software
development capability was too immature to ensure that the agency could
reliably develop and maintain high-quality software on any major
project within cost and schedule constraints. In 2002, we offered a
number of recommendations to improve managerial and program weaknesses,
including that VBA appoint a project manager; thoroughly analyze its
initiative; and develop a number of plans, including a revised
compensation and pension replacement strategy and an integrated project
plan. VA concurred with our recommendations, and as we last reported in
June 2006,[Footnote 3] it took several actions to address them. For
example, it appointed a full-time project manager, and the project team
reported that it had completed certifications of users' requirements
for the system's applications. Nonetheless, these actions did not
implement all our recommendations and were not sufficient to establish
the program on a solid footing: certain basic requirements of sound
project management, such as an integrated project plan for the
replacement system, continued to be lacking.
In 2005, the VA Chief Information Officer (CIO) and Under Secretary for
Benefits became concerned about continuing problems with the
replacement project and contracted for an independent assessment of the
department's options for the project, including whether the project
should be terminated. This assessment, conducted by the Carnegie Mellon
Software Engineering Institute (SEI), concluded that the replacement
project faced many risks arising from management and organizational
issues, but no technical barriers that could not be overcome.[Footnote
4] According to SEI, a new system was still needed, and VBA would not
be able to successfully deliver a full, workable solution unless it
addressed its management and organizational weaknesses. SEI recommended
that VBA continue to work on the project at a reduced pace, while
taking an aggressive approach to addressing the identified weaknesses.
Given the importance of ensuring the effective and efficient delivery
of veterans' benefits, we were requested to review the department's
continuing efforts to develop and implement the compensation and
pension replacement system. Specifically, our objectives were to
determine (1) to what extent VA has followed the course of action
recommended by SEI and addressed the concerns that it raised and (2)
the current status of the replacement project.
In conducting this study, we visited VBA's Nashville Regional Office to
observe processing of new benefits claims in the compensation and
pension replacement system, and we visited the St. Petersburg Regional
Office to observe processes and procedures used to test and validate
key functionalities of the replacement system. We obtained and analyzed
documents related to the replacement initiative and to SEI's review of
the initiative. We supplemented our analysis with interviews of VBA and
contractor personnel at headquarters and the two regional offices.
Appendix I contains a more detailed discussion of our objectives,
scope, and methodology. We conducted our study between April 2006 and
April 2007 in accordance with generally accepted government auditing
standards.
Results in Brief:
VBA is generally following the course of action recommended by SEI by
continuing to work on the replacement initiative at a reduced pace,
while taking action to address identified weaknesses in overall
management and software development processes. For example, VBA
established a new governance structure for the initiative that included
senior management and involved all stakeholders, and it incorporated
all critical areas of system development in an integrated master
schedule. However, VBA did not address all of SEI's management
concerns. Specifically, SEI advised VBA to ensure that stakeholders
take ownership responsibility for the project, including the total
system and process operating costs; however, although VBA is tracking
costs incurred by contractors, it is not yet tracking and reporting in-
house costs incurred by the project. Further, VBA has not yet developed
sufficient plans for performing the substantial task of moving records
from the BDN to the replacement system (approximately 3.5 million
beneficiaries are currently being served by the older system). VBA did
take steps to improve its software development processes, such as
establishing risk and requirements management processes, but some have
not been addressed. For example, still to be addressed are processes
for capacity planning and management, which will be important for
ensuring that further development does not lead to slowdowns in
processing of benefits. Finally, VBA has not yet documented policies
and procedures to institutionalize all the improvements that it has
incorporated in the replacement initiative. According to the
replacement project's management team, it made a conscious decision
first to establish the governance and build the organization, among
other things, and it is still prioritizing remaining tasks. However, if
VBA does not institutionalize the improvements made, it increases the
risk that these process improvements may not be maintained through the
life of the project or be available for application to other
development initiatives.
After more than 10 years of effort, including the recent management,
organizational, and process improvements, VBA has developed critical
functionalities needed to process and pay certain original compensation
claims using the replacement system, but it remains far from completing
the project. According to VBA officials, all five of the major software
applications that make up the new system are now being used in VA's
regional offices to establish and process new compensation claims for
veterans. In total, the replacement system is currently providing
monthly compensation payments to almost 50,000 veterans (out of about 3
million veterans who receive such payments); the system was used to
process about 83 percent of all new compensation claims completed in
March 2007. Nonetheless, the system requires further development before
it can be used to process claims for the full range of compensation and
pension benefits available to veterans and their dependents. In
addition, VBA still faces the substantial task of converting records
for the approximately 3.5 million beneficiaries currently being served
by the BDN to the replacement system.
To sustain the improved management and software development processes
currently being used by VETSNET project management, we are making
recommendations to the Secretary of Veterans Affairs in several areas,
including cost tracking, capacity planning and management, and
performance measures.
In providing written comments on a draft of this report, the Secretary
of Veterans Affairs agreed with our conclusions and concurred with the
report's recommendations. (The department's comments are reproduced in
app. II.) The comments described actions planned that respond to our
recommendations, as well as providing further information on relevant
actions already taken. If the planned actions are properly implemented,
they could help strengthen the department's management of the
replacement system project and improve the chances that the system will
be successfully completed.
Background:
VBA provides benefits for veterans and their families through five
programs: (1) compensation and pension, (2) education, (3) vocational
rehabilitation and employment (VRE) services, (4) loan guaranty, and
(5) life insurance. It relies on the BDN to administer benefit programs
for three of VBA's five programs: compensation and pension, education,
and VRE services.
Replacing the aging BDN has been a focus of systems development efforts
at VBA since 1986.[Footnote 5] Originally, the administration planned
to modernize the entire system, but after experiencing numerous false
starts and spending approximately $300 million on the overall
modernization of the BDN, VBA revised its strategy in 1996. It narrowed
its focus to replacing only those functionalities that support the
compensation and pension program, and began developing a replacement
system, which it called VETSNET.
As reported by the department in its fiscal year 2008 budget
submission, the compensation and pension program is the largest of the
three programs that the BDN supports:
* The compensation and pension program paid about $35 billion in
benefits in fiscal year 2006 to about 3.6 million veterans or veterans'
family members.
- Of this amount, compensation programs paid benefits of about $31
billion to about 3.1 million recipients.
- Pension programs paid benefits of about $3.5 billion to about 535,000
recipients.
* The education program paid about $2.8 billion to about 498,000
veterans or their dependents in fiscal year 2006.[Footnote 6]
* The VRE services program paid about $574 million for VRE services in
2006 and provides rehabilitation services to approximately 65,700
disabled veteran participants per year.[Footnote 7]
The Processes Supporting the Variety of Compensation and Pensions
Benefits Are Complex:
One of the challenges of developing the replacement system is that it
must include processes to support the administration of a complex set
of benefits. Different categories of veterans and their families are
eligible for a number of different types of benefits and payments, some
of which are based on financial need. Compensation programs, which are
based on service-connected disability or death, provide direct payments
to veterans and/or veterans' dependents and survivors. These programs
are not based on income. Pension benefits programs, on the other hand,
are income based; these are designed to provide income support to
eligible veterans and their families who experience financial hardship.
Eligible veterans are those who served in wartime and are permanently
and totally disabled for reasons that are not service-connected (or who
are age 65 or older). Veterans are also eligible for burial benefits.
Survivor benefits may be paid to eligible survivors of veterans,
depending on the circumstances. Some of these benefits are based on
financial need, such as death pensions for some surviving spouses and
children of deceased wartime veterans, and Dependency and Indemnity
Compensation to some surviving parents.
Finally, certain benefits may be paid to third parties, such as
individuals to whom a veteran has given power of attorney or medical
service providers designated to receive payments on the veteran's
behalf.
Generally, VBA administers benefit programs through 57 veterans
benefits regional offices[Footnote 8] in a process that requires a
number of steps, depending on the type of claim. When a veteran
submits, for example, a compensation claim to any of the regional
offices, a veterans service representative must obtain the relevant
evidence to evaluate the claim (such as the veteran's military service
records, medical examinations, and treatment records from VA medical
facilities or private medical service providers). In the case of
pension claims, income information would also be collected.
Once all the necessary evidence has been compiled, a rating specialist
evaluates the claim and determines whether the claimant is eligible for
benefits. If the veteran is determined to be eligible for disability
compensation, the Rating Veterans Service Representative assigns a
percentage rating based on the veteran's degree of disability. This
percentage is used in calculating the amount of payment.
Benefits received by veterans are subject to change depending on
changing circumstances. More than half of VBA's workload consists of
dealing with such changes. If a veteran believes that a service-
connected condition has worsened, for example, the veteran may ask for
additional benefits by submitting another claim.[Footnote 9] The first
claim submitted by a veteran is referred to as the original claim, and
a subsequent change is referred to as a reopened claim.
Our Prior Products Identified Weaknesses in the Development and
Implementation of the Replacement System:
Since its inception, VETSNET has been plagued by problems. Over the
years, we have reported on the project, highlighting concerns about
VBA's software development capabilities.
In 1996, our assessment of the department's software development
capability determined that it was immature.[Footnote 10] In our
assessment, we specifically examined VETSNET and concluded that VBA
could not reliably develop and maintain high-quality software on any
major project within existing cost and schedule constraints. The
department showed significant weaknesses in requirements management,
software project planning, and software subcontract management, with no
identifiable strengths. We also testified that VBA did not follow sound
systems development practices on VETSNET, and we concluded that its
modernization efforts had inherent risks.
Between 1996 and 2002, we continued to identify the department's weak
software development capability as a significant factor contributing to
persistent problems in developing and implementing the system. We also
reported that VBA continued to work on VETSNET without an integrated
project plan. As a result, the development of the system continued to
suffer from problems in several areas, including project management,
requirements development, and testing.
Over the years, we made several recommendations aimed at improving VA's
software development capabilities. Among our recommendations was that
the department take actions to achieve greater maturity in its software
development processes[Footnote 11] and that it delay any major
investment in software development (beyond that needed to sustain
critical day-to-day operations) until it had done so. In addition, we
made specific recommendations aimed at improving VETSNET development.
For example, we recommended that VA appoint a project manager,
thoroughly analyze its current initiative, and develop a number of
plans, including a revised compensation and pension replacement
strategy and an integrated project plan.
VA concurred with our recommendations and took several actions to
address them. For example, it appointed a full-time project manager and
ensured that business needs were met by certification of user
requirements for the system applications. The actions taken addressed
some of our specific concerns; however, they were not sufficient to
fully implement our recommendations or to establish the program on a
sound footing.
SEI Assessed System Replacement Initiative and Recommended a Changed
Approach:
As a result of continuing concerns about the replacement project, in
2005 VA's CIO and its Under Secretary for Benefits contracted for an
independent assessment of the department's options for the
initiative.[Footnote 12] The chosen contractor, SEI, is a federally
funded research and development center operated by Carnegie Mellon
University. Its mission is to advance software engineering and related
disciplines to ensure the development and operation of systems with
predictable and improved cost, schedule, and quality.
SEI recommended that the department reduce the pace of development
while at the same time taking an aggressive approach to dealing with
management and organizational weaknesses hampering VBA's ability to
complete the replacement system. According to SEI, these management and
organizational concerns needed to be addressed before the replacement
initiative or any similar project could deliver a full, workable
solution.
For example, the contractor stressed the importance of setting
realistic deadlines and commented that there was no credible evidence
that VETSNET would be complete by the target date, which at the time of
the review had slipped to December 2006. According to the assessment,
because this deadline was unrealistic, VBA needed to plan and budget
for supporting the BDN so that its ability to pay veterans' benefits
would not be disrupted. SEI also noted that different organizational
components had independent schedules and priorities, which caused
confusion and deprived the department of a program perspective.
Further, the contractor concluded that VBA needed to give priority to
establishing sound program management to ensure that the project could
meet targeted dates. These and other observations were consistent with
our long-standing concerns regarding fundamental deficiencies in VBA's
management of the project.
To help VBA implement the overall recommendation, the contractor's
assessment included numerous discussions of activities needed to
address these areas of concern, which can be generally categorized as
falling into two major types:
* Overall management concerns with regard to the initiative included:
- governance structure, including assigning ownership for the project
and its costs;
- project planning, including the development schedule and capacity
planning; and:
- conversion of records currently on the BDN to the replacement system.
* Software development process improvements were needed in the
following areas:
- risk management,
- requirements management,
- defect management, and:
- program measures.
VBA Is Following Course Recommended by SEI, but Weaknesses Remain to Be
Addressed:
As recommended by SEI, VBA is continuing to work on the replacement
initiative at a reduced pace and taking action to address identified
weaknesses in the project's overall management and software development
processes. For example, VBA has established a new governance structure
and has developed an integrated master schedule that provides
additional time and includes the full range of project activities.
However, additional effort is needed to complete a number of the
corrective actions, such as improving project accountability through
monitoring and reporting all project costs. Further, VBA has not yet
institutionalized many of the improvements that it has undertaken for
the initiative. In particular, process improvements remain in draft and
have not been established through documented policies and procedures.
According to the VETSNET management team, it gave priority to other
activities, such as establishing appropriate governance and
organizational structures, and it is still gathering information to
assist in prioritizing the activities that remain. Nonetheless, if VBA
does not institutionalize these improvements, it increases the risk
that these process improvements may not be maintained through the life
of the project or be available for application to other development
initiatives.
VBA Is Taking Action on Certain Overall Management Concerns:
SEI concluded that VBA's management issues would need to be addressed
as part of the implementation of its overall recommendation. SEI's
overall management concerns focused on the project governance, project
planning, and conversion of records currently on the BDN to the
replacement system.
Top Management Included in Revised Governance Structure:
SEI guidance for software development stresses the need for
organizational commitment and the involvement of senior management in
overall project governance.[Footnote 13] In its assessment, SEI noted
that because management of the VETSNET project had been assigned to
VBA's information technology (IT) group, certain activities critical to
the veterans' benefits program, but not traditionally managed by the IT
group, had not been visible to the project's management. The contractor
pointed out that the IT group, business lines, and regional offices
needed to share ownership and management of the replacement project
through an established governance process and that the project
management office should include business representatives. According to
SEI, the project needed to establish ownership responsibility,
including addressing total system and process operating costs.
In response to the assessment, VBA developed a new governance structure
for the initiative, which the Under Secretary for Benefits approved in
March 2006. In the new structure, the VETSNET Executive Board that had
been in place was expanded and reorganized to serve as a focal point
and major governance mechanism for the replacement initiative. A
Special Assistant (reporting directly to the Under Secretary) was
appointed to coordinate and oversee the initiative as the head of the
VETSNET Executive Team, which was established to provide day-to-day
operational control and oversight of the replacement initiative.
Implementation Teams were also established to conduct the day-to-day
activities associated with implementing the initiative. This governance
structure established a process for IT, business lines, and regional
offices to share ownership and management, as SEI advised.
The roles and membership of each of the organizational elements in the
new governance structure are described in table 1.
Table 1: Roles and Responsibilities of VETSNET Organizational Elements:
Organization: VETSNET Executive Board;
Role: Provides executive direction to the VETSNET Executive Team;
Membership: Deputy Under Secretary for Benefits; Special Assistant to
the Under Secretary for Benefits; VETSNET Program Manager; VA Office of
Information and Technology Principal Associate Deputy Assistant
Secretary; Chief Financial Officer; Director, Compensation and Pension
Service; Director, Office of Performance Analysis and Integrity;
Associate Deputy Under Secretary for Policy and Programs; Associate
Deputy Under Secretary for Field Operations; VBA Chief Information
Officer.
Organization: VETSNET Executive Team;
Role: Responsible for the delivery of the project and acts as the
review and approval authority for the Implementation Teams;
Membership: VETSNET Program Executive (Special Assistant to Under
Secretary); VETSNET Program Manager; Contract support; Business
Architect; Technical Architect; MITRE support.
Organization: VETSNET Implementation Teams;
Role: Responsible for the day-to-day activities necessary to develop
and implement the replacement system;
Membership: Compensation and Pension; Office of Resource Management;
Office of Performance Analysis and Integrity; Office of Field
Operation; Office of Information Management, VA Central Office; St.
Petersburg Systems Development Center; Hines Information Technology
Center; Austin Automation Center.
Source: GAO analysis of VBA information.
[End of table]
When the new governance structure was approved in March 2006, the Under
Secretary ensured that those involved in the project gave it high
priority, directing certain key personnel (such as members of the
executive and implementation teams) to make the initiative their
primary responsibility, and other personnel (technical staff that
provide support to other systems) with collateral (non-VETSNET) duties
to make the project their first priority. He also placed limitations on
the transfer of personnel away from the project, recognizing the
importance of staff continuity in successfully completing the
initiative. Staff members assigned project responsibilities could be
reassigned (i.e., given nonpromotion, lateral reassignments) only with
approval from the Under Secretary or his deputy.
By implementing the new governance and organizational structure and
ensuring that the project has priority, VBA partially responded to
SEI's concerns in this area; however, VBA has not yet taken action with
regard to ownership responsibility for total system and process
operating costs, as SEI advised. According to administration officials,
the replacement initiative is an in-house, contractor-assisted
development effort, in which three different contractors provide
support for program management, system development, and testing and
validation of requirements. VA reported VETSNET system costs to the
Congress totaling about $89 million for fiscal years 1996 through 2006,
with additional estimated costs for completion of the initiative in
2009 of about $62.4 million. However, according to project management
officials, these costs do not include expenditures for in-house
development work. This in-house work involves many VA personnel, as
well as travel to various locations for testing and other project
related activities. Thus, considerable costs other than contract cost
have been incurred, which have not been tracked and reported as costs
for the replacement initiative. Without comprehensive tracking and
reporting of costs incurred by the replacement project, the ability of
VBA and the Congress to effectively monitor progress could be impaired.
VBA Took Action to Address Project Planning and Management:
A second major area of overall management concern was project planning.
In particular, the lack of an integrated master schedule for the
VETSNET project was a major concern articulated by SEI, as well as in
our prior work. An integrated project plan and schedule should
incorporate all the critical areas of system development and be used as
a means of determining what needs to be done and when, as well as
measuring progress. Such an integrated schedule should consider all
dependencies and include subtasks so that deadlines are realistic, and
it should incorporate review activities to allow oversight and approval
by high-level managers. Among other things, the program plan should
also include capacity requirements for resources and technical
facilities to support development, testing, user validation, and
production.
SEI was specifically concerned that releases with overlapping
functionality were being developed at the same time, with insufficient
time to document or test requirements; this approach constrained
resources and added complexity because of the need to integrate
completed applications and newly developed functionality. In addition,
SEI observed that the VETSNET program suffered from lack of sufficient
test facilities because it did not have enough information to plan for
adequate capacity.
In response to these project planning concerns, VETSNET management,
with contractor support, developed an integrated master schedule to
guide development and implementation of the remaining functionalities
for the replacement system. The VETSNET Integrated Master Schedule,
finalized in September 2006, includes an end-to-end plan and a master
schedule. According to VBA, the end-to-end plan documents the end state
of the project from a business perspective, which had not previously
been done. The master schedule identifies the necessary activities to
manage and control the replacement project through completion. The
schedule also describes a new software release process that provides
more time to work on requirements definition and testing, and allows
for more cross-organizational communications to lessen the possibility
of not meeting requirements.
In addition, the new release process includes a series of management
reviews to help control the software development process and ensure
that top management has continuous visibility of project related
activities. These reviews occur at major steps in the system
development life cycle (as described in fig. 1: initiation, preliminary
design, and so on). Such reviews are intended to ensure that the
VETSNET Executive Team and the VETSNET Executive Board agree and accept
that the major tasks of each step have been properly performed.
Figure 1: VETSNET System Life Cycle and Review Concept:
[See PDF for image]
Source: VBA.
[End of figure]
Nonetheless, while the Integrated Master Schedule is an important
accomplishment, it may not ensure that the project sufficiently
addresses capacity planning, one of SEI's areas of concern. According
to its assessment, capacity requirements for the fully functional
production system were unclear. Capacity planning is important because
program progress depends on the availability of necessary system
capacity to perform development and testing; adjustments to such
capacity take time and must be planned. If systems do not have adequate
capacity to accommodate workload, interruptions or slowdowns could
occur. According to SEI, capacity adjustments cannot be made instantly,
and program progress will suffer without sufficient attention to
resource requirements.
However, the VETSNET Integrated Master Schedule does not identify
activities or resources devoted to capacity planning. According to
officials, the capacity of the corporate environment (that is,
corporate information systems, applications, and networks) is being
monitored by operational teams with responsibility for maintaining this
environment. According to project officials, VETSNET representatives
participate in daily conference calls in which the performance of
corporate applications is discussed, and changes in application
performance are reported to the VETSNET developers for investigation
and corrective action. Project officials reported that when a
performance degradation occurred in some transactions during
performance testing, it was determined that additional computing
capacity was needed and would be acquired. One reason why the
occurrence of degradation had not been anticipated by the VETSNET
project was that capacity planning had not taken place. Unless it
ensures that capacity planning and activities are included in the
Integrated Master Schedule, the replacement project may face other
unanticipated degradations that it must react to after the fact, thus
jeopardizing the project's cost, schedule, and performance.
Conversion Efforts Were Suspended as Advised, but Issues Remain:
In its assessment, SEI questioned VBA's approach to developing
functionality while concurrently converting records from the BDN to the
replacement system. It noted that VBA had chosen to complete software
development according to location rather than according to the type of
functionality. Specifically, in 2004, VBA began an effort to remove all
claims activity (both new and existing claims) at one regional office
(Lincoln, Nebraska) from the BDN to the replacement system, developing
the software as necessary to accommodate processing the types of claims
encountered at that site. The intention was to address each regional
office in turn until all sites were converted. According to SEI, this
approach had resulted in the development being stalled by obstacles
arising from the variety of existing claims.[Footnote 14] The
contractor advised VBA to focus first on developing functionality to
process original claims and discontinue efforts to convert existing
claims until all the necessary functionality had been developed, and
the replacement system's ability to handle new cases of any complexity
had been proven by actual experience.
In accordance with this advice, VBA stopped converting existing records
from the BDN and changed its focus to developing the necessary
functionality to process all new compensation claims. According to the
integrated master schedule, conversion activities are now timed to
follow the release of the needed functionality. That is, according to
the schedule, VBA plans to begin converting each type of record from
the BDN only after the necessary functionality for the replacement
system has been developed and deployed to process that type of record.
In addition, the project is mitigating risk by resuming conversions
beginning with a test phase. Its strategy is first to convert records
for terminated claims--claims that are no longer being paid. Conversion
of the terminated records will be followed by additional conversions of
records for claims receiving payment at Lincoln and Nashville (these
two sites are being used to test system functionality during
development). The VETSNET leadership will consider testing complete
with the successful conversion at these two sites.
However, SEI raised three additional issues with regard to the
conversion of records that VBA has not fully addressed:
* First, SEI expressed concerns that conversion failures could lead to
substantial numbers of records being returned to the BDN. Because of
differences in the database technologies used for the old system and
the replacement system, certain types of errors in BDN records cause
conversion to fail (according to SEI, approximately 15 percent of all
these records are estimated to have such errors). If records fail to
convert correctly, they may need to be returned to the BDN so that
benefits can continue to be paid. However, this process is not simple
and may involve manually reentering the records.[Footnote 15]
* Second, SEI observed that VBA was also depending on manual processes
for determining that records were converted successfully, including the
use of statistically random samples, and that it was aiming to ensure
correctness to a confidence level of 95 percent. However, in the
absence of a straightforward method for automatically returning records
to the BDN, SEI considered the 5 percent risk of error unacceptable for
conversions of large numbers of records.
* Finally, SEI observed that the lack of automated methods and the
complexity of the processes meant that conversions required careful
planning and assurance that adequate staff would be available to
validate records when the conversions took place.
However, the VETSNET leadership has not developed any strategy to
address the possibility that a large number of cases might need to be
returned to the BDN during the testing phase. For example, it has not
included this possibility as a risk in its risk management plan. The
absence of a strategy to address this possibility could lead to delays
in program execution.
Further, VBA has not yet decided whether a possible 5 percent error
rate is acceptable or developed a plan for addressing the resulting
erroneous records. If VBA does not address these issues in its
planning, it increases the risk that veterans may not receive accurate
or timely payments.
Finally, the VETSNET leadership has not yet developed detailed plans
that include the scheduled conversions for each regional office and
identified staff to perform the necessary validation. Having such plans
would reduce the risk that the conversion process could be delayed or
fail.
VBA Is Improving Software Development Processes, but Improvements Are
Not Yet Institutionalized:
In addition to actions addressing the overall management concerns
identified by SEI, VBA has steps action to improve its software
development processes in risk management, requirements management,
defect/change management, and performance measures. SEI described
weaknesses in all of these areas. The steps taken have generally been
effective in addressing the identified weaknesses, but VBA has not yet
institutionalized many of these improvements. According to the VETSNET
management team, it made a conscious decision first to establish the
governance, build the organization, implement processes to gain
control, and gather additional information about the project to assist
in prioritizing the remaining activities. The team also stated that
some of the processes are no longer VBA's responsibility but are now
that of the newly realigned Office of Information and
Technology.[Footnote 16] Nonetheless, if VA does not develop and
establish documented policies and procedures to institutionalize these
improvements, they may not be maintained through the life of the
project or available to be applied to other development initiatives.
Risk Management Plan Has Been Revised:
Risk management is a process for identifying and assessing risks, their
impact and status, the probability of their occurrence, and mitigation
strategies. Effective risk management includes the development of a
risk management plan and tracking and reporting progress against the
plan. According to SEI, to the extent that risk management existed at
all in the replacement program, it was conducted on a pro forma basis
without real effect on program decisions. SEI said that risks and risk
mitigation activities needed to be incorporated into all aspects of
program planning, budgeting, scheduling, execution, and review.
In response to these concerns, VBA has instituted risk management
activities that, if properly implemented, should mitigate the risks
associated with the project. Specifically, the VETSNET team, with
contractor support, developed a risk management plan that was adopted
in January 2007. The plan includes procedures for identifying,
validating, analyzing, assessing, developing mitigation strategies for,
controlling and tracking, reporting, and closing risks. It also
establishes criteria for assessing the severity of the risks and their
impact.
The VETSNET leadership also developed a Risk Registry database, and its
contractor reviewed and prioritized the open risks. Each open risk was
evaluated, and a proposed disposition of the risk was submitted to
VETSNET management. Of the 39 open risks, all but 3 had been addressed
as of January 2007.
The development documentation for each planned software release also
includes sections on risk. In accordance with these plans, the VETSNET
leadership is currently capturing potential risks and tracking action
items and issues. At weekly status meetings, VETSNET leadership reviews
Risk Registry reports of open risks. According to the contractor, the
reports identify each risk and provide information on its age,
ownership, and severity.
However, these risk management activities have not yet been
institutionalized through the definition and establishment of
associated policies and procedures. If it does not institutionalize
these improvements, VBA increases the possibility that the VETSNET
project's improvements in risk management may not be maintained through
the life of the project.
Requirements Management Has Improved:
Requirements management is a process for establishing and maintaining a
common understanding between the business owners and the developers of
the requirements to be addressed, as well as verifying that the system
meets the agreed requirements. SEI's report commented that the VETSNET
project requirements were not stable, and that the business owners
(including subject-matter experts) and developers were separated by
many organizational layers, resulting in confusion and delays in
development of the system. SEI suggested that VA restructure project
activities to focus on defining an effective requirements process.
According to SEI, the project needed to ensure that subject-matter
experts were included in developing requirements and that evaluation
criteria were established for prioritizing requests for changes to
requirements. Finally, business owners should confirm that the system
is meeting organizational needs.
VBA has instituted requirements management activities that, if properly
implemented, should help avoid the instability and other requirements
problems identified by SEI. Specifically, VBA took steps to establish a
requirements management process and to stabilize the requirements. For
example, the development release process in the Integrated Master
Schedule includes a phase for requirements identification. In addition,
the project has established and begun applying evaluation criteria to
prioritize change requests for its development releases. Further, until
all claims are completely migrated from the BDN to the replacement
system, in July 2006, the Under Secretary directed that any additional
requirements would have to have his approval.
Responding to SEI's advice regarding the involvement of subject-matter
experts and business owners, VBA designed the new release process to
directly involve subject-matter experts in requirements workshops.
Further, the business teams participate in user-acceptance testing.
However, these requirements management activities have not yet been
institutionalized through the definition and establishment of policies
and procedures. Until they are established, VBA runs the risk that the
improved processes will not be maintained through the life of the
VETSNET project or used in other software development projects.
Management Attention Is Being Focused on Major Software Defects:
SEI raised numerous concerns regarding the defect process for the
replacement system. These concerns for defect management included (1)
identification of defects, (2) determination of cause, and (3)
disposition of defects--either by correction or workaround. According
to SEI guidance, defect management prevents known defects from
hampering the progress of the program. The management process should
include clearly identifying and tracking defects, analyzing defects to
establish their cause, tracking their disposition, clearly identifying
the rationale for not addressing any defects (as well as proposing
workarounds), and making information on defects and their resolution
broadly available. SEI's report stated that VBA needed to distinguish
defects from changes to requirements and develop a process for defect
management.
To respond to these concerns and focus program management attention on
major defects, the VETSNET Executive Team, with contractor support,
conducted an audit of existing defects and revised the defect
management process. The audit of the defect database determined that
the VETSNET database used to capture software defects also included
change requests; as a result, work required to address processes that
did not work properly was not distinguished from requests for added or
changed functionality, which would require review and approval before
being addressed. To address this issue, the team separated defects from
change requests, and a new severity rating scale was developed. All
open defects were recategorized to ensure the major defects would
receive appropriate program management attention. Also, all defect
categorizations must meet the approval of the VETSNET Business
Architect and are scheduled for action as dictated by the severity
level.
Although these steps address many of SEI's concerns regarding VBA's
defect management process, more remains to be done before the process
is institutionalized. The Program Management Office has reported that
actions to revise the defect management process are complete, but the
process description is still in draft, and policies and procedures have
not been fully established. Without institutionalized policies and
procedures for the defect management process, it may not be maintained
consistently through the life of the project.
The Replacement Initiative Is Tracking Certain Performance Measures:
According to SEI, performance measures are the only effective mechanism
that can provide credible evidence of a program's progress. The chosen
measures must link directly to the expected accomplishments and goals
of the system, and they must be applied across all activities of the
program. In its report, SEI stated that although VBA was reporting
certain types of performance measures, it was not relating these to
progress in system development. For example, VBA reported the total
number of veterans paid, but did not provide estimates of how many
additional veterans would be paid when the system incorporated specific
functionalities that were under development. SEI suggested several
measures that would provide more evidence of progress, such as
increases in the percentage of original claims being paid by the
replacement system, as well as user satisfaction and productivity gains
resulting from use of the replacement system applications at regional
offices.
In response to these concerns, the replacement project has begun
tracking a number of the measures suggested by SEI, including:
* increases in the percentage of original claims being paid by the
replacement system,
* increases in the percentage of veterans' service representatives
using the new system,
* decreases in the percentage of original claims being entered in the
BDN rather than the replacement system.
Although these measures provide indications of VA's progress, other
measures that could demonstrate the effectiveness of the replacement
system have not been developed. For example, VBA has not developed
results-oriented measures to capture user satisfaction or productivity
gains from the system. Without measuring user satisfaction, VBA has
reduced assurance that the replacement system will be accepted by the
users. In addition, measures of productivity would provide VBA with
another indication of progress toward meeting business needs.
VETSNET Is Currently Processing a Portion of Compensation Claims, but
Much Work Remains:
After more than 10 years of effort, including the recent management,
organizational, and process improvements, VBA has achieved critical
functionalities needed to process and pay certain original compensation
claims using the replacement system, but it remains far from completing
the project. For example, the replacement system is currently being
used to process a portion of the original claims that veterans file for
compensation. Nonetheless, the system requires further development
before it can be used to process claims for the full range of
compensation and pension benefits available to veterans and their
dependents. In addition, VBA still faces the substantial task of moving
approximately 3.5 million beneficiaries who are currently being served
by the BDN to the replacement system.
As designed, VETSNET consists of five major system applications that
are used in processing benefits:
* Share--used to establish claims;[Footnote 17] it records and updates
basic information about veterans and dependents both in the BDN and the
replacement system.
* Modern Award Processing-Development (MAP-D)--used to manage the
claims development process, including the collection of data to support
the claims and the tracking of claims.
* Rating Board Automation 2000 (RBA 2000)--provides laws and
regulations pertaining to disabilities, which are used by rating
specialists in evaluating and rating disability claims.
* Award Processing (Awards)--used to prepare and calculate the benefit
award based on the rating specialist's determination of the claimant's
percentage of disability. It is also used to authorize the claim for
payment.
* Finance and Accounting System (FAS)--used to develop the actual
payment record. FAS generates various accounting reports and supports
generation and audit of benefit payments.
According to VBA officials, all five of the software applications that
make up the new system are now being used in VA's 57 regional offices
to establish and process new compensation claims for veterans. As of
March 2007, VBA leadership reported that the replacement system was
providing monthly compensation payments to almost 50,000 veterans (out
of about 3 million veterans who receive such payments). In addition,
the replacement system has been processing a steadily increasing
percentage of all new compensation claims completed: this measure was
47 percent in January 2007, increasing to 60 percent in February and 83
percent in March.
Nonetheless, considerable work must be accomplished before VBA will be
able to rely on the replacement system to make payments to all
compensation and pension beneficiaries. Specifically, while all five
software applications can now be used to process original compensation
claims for veterans, two of the applications--Awards and FAS--require
further development before the system will be able to process claims
for the full range of benefits available to veterans and their
dependents. Table 2 shows the status of development of all five
applications.
Table 2: VETSNET Component Applications and Status of Development:
Application: Share;
Development status: Completed;
Deployment status: Fully deployed at all regional offices.
Application: MAP-D;
Development status: Completed;
Deployment status: Fully deployed at all regional offices.
Application: RBA 2000;
Development status: Completed;
Deployment status: Fully deployed at all regional offices.
Application: Awards;
Development status: Functionality to process third-party/nonveteran
payee claims is in application testing; Requirements for processing
survivor benefits are being developed; Functionality to process
pensions is not yet under development;
Deployment status: In partial use in all regional offices.
Application: FAS;
Development status: Functionality to process third- party/nonveteran
payee claims is in application testing; Functionality to generate
management reports is partially in requirements development and
partially in application testing; Requirements for processing survivor
benefits are being developed; Functionality to process pensions is not
yet under development;
Deployment status: In partial use in all regional offices.
Source: GAO analysis of VBA data.
[End of table]
According to VBA officials, Awards and FAS do not yet have the
capability to process original claims for payment to recipients other
than veterans: that is, the applications do not have the functionality
to process claims for survivor benefits[Footnote 18] and third-party/
nonveteran payee claims.[Footnote 19] In addition, further development
of these applications is needed to process pension benefits for
qualified veterans and their survivors. Until enhancements are made to
Awards and FAS, these claims must continue to be processed and paid
through the BDN. Also, according to VBA, FAS does not yet have the
capability to generate all the necessary accounting reports that
support the development of benefits payments to claimants.
As described earlier, VBA now has an Integrated Master Schedule that
incorporates the activities that VBA needs to manage in order to
complete the replacement project. According to the schedule, the
remaining capabilities necessary to process compensation and pension
claims are to be developed and deployed in three software releases, as
shown in table 3.
Table 3: Software Releases Planned to Complete the VETSNET System:
Release number and title: 1. Complete Compensation;
Functionality to be attained: Process all compensation claims and add
significant FAS functionality (generate management reports);
Estimated completion date: August 2007.
Release number and title: 2. Nonincome-Based Survivor Benefits;
Functionality to be attained: Process burial, accrued, and survivor
claims;
Estimated completion date: February 2008.
Release number and title: 3. Income-Based Pension;
Functionality to be attained: Process nonservice-connected pension
(veteran and survivor) and parents' Dependent and Indemnity
Compensation Benefit;
Estimated completion date: August 2008.
Source: VBA.
[End of table]
As table 3 shows, VBA does not expect to complete the development of
the functionalities needed to process all new compensation and pension
claims until August 2008. However, according to VBA, the estimated
completion date is the planned date for completing all development and
testing, but it is not necessarily the date when users will be able to
begin using the new system. Before such use can begin, other activities
need to occur. For example, users must receive training, and VETSNET
program management must authorize the use of the system at each
regional office.
In addition to its remaining software development activities, VBA also
faces the challenge of converting records for claims currently paid by
the BDN to the replacement system. Existing compensation and pension
cases on the BDN number about 3 million and about 535,000,
respectively. Table 4 shows the phases in which VA is planning to
perform conversions, according to its Integrated Master Schedule.
Table 4: Conversion Phases:
Phases: Phase 1;
Conversions: Conversion testing on terminated records and selected live
records;
Estimated start date: March 2007;
Estimated completion date: August 2007.
Phases: Phase 2;
Conversions: Conversion of live compensation records;
Estimated start date: January 2008;
Estimated completion date: October 2008.
Phases: Phase 3;
Conversions: Conversion of pension records;
Estimated start date: November 2008;
Estimated completion date: June 2009.
Source: GAO analysis of VBA data.
[End of table]
As the table shows, VBA conversion efforts began in March 2007. VBA
first performed conversion testing on 310,000 terminated (that is,
inactive) compensation cases so that it could develop and apply lessons
learned to the conversion of live records. According to VETSNET
officials, VBA planned to continue testing by converting live cases at
two regional offices (Lincoln and Nashville) that were used as testing
sites during development. It then plans to perform the conversion of
all live compensation cases. After the compensation conversion is
complete, VBA plans to begin efforts to convert pension benefits cases.
Based on VETSNET project documentation, activities supporting the
releases have so far been performed on time, consistent with the
milestones in the recently finalized Integrated Master Schedule. For
example, VA completed the Project Initiation and Review Authorization
for Release 1 on September 7, 2006, as scheduled (see fig. 1, shown
earlier in the report, for the phases of system development and the
required milestone reviews). It also completed the Preliminary Design
Review and the Critical Design Review as scheduled (on November 20 and
December 22, respectively). Planning for Release 2 is also on schedule:
a kickoff meeting was held on January 24, 2007, which established the
scope of the release, and the Project Initiation and Review
Authorization was conducted on February 8.
Conclusions:
VA has responded to SEI's assessment by making significant changes in
its approach to the project and its overall management, including
slowing the pace of development, establishing a new governance
structure, and ensuring staff resources. However, VBA has not yet
addressed all the issues raised by the SEI assessment. That is, it has
not ensured ownership responsibility for total system and process
operating costs, because it is not currently monitoring and reporting
in-house expenditures for the project. It has not defined processes and
resources for capacity planning for the project. In addition, VBA has
not yet addressed issues related to the conversion of records now on
the BDN to the replacement system. Specifically, it has not addressed
the risk that large numbers of records may need to be returned to the
BDN, decided on the degree of confidence it will require that records
are converted accurately, or developed complete plans for converting
and validating records. In addition, although VBA has improved key
processes for managing the software development, these processes have
not yet been institutionalized in defined policies and procedures, and
performance measures of productivity and user satisfaction have not
been developed. VETSNET management has stated that it gave priority to
other activities, such as establishing appropriate governance and
organizational structures, and that it is still gathering information
to assist in prioritizing the activities that remain.
Much work remains to be done to complete the VETSNET initiative.
Although VBA has substantially increased the number of claims being
paid by the replacement system, it must not only finish the development
and deployment of the software, it must also convert the over 3.5
million records now on the BDN to the replacement system. Addressing
the remaining issues identified by SEI would improve VBA's chances of
successfully completing the replacement system and ending reliance on
the aged BDN to pay compensation and pension benefits.
Recommendations for Executive Action:
To enhance the likelihood that the replacement system will be
successfully completed and implemented, we are recommending that the
Secretary of Veterans Affairs take the following five actions:
* Direct the CIO to institute measures to track in-house expenditures
for the project.
* Direct the VETSNET project to include activities for capacity
planning in the VETSNET Integrated Master Schedule and ensure that
resources are available for these activities.
* Direct VBA to (1) develop a strategy to address the risk that large
numbers of records may need to be returned to the BDN; (2) determine
whether a greater confidence level for accuracy should be required in
the conversion process; and (3) develop a detailed validation plan that
includes the scheduled conversions for each regional office and the
validation team members needed for that specific conversion.
* Direct the CIO to document and incorporate the improved processes for
managing risks, requirements, and defects into specific policy and
guidance for the replacement initiative and for future use throughout
VBA.
* Direct the replacement project to develop effective results-oriented
performance measures that show changes in efficiency, economy, or
improvements in mission performance, as well as measures of user
satisfaction, and to monitor and report on the progress of the
initiative according to these measures.
Agency Comments and Our Evaluation:
In providing written comments on a draft of this report, the Secretary
of Veterans Affairs agreed with our conclusions and concurred with the
report's recommendations. (The department's comments are reproduced in
app. II.) The comments described actions planned that respond to our
recommendations, such as incorporating processes developed for the
VETSNET project in standard project management policies, processes, and
procedures that would be used for all IT projects in the department. In
addition, the comments provided further information on actions already
taken, such as details of the records conversion process. If the
planned actions are properly implemented, they could help strengthen
the department's management of the replacement system project and
improve the chances that the system will be successfully completed.
We are sending copies of this report to the Chairman and Ranking
Minority Member of Committee on Veterans' Affairs. We are also sending
copies to the Secretary of Veterans Affairs and appropriate
congressional committees. We will make copies available to other
interested parties upon request. Copies of this report will also be
made available at no charge on GAO's Web site at http://www.gao.gov.
Should you or your staff have any questions about this report, please
contact me at (202) 512-6304 or by e-mail at melvinv@gao.gov. Contact
points for our Offices of Congressional Relations and Public Affairs
may be found on the last page of this report. Key contributors to this
report are listed in appendix III.
Sincerely yours,
Signed by:
Valerie C. Melvin:
Acting Director:
Human Capital and Management Information Systems:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
Our objectives were to determine (1) to what extent the Department of
Veterans Affairs (VA) has followed the course of action recommended by
the Carnegie Mellon Software Engineering Institute (SEI) and addressed
the concerns that it raised and (2) the current status of the
replacement project, the Veterans Service Network (VETSNET).
To determine the actions taken to implement SEI's recommended approach
and address the concerns it raised, we:
* determined the recommended actions by analyzing the report;
* compared the concerns identified in the assessment to actions
planned, actions undertaken but not completed, and actions implemented
by VA officials or contractors;
* interviewed contractor, VA, and VETSNET program office officials to
gain an understanding about processes developed and procedures
implemented; and:
* obtained and reviewed relevant VA and contractor documents that
disclosed or validated VA responses to SEI's concerns.
To determine the status of system development efforts and the extent
that tasks planned for the initiative were completed, we analyzed VA
and contractor documentation regarding system operations and
development, time frames, and activities planned. We analyzed VA
documents that disclosed costs to date and costs planned for completion
of the initiative. We did not assess the accuracy of the cost data
provided to us. We supplemented our analyses with interviews of VA and
contractor personnel involved in the replacement initiative.
We visited the Nashville and St. Petersburg regional offices to observe
the replacement system in operation and the processes and procedures
used to test and validate the replacement system as it was being
developed and implemented. We analyzed VA documentation and relevant
evidence from contractors involved in the replacement effort to
establish the work remaining to complete the project. Finally, we
interviewed cognizant VA and contractor officials, responsible for
developing, testing, and implementing the replacement system.
We performed our work at VA offices in Washington, D.C., and at VA
regional offices in Nashville, Tennessee, and St. Petersburg, Florida,
from April 2006 to April 2007 in accordance with generally accepted
government auditing standards.
[End of section]
Appendix II: Comments from the Department of Veterans Affairs:
The Secretary Of Veterans Affairs:
Washington:
April 18, 2007:
Ms. Valerie C. Melvin:
Acting Director:
Human Capital and Management Information Systems:
Information Technology Team:
U. S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Ms. Melvin:
The Department of Veterans Affairs (VA) has reviewed your draft report,
Veterans Benefits Administration: Progress Made in Long-Term Effort to
Replace Benefits Payment System, but Challenges Persist, GAO 07-614)
and agrees with your conclusions and concurs in your recommendations.
VA is eager to move to a new automated system that will streamline its
benefit claims processing functions in a manner that is accurate and
timely to serve our Nation's veterans. The Government Accountability
Office's recommendations should provide valuable assistance in
achieving that goal. The enclosure addresses your recommendations in
detail.
I appreciate the opportunity to comment on your draft report.
Sincerely yours,
Signed by:
R. James Nicholson:
Enclosure:
Department of Veterans Affairs (VA) comments to Government
Accountability Office (GAO) draft report Veterans Benefits
Administration: Progress Made in Long-Term Effort to Replace Benefits
Payment System, but Challenges Persist (GAO-07-614):
To enhance the likelihood that the replacement system will be
successfully completed and implemented, GAO recommends that the
Secretary of Veterans Affairs take the following actions:
* direct the CIO to institute measures to track in-house expenditures
for the project;
Concur - VA's Chief Information Officer (CIO) is committed to tracking
the in-house expenditures required to complete the Veterans Service
Network (VETSNET) project. In addition to tracking Government full time
employee (FTE) and contractor costs, the VETSNET Project Management
Office will take responsibility for tracking travel costs associated
with the project to include those related to application testing and
training.
* direct the VETSNET project to include activities for capacity
planning in the VETSNET Integrated Master Schedule and ensure that
resources are available for these activities;
Concur - The Veterans Benefits Administration (VBA) has already
implemented capacity planning activities in conjunction with the
Integrated Master Schedule (IMS). As noted during the review, capacity
planning was difficult in advance of the IMS development, because VBA
had not identified when specific functionality would be delivered and
available for claims processing. The IMS identifies when specific types
of functionality will be delivered into the production environment and
when records will be converted from the Benefits Delivery Network
(BDN). Increased levels of claims processing are tied to each of these
releases and conversions, which in turn, allows for more effective
capacity planning. As reported, VBA has a staff dedicated to
performance monitoring and capacity planning (note, this organization
now falls under the Department's Office of Information and Technology).
Performance testing, which is a component of capacity planning, is a
defined element of the overall VETSNET development and testing
timeline. Results of performance testing for each release are factored
in and used to validate capacity assumptions. To ensure functionality
already delivered to production continues to meet performance
expectations, VETSNET leadership attends daily conference calls
regarding corporate database performance. This ensures any issues that
may unexpectedly arise are rapidly addressed. In addition, VETSNET
leadership has fully briefed its overall schedule and claims processing
estimates to the capacity planning staff to assist that staff in better
understanding VETSNET's plans and needs. Finally, to ensure sufficient
attention is given this vital element of the project, VBA has provided
for assistance in capacity planning in MITRE Corporation's 2007
contract for VETSNET support.
* direct VBA to (1) develop a strategy to address the risk that large
numbers of records may need to be returned to the BDN; (2) determine
whether a greater confidence level for accuracy should be required in
the conversion process; and (3) develop a detailed validation plan that
includes the scheduled conversions for each regional office and the
validation team members needed for that specific conversion;
Concur-(1) As part of addressing the Software Engineering Institute's
(SEI) recommendations, and in developing the Integrated Master
Schedule, VBA has taken significant steps in addressing the conversion
of existing records from the BDN to the corporate database. As
reported, VBA stopped converting records from the BDN in order to
develop a more coordinated approach. Conversions are now planned for
execution in accordance with delivery of application functionality into
production, followed by a period of stabilization. Since the initial
conversions of VA Regional Office (VARO) Lincoln, Nebraska, records,
data show that the only reason records required return to BDN was
because application functionality no longer supported the record. For
example, the record of a veteran in receipt of disability compensation
was successfully converted from the BDN. The veteran then elected to
receive non-service connected pension as the greater monetary benefit.
Since pension is not fully supported in VETSNET, the veteran's record
required return to the BDN to continue payment. At no time has a record
required return to the BDN due to the conversion being "wrong" or
paying the veteran incorrectly.
VBA's current strategy, as noted in the report, is to convert a large
volume of terminated records, followed by conversions of records at
Lincoln and then VARO Nashville, Tennessee. These conversions were
designed to test both conversion processes and validation methods to
ensure the correct conversion of the selected records. A "correct"
conversion is defined as uninterrupted payment at the correct rate. In
the event modifications to our conversion methods are required, the
schedule allows sufficient time to address those changes.
In addition, a core conversion team has been assembled and is testing
and validating. This team includes technical resources, individuals
from headquarters staff of the Compensation and Pension Service (C&P),
Office of Resource Management, and field resources for both the claims
processing and finance activities. These individuals will serve as the
core team throughout the entirety of our conversion process.
(2) During these initial test conversions, physical record validation
is performed at the 95 percent confidence level. In addition, a number
of automated reports have been and are being developed, to provide
additional oversight for the process. These reports will compare the
state of records pre-and post-conversion, for example, the number of
records submitted for conversion and the number converted; the amount
of payment associated with the records pre-and post-conversion, etc.
Similarly, an automated testing process was developed and is in use to
validate payment processing for application development purposes. This
automated process is also being used to ensure the converted records
conform to the required format for Treasury processing.
(3) As noted previously, a core conversion team has been assembled and
is actively engaged in testing and validation efforts. These
individuals will remain on the core team throughout the conversion
process, continuing to test and validate the conversion logic and
outcome for each scheduled conversion. As incorporated in the IMS,
conversions will follow the release and stabilization of application
functionality in production. Live compensation records are planned for
three or four conversions, depending upon the number of records we
decide to convert in a given "lot". The number of records will be
determined once we complete the current testing phase underway. The
order of regional offices will also be determined at that time, based
on a variety of factors, to include business process changes currently
underway in VBA, as well as workload considerations. Service-connected
death records (Dependency and Indemnity Compensation) will be converted
in one lot, following release of that functionality. Similarly,
existing income-based pension records will be converted in two lots,
once functionality is in place. The decision to convert in two lots is
based on workload considerations for the Pension Maintenance Centers.
VBA and MITRE Corporation believe these efforts will significantly
reduce the risks associated with conversions. During its weekly
meetings, the VETSNET Executive Team formally tracks these risks.
* direct the CIO to document and incorporate the improved processes for
managing risks, requirements, and defects into specific policy and
guidance for the replacement initiative and for future use throughout
VBA; and:
Concur - As part of the centralization of information technology (IT)
within VA, the CIO has undertaken the effort of implementing best
business practices to assist VA in its consolidation. The goal is to
establish application development processes that employ industry best
practices and that have the potential to accelerate the successful
completion of existing projects and ensure continued success in future
projects.
As part of this effort, VETSNET's project management processes,
including risk management, requirements management, and testing
methodology to include the management of defects are under review.
VETSNET project management processes will be incorporated into a set of
standard project management policies, processes and procedures for all
IT projects in VA.
The VETSNET governance model includes the VETSNET Executive Team
consisting of representatives from the technical, project management,
and business lines. The CIO has identified this governance model as the
model for all VA enterprise-wide IT projects. The governance model is
being implemented in other VA priority IT development programs, such as
Financial and Logistics Integrated Technology Enterprise (FLITE).
* direct the replacement project to develop effective results-oriented
performance measures that show changes in efficiency, economy, or
improvements in mission performance, as well as measures of user
satisfaction, and to monitor and report on the progress of the
initiative according to these measures.
Concur - As noted in the report, VBA has implemented a number of
performance metrics to track progress toward transition of all C&P
claims processing and payments from BDN to VETSNET. Some examples were
provided in the body of the report. In addition, VBA recently completed
its periodic Work Measurement Study, which assesses the level of effort
required for various C&P work tasks. One element of this year's study
will assess task time for work processed using VETSNET Award, as well
as through BDN. The other components of the VETSNET suite (i.e., MAP-D
and RBA2000) have already been in full deployment for a number of
years, and exceed functionality currently in BDN. In that regard,
comparisons to BDN processing are not applicable. The efficiencies
gained by these applications are directly related to the reusability of
data and the ability to determine the status of a veteran's claim
without the need to review the physical claims file. This reduces staff
time required to physically locate and review claims files, provides
enhanced customer service, and facilitates the sharing of work across
regional offices to balance workload and provide more timely service to
veterans.
With respect to user satisfaction, while VBA does not formally survey
users as to their level of satisfaction, we have a variety of forums in
place to solicit and receive feedback from the user community. This
includes, for example, an electronic mailbox to receive questions or
comments, and regular "super user" conference calls to discuss issues
and suggestions. In addition, the Special Assistant to the Under
Secretary regularly addresses regional office managers to ensure
information reaches the executive level, and to address any concerns or
comments they may have. As SEI noted, there is a strong "pull" from the
user community for expanded VETSNET functionality. That has been borne
out by the increase in claims processing volume once field stations
were authorized to increase the scope of claims available for
processing. A more formal measure of user satisfaction can be
implemented, but we believe our core measure of project success is
reflected by the metrics currently in place. These metrics track the
current volume of claims processing and beneficiaries being paid
through VETSNET, as well as tracking the activities necessary to
transition all C&P processing off of the BDN.
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Valerie Melvin, (202) 512-6304 or melvinv@gao.gov:
Staff Acknowledgments:
In addition to the individual named above, key contributions were made
to this report by Barbara Oliver, Assistant Director; Nabajyoti
Barkakati; Barbara Collier; Neil Doherty; Matt Grote; Robert Williams;
and Charles Youman.
FOOTNOTES
[1] It also refers to the initiative as the compensation and pension or
C&P replacement system.
[2] GAO, Software Capability Evaluation: VA's Software Development
Process Is Immature, GAO/AIMD-96-90 (Washington, D.C.: June 19, 1996);
Veterans Benefits Modernization: VBA Has Begun to Address Software
Development Weaknesses but Work Remains, GAO/AIMD-97-154 (Washington,
D.C.: Sept. 15, 1997); VA Information Technology: Progress Continues
Although Vulnerabilities Remain, GAO/T-AIMD-00-321 (Washington, D.C.:
Sept. 21, 2000); VA Information Technology: Important Initiatives
Begun, Yet Serious Vulnerabilities Persist, GAO-01-550T (Washington,
D.C.: Apr. 4, 2001); VA Information Technology: Management Making
Important Progress in Addressing Key Challenges, GAO-02-1054T
(Washington, D.C.: Sept. 26, 2002); and Information Technology: VA and
DOD Face Challenges in Completing Key Efforts, GAO-06-905T (Washington,
D.C.: June 22, 2006).
[3] GAO-06-905T.
[4] Kathryn Ambrose, William Novak, Steve Palmquist, Ray Williams, and
Carol Woody, Report of the Independent Technical Assessment on the
Department of Veterans Affairs VETSNET Program (Carnegie Mellon
Software Engineering Institute, September 2005).
[5] The BDN currently runs on aging software: COBOL programs and a
nonrelational database. Analysts have indicated that moving from a
nonrelational database of the BDN type to a more modern relational
database is a challenging task.
[6] This program provides veterans, service members, reservists, and
certain veterans' dependents with educational resources.
[7] To help veterans with service-connected disabilities become
employable and obtain and maintain suitable employment, the program
provides a range of direct and supportive services. These services
include comprehensive evaluation of rehabilitation needs (vocational or
independent living); training and employment services to obtain or
maintain suitable employment; and independent living services, such as
training and specialized equipment to enable independence in the
activities of daily living.
[8] Not all regional offices process all the different types of
benefits. For example, adjustments to pension claims are processed at
three pension maintenance centers in St. Paul, Milwaukee, and
Philadelphia, and education claims are processed only at the regional
offices in Atlanta, Buffalo, Muskogee, and St. Louis. The Philadelphia
Regional Office and Insurance Center has sole responsibility for
insurance benefit processing, and nine Regional Loan Centers administer
the loan guaranty program.
[9] Claims may also be reopened if a veteran provides additional
information on a claim that was denied.
[10] GAO/AIMD-96-90.
[11] Specifically, at the repeatable level of process maturity, basic
project management processes are established to track cost, schedule,
and functionality, and the necessary process discipline is in place to
repeat earlier successes on projects with similar applications.
[12] Kathryn Ambrose, William Novak, Steve Palmquist, Ray Williams, and
Carol Woody, Report of the Independent Technical Assessment on the
Department of Veterans Affairs VETSNET Program (Carnegie Mellon
Software Engineering Institute, September 2005).
[13] CMMI Product Team, CMMI for Development, Version 1.2 (Carnegie
Mellon Software Engineering Institute, August 2006).
[14] As previously noted, there are variations in types of compensation
and pension benefits, including death pensions for survivors of
deceased wartime veterans, burial benefits, Dependency and Indemnity
Compensation to some surviving parents, and benefits paid to third
parties, such as people to whom a veteran has given power of attorney,
medical service providers, and so on. In addition, different forms of
payment must be accommodated (check or electronic funds transfer).
[15] SEI quoted program staff as calling the process "extraordinarily
time consuming," saying that "It takes 5 minutes to get a veteran into
VETSNET and then 5 days to get him back out."
[16] This realignment was approved on February 27, 2007.
[17] The functionalities for the Search and Participant Profile,
formerly a separate application, are now in Share.
[18] Survivor benefits may be paid to survivors of veterans who were
eligible for either compensation or pension benefits, depending on the
circumstances.
[19] Examples of third-party/nonveteran payees include people to whom a
veteran has given power of attorney, medical service providers
designated to receive payments, and so on.
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site (www.gao.gov). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site.
To have GAO e-mail you a list of newly posted products every afternoon,
go to www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125
Washington, D.C. 20548:
Public Affairs:
Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548:
