Information Technology
Actions Needed to Fully Establish Program Management Capability for VA's Financial and Logistics Initiative Gao ID: GAO-10-40 October 26, 2009Since 2005, the Department of Veterans Affairs (VA) has been undertaking an initiative to develop an integrated financial and asset management system known as the Financial and Logistics Integrated Technology Enterprise (FLITE). FLITE is the successor to an earlier initiative known as the Core Financial and Logistics System (CoreFLS) that the department undertook in 1998 and discontinued in 2004 because it failed to support VA's operations. In light of the past performance of CoreFLS and the Office of Management and Budget's designation of FLITE as high risk, GAO was asked to (1) determine the status of pilot system development and (2) evaluate key program management processes, including VA's efforts to institute effective human capital management, develop a reliable program cost estimate, use earned value management (a recognized means for measuring program progress), establish a realistic program schedule, employ effective requirements development and management, and perform independent verification and validation. To do so, GAO reviewed program documentation and interviewed relevant officials.
Contract award and performance of work tasks had been started for one of two planned pilot systems--the Strategic Asset Management system. However, as of mid-September, the project had fallen behind (by 2 months) and the contractor had missed the deadline for initiating and completing planned tasks and delivering work products such as a system security plan. In particular, the contractor had not started 11 of 34 tasks, including conducting a security assessment, and was behind schedule on 16 of the remaining 23 tasks, including analyzing business processes. Program officials generally attributed the delays to VA having insufficient program and acquisition staff to perform necessary activities associated with awarding and executing the pilot contract and to poor project management by the pilot system contractor. A second project--for the Integrated Financial Accounting System pilot--is expected to start in October 2009. VA has taken steps to institute effective management of FLITE; however, the department has not yet fully established capabilities needed to ensure that the program will be successfully implemented. Specifically, VA has (1) recently filled long-standing staff vacancies, and only one program office staff opening remains; (2) not developed a cost estimate that includes total program costs or reconciled its estimate with an independent estimate; (3) not performed key actions necessary for reliable earned value management; (4) not yet established a schedule that is reliable; (5) not identified all mandatory federal financial management system requirements and ensured that system requirements are based on business requirements; and (6) not addressed all of the findings of its independent verification and validation organization in a timely manner. Until VA reconciles its cost estimate, ensures compliance with earned value management system standards, establishes a reliable schedule, ensures all relevant federal and system requirements are identified and traceable, and addresses all independent verification and validation findings, it could continue to experience schedule delays and further increase its risk of not providing the financial and asset management capabilities that users need.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-10-40, Information Technology: Actions Needed to Fully Establish Program Management Capability for VA's Financial and Logistics Initiative
This is the accessible text file for GAO report number GAO-10-40
entitled 'Information Technology: Actions Needed to Fully Establish
Program Management Capability for VA's Financial and Logistics
Initiative' which was released on December 1, 2009.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Committee on Veterans' Affairs, House of Representatives:
United States Government Accountability Office:
GAO:
October 2009:
Information Technology:
Actions Needed to Fully Establish Program Management Capability for
VA's Financial and Logistics Initiative:
GAO Highlights:
Highlights of GAO-10-40, a report to the Committee on Veterans'
Affairs, House of Representatives.
Why GAO Did This Study:
Since 2005, the Department of Veterans Affairs (VA) has been
undertaking an initiative to develop an integrated financial and asset
management system known as the Financial and Logistics Integrated
Technology Enterprise (FLITE). FLITE is the successor to an earlier
initiative known as the Core Financial and Logistics System (CoreFLS)
that the department undertook in 1998 and discontinued in 2004 because
it failed to support VA‘s operations. In light of the past performance
of CoreFLS and the Office of Management and Budget‘s designation of
FLITE as high risk, GAO was asked to (1) determine the status of pilot
system development and (2) evaluate key program management processes,
including VA‘s efforts to institute effective human capital management,
develop a reliable program cost estimate, use earned value management
(a recognized means for measuring program progress), establish a
realistic program schedule, employ effective requirements development
and management, and perform independent verification and validation. To
do so, GAO reviewed program documentation and interviewed relevant
officials.
What GAO Found:
Contract award and performance of work tasks had been started for one
of two planned pilot systems”the Strategic Asset Management system.
However, as of mid-September, the project had fallen behind (by 2
months) and the contractor had missed the deadline for initiating and
completing planned tasks and delivering work products such as a system
security plan. In particular, the contractor had not started 11 of 34
tasks, including conducting a security assessment, and was behind
schedule on 16 of the remaining 23 tasks, including analyzing business
processes. Program officials generally attributed the delays to VA
having insufficient program and acquisition staff to perform necessary
activities associated with awarding and executing the pilot contract
and to poor project management by the pilot system contractor. A second
project”for the Integrated Financial Accounting System pilot”is
expected to start in October 2009.
VA has taken steps to institute effective management of FLITE; however,
the department has not yet fully established capabilities needed to
ensure that the program will be successfully implemented. Specifically,
VA has:
* recently filled long-standing staff vacancies, and only one program
office staff opening remains;
* not developed a cost estimate that includes total program costs or
reconciled its estimate with an independent estimate;
* not performed key actions necessary for reliable earned value
management;
* not yet established a schedule that is reliable;
* not identified all mandatory federal financial management system
requirements and ensured that system requirements are based on business
requirements; and;
* not addressed all of the findings of its independent verification and
validation organization in a timely manner.
Until VA reconciles its cost estimate, ensures compliance with earned
value management system standards, establishes a reliable schedule,
ensures all relevant federal and system requirements are identified and
traceable, and addresses all independent verification and validation
findings, it could continue to experience schedule delays and further
increase its risk of not providing the financial and asset management
capabilities that users need.
What GAO Recommends:
GAO is making recommendations aimed at improving program management. In
written comments on a draft of this report, VA concurred with the
recommendations and identified actions to address them.
View [hyperlink, http://www.gao.gov/products/GAO-10-40] or key
components. For more information, contact Valerie C. Melvin at (202)
512-6304 or melvinv@gao.gov.
[End of section]
Contents:
Letter:
Background:
Pilot Project Is Behind Schedule:
VA Has Recently Filled FLITE Program Staff Positions but Has Not Yet
Fully Established Other Program Management Capabilities:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Comments from the Department of Veterans Affairs:
Appendix III: GAO Contacts and Staff Acknowledgments:
Table:
Table 1: Description of FLITE Program Office Entities and Supporting VA
Organizations:
Figures:
Figure 1: Simplified View of FLITE Components:
Figure 2: Relationship of VA's Oversight and Management Structure to
FLITE Program:
Figure 3: Simplified FLITE Program Structure and Supporting VA
Organizations:
Figure 4: FLITE Timeline:
Abbreviations:
ANSI: American National Standards Institute:
CoreFLS: Core Financial and Logistics System:
EIA: Electronic Industries Alliance:
EVM earned value management:
FLITE: Financial and Logistics Integrated Technology Enterprise:
FMS: Financial Management System:
FSIO: Financial Systems Integration Office:
IFCAP: Integrated Funds Distribution, Control Point Activity,
Accounting, and Procurement:
IFAS: Integrated Financial Accounting System:
IT: information technology:
MQAS: Management Quality Assurance Service:
OIG: Office of Inspector General:
OMB: Office of Management and Budget:
PDO: program director's office:
PMO: program management office:
SAM: Strategic Asset Management:
SEI: Software Engineering Institute:
SPAWAR: Space and Naval Warfare Systems Command:
VA: Department of Veterans Affairs:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
October 26, 2009:
The Honorable Bob Filner:
Chairman:
The Honorable Steve Buyer:
Ranking Member:
Committee on Veterans' Affairs:
House of Representatives:
Since 2005, the Department of Veterans Affairs (VA) has been
undertaking an initiative to develop an integrated financial management
and information system known as the Financial and Logistics Integrated
Technology Enterprise (FLITE). FLITE is the successor to an earlier
initiative that the department undertook known as the Core Financial
and Logistics System (CoreFLS). After having reportedly spent more than
$249 million on its development, the department discontinued CoreFLS
because the pilot system failed to support VA's operations.
According to the department, FLITE is intended to fulfill the critical
need for a modernized and integrated financial and asset management
capability. Such a capability would support VA's strategic goal to
deliver world-class service to veterans and their families through
effective communication and management of people, technology, business
processes, and financial resources. In early 2007, the Office of
Management and Budget (OMB) designated FLITE as a high-risk information
technology investment.[Footnote 1]
In light of VA's past performance with CoreFLS and the designation of
FLITE as high risk, you requested that we (1) determine the status of
pilot system development and (2) evaluate key program management
processes for the initiative, including VA's efforts to institute
effective human capital management, develop a reliable program cost
estimate, use earned value management (EVM), establish a realistic
program schedule, employ effective requirements development and
management, and perform independent verification and validation.
To accomplish our objectives, we reviewed relevant program
documentation and interviewed appropriate VA and contractor officials.
Specifically, to determine the status of FLITE pilot system
development, we reviewed documentation such as program management plans
and project status reports. To evaluate key FLITE program management
processes, we compared VA's activities to plans and best practices.
We performed our work at the Department of Veterans Affairs
headquarters in Washington, D.C., from November 2008 to October 2009 in
accordance with generally accepted government auditing standards. Those
standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives. See appendix I for a more
complete description of our objectives, scope, and methodology.
Background:
In recognition of their service to our country, the Department of
Veterans Affairs (VA) provides medical care, benefits, social support,
and lasting memorials to veterans and their families. It is the second-
largest federal department with approximately 250,000 employees. In
fiscal year 2008, VA reported incurring $97 billion in obligations for
its overall operations.
VA provides services to veterans and their families primarily through
its three line administrations:
* The Veterans Health Administration operates a nationwide network of
154 hospitals, 995 outpatient clinics, 135 community living centers, 49
residential rehabilitation treatment programs, and 232 community-based
counseling centers.
* The Veterans Benefits Administration provides assistance and benefits
such as veterans' compensation, survivors' benefits, and employment
assistance through 57 veterans' benefits regional offices.
* The National Cemetery Administration manages 130 national cemeteries.
To support its services to veterans and their families, VA relies on an
assortment of business systems, including 13 different systems that
currently support its asset and financial management. However, the
department has long recognized that its business systems and processes
are inefficient and do not effectively support the department's
mission. For example, according to the department,
* systems are not integrated,
* manual entry that involves labor-intensive accounting processes is
required,
* business processes are not standardized,[Footnote 2] and:
* processes and systems require multiple entry of business information
and result in untimely financial reporting.
Since fiscal year 1991, the department has reported on the need for an
integrated financial management system and has reported financial
management system functionality as a material weakness.[Footnote 3]
This weakness continues to exist because many of VA's systems are
outdated, leading to inefficiencies in the reliable, timely, and
consistent preparation, processing, and analysis of financial
information for the department's consolidated financial statements. To
address this weakness and to improve stewardship and accountability
over its resources, VA has for over a decade been pursuing improvements
in its business processes and replacement of its existing financial and
asset management systems with an integrated financial management
system.
Assessments of CoreFLS:
The department's first attempt to replace its financial and asset
management systems, CoreFLS, began in 1998. The goal of this
modernization effort was to develop a single system to integrate the
many financial and asset management systems used across the department.
VA had planned to complete CoreFLS in March 2006; however, it
terminated development of the system in July 2004 after CoreFLS pilot
tests determined it did not fully support the department's operations
and that the initiative suffered from significant project management
weaknesses. According to VA's Office of Inspector General (OIG), the
department had obligated about $249 million of the $472 million that
had been budgeted for the initiative by the time of its termination.
[Footnote 4]
Following the failed CoreFLS pilot tests, VA hired Carnegie Mellon
University's Software Engineering Institute (SEI) to perform an
independent assessment of the project. In June 2004, SEI identified a
number of management and technical deficiencies that had undermined the
success of the initiative.[Footnote 5] SEI identified multiple findings
related to problematic technical and functional execution, as well as
poor management execution. Technical and functional problems included
CoreFLS's inability to perform essential financial management
functions, security weaknesses, and usability. Management problems were
identified in the areas of acquisition and program management, business
process re-engineering, and transition planning.
In addition, in August 2004, VA's OIG reported multiple findings
related to CoreFLS deployment, such as inadequate training, inability
to monitor fiscal and acquisition operations, inaccurate data, and
project management and security weaknesses.[Footnote 6]
Further, in August 2007, VA's Management Quality Assurance Service
(MQAS) summarized findings from four CoreFLS reviews completed between
August 2005 and August 2006.[Footnote 7] Among the findings, MQAS
identified numerous fiscal and contract administration issues resulting
from poor administrative internal controls such as improper
reimbursements of task orders and travel expenses.
Collectively, VA identified 141 findings related to problems with the
CoreFLS initiative, which the department categorized into functional
areas of responsibility such as acquisition management, organizational
change management, program management, and systems engineering.
[Footnote 8] In a subsequent effort to capture lessons learned[Footnote
9] and ensure that mistakes from CoreFLS would not be repeated in later
initiatives, VA developed a repository, in which it aggregated the
findings from the three independent reviews of the initiative.
Establishment and Management of the FLITE Program:
In September 2005, in a subsequent effort to replace its financial and
asset management systems, VA began work on FLITE. In this regard, the
department undertook activities related to planning and requirements
development. For example, the department:
* documented business requirements and business processes,
* initiated coordination for reporting and financial data warehouse
development,
* conducted a market analysis of providers with the software and
hosting capability to support VA's existing financial management
system,
* established key personnel requirements to provide program support and
awarded a program support contract, and:
* started developing numerous planning documents (e.g., program
management plan, acquisition plan, and concept of operations).
According to VA's planning documents, FLITE is a multiyear development
effort that is projected to deliver a fully operational system by 2014
at a total estimated cost of $608.7 million.[Footnote 10] The overall
objectives of the FLITE program are to:
* implement accessible and enterprise-level standardized business
processes that result in increased efficiencies and enhanced internal
controls;
* provide VA executives and managers with timely, transparent financial
and asset management information to make and implement effective
policy, management, stewardship, and program decisions; and:
* provide business data and information in a secure, shareable, open,
and efficient manner to facilitate a service-oriented atmosphere.
The FLITE program includes two main projects to acquire the integrated
asset and financial management system: an asset management component,
referred to as the Strategic Asset Management (SAM) initiative, and the
financial management component, referred to as the Integrated Financial
Accounting System (IFAS). The program also includes a third project, to
acquire a data warehouse that is intended to provide financial and
logistics data reporting and analysis.
* SAM is intended to consolidate the asset and inventory management
functions and the associated work management processes currently
performed by multiple legacy applications into an advanced integrated
system. It is to be the system of record for VA's physical assets and
perform asset and inventory management, real property management,
information technology (IT) asset management, and work order and
project management functions currently performed by multiple legacy
applications. VA has chosen IBM's Maximo Enterprise Asset Management
software suite[Footnote 11] to implement these capabilities.
* IFAS is to be the financial, procurement, and accounting management
component, and, together with SAM, is intended to replace VA's legacy
Financial Management System (FMS) and the Integrated Funds
Distribution, Control Point Activity, Accounting, and Procurement
(IFCAP)[Footnote 12] system.
* The data warehouse is projected to consolidate data from multiple
transactional systems, primarily SAM and IFAS, for improved reporting,
querying, and analysis capability. It is also intended to allow users
to run larger and more complex queries and reports faster, without
affecting the performance of the source systems.
Figure 1 shows a simplified view of the program's components.
Figure 1: Simplified View of FLITE Components:
[Refer to PDF for image: illustration]
Strategic Asset Management and Integrated Financial Accounting System
interact, and both provide material to the Data Warehouse.
Strategic Asset Management:
Facility management;
Inventory management;
Asset management;
Requisitioning and procurement;
Item management;
Real property management;
Work order management.
Integrated Financial Accounting System:
Project accounting;
Accounting;
Debt management;
Reporting;
Travel;
Payment management;
Vendor management;
Requisitioning and procurement;
Fixed assets;
Budget execution.
Source: GAO analysis of VA data.
[End of figure]
The program is a collaborative effort between the Assistant Secretary
for Information and Technology, who serves as VA's Chief Information
Officer, and the Assistant Secretary for Management, who serves as VA's
Chief Financial Officer. Various groups within VA have different roles
and responsibilities for overseeing and managing programs. Figure 2
depicts the relationships between these oversight groups and the FLITE
program.
Figure 2: Relationship of VA's Oversight and Management Structure to
FLITE Program:
[Refer to PDF for image: illustration]
FLITE Program Office; interaction with:
FLITE Oversight Board; Budget execution and other issues interaction
with:
* Budgeting and Near Term Issues Board; and;
* Programming and Long Term Issues Board;
Interaction with:
Strategic Management Council; interaction with:
VA Executive Board; Budget execution and other issues interaction with:
Information Technology Leadership Board.
Budgeting and Near Term Issues Board; and;
Programming and Long Term Issues Board; interaction with:
Information Technology Leadership Board.
Source: GAO analysis of VA data.
[End of figure]
The roles and responsibilities of each oversight group are as follows:
* The VA Executive Board provides the Secretary of Veterans Affairs
with a forum for discussing programs with senior leadership before
decisions are made.
* The Strategic Management Council makes recommendations about programs
to the VA Executive Board.
* The Programming and Long Term Issues Board focuses on long term
multiyear program planning.
* The Budgeting and Near Term Issues Board is responsible for
overseeing budget formulation and execution activities.
* The IT Leadership Board is responsible for adjudicating inter-and
intraboard issues about programs that cannot be resolved between the
Programming and Long Term Issues and Budgeting and Near Term Issues
Boards.
* The FLITE Oversight Board is responsible for making decisions
regarding FLITE business requirements, policies, and standards.
The FLITE Program Office is responsible for overseeing and coordinating
all aspects of the program. The office is responsible for performing
these functions through the Program Director's Office (PDO), which is
responsible for business requirements and processes, and the IT Program
Management Office (PMO), which is responsible for technical solutions.
Project teams are responsible for managing SAM, IFAS, and the data
warehouse. In addition, other VA organizations provide the office with
quality assurance, acquisition, and technology support. These program-
specific and VA supporting organizations are depicted in figure 3.
Figure 3: Simplified FLITE Program Structure and Supporting VA
Organizations:
[Refer to PDF for image: illustration]
Systems Quality Assurance Service:
Center for Acquisition Innovation:
Austin Information Technology Center:
(supporting organizations for):
FLITE Program Office:
* Program Director‘s Office;
* IT Program Management Office;
IFAS project team:
SAM project team:
Data warehouse:
Also provide support to FLITE Program Office.
Source: GAO analysis of VA data.
[End of figure]
Table 1 describes the components that comprise the program office and
supporting VA organizations.
Table 1: Description of FLITE Program Office Entities and Supporting VA
Organizations:
Title: FLITE Program Office;
Description: This office includes the PDO and the IT PMO. Together,
these offices are responsible for overseeing and coordinating all
aspects of the FLITE program, including systems engineering,
organizational change management, training, program management,
communications, and risk and investment management, and for directing
multidisciplinary efforts of VA and contractor personnel to accomplish
various tasks.
Title: IFAS project team;
Description: This team is responsible for replacing VA's current FMS
and portions of the IFCAP system.
Title: SAM project team;
Description: This team is responsible for consolidating the asset and
inventory management, real property management, work order, and project
management functionality currently performed by multiple legacy
applications.
Title: Data warehouse;
Description: This project office has not been formally defined by the
program office.
Title: Center for Acquisition Innovation;
Description: This entity is responsible for providing acquisition
services to procure supplies and services to support FLITE.
Title: Austin Information Technology Center;
Description: This entity is responsible for providing IT enterprise
solutions to support information technology needs, including providing
design and conceptual support for IFAS and systems hosting services,
application administration, and operational support for SAM and the
financial reporting data warehouse.
Title: Systems Quality Assurance Service;
Description: This entity is responsible for managing and providing
oversight of independent verification and validation activities and
processes employed throughout the FLITE, SAM, and IFAS system
development life cycles.
Source: GAO analysis of VA data.
[End of table]
Planned Implementation of FLITE:
VA is employing a multiphase approach for both the SAM and IFAS
projects, which are to be implemented by contractors using commercial
off-the-shelf systems. Specifically, these components are to be
implemented through sequenced acquisitions and phased deployment and
integration. The systems are planned to be implemented initially at
pilot sites and subsequently refined and validated at beta sites before
national deployment. The purpose of the pilot phase is to perform a
final validation of the selected commercial off-the-shelf system and
associated business processes in a production environment, gain
experience in deploying the system, and obtain acceptance from the user
community. The beta phase is to further hone the rollout capabilities
by deploying the system to a limited number of sites that span the
range of VA's organizational environments. Following the beta phase,
the department plans to incorporate lessons learned from both phases
and produce a set of repeatable processes that can be employed during
national deployment of the system.
For SAM, the department's plans include implementation at one pilot
site and 15 beta sites. The SAM pilot contractor is to evaluate and
analyze VA's business processes and requirements for a fit with the
Maximo software's capabilities and produce updated business process
documents based on the department's needs. Also, the contractor is to
train the users at the pilot site, as well as provide operations and
maintenance and help desk services. The pilot phase is expected to last
for an estimated 12 months. Subsequent to the pilot, the department
plans to deploy SAM at 15 VA beta sites over a period of approximately
12 months. The component is expected to be deployed nationwide over 21
months, with its completion expected by May 2013.
Plans for IFAS include implementing the FMS replacement at five pilot/
beta sites and implementing the IFCAP replacement at two pilot/beta
sites. The IFAS pilot phase is currently scheduled to begin in the
first quarter of fiscal year 2010. The department plans to deploy this
component in two separate subphases over approximately 4 years. The
first subphase, which will replace FMS with a commercial off-the-shelf
financial management system, is expected to take about 2 years to
complete. The second subphase, which is planned to be done concurrently
with the first phase, will replace IFCAP with the IFAS commercial off-
the-shelf financial management system and is expected to take just over
4 years to complete.
VA's approach to implementing the data warehouse calls for developing
the warehouse after the underlying data structures of SAM and IFAS are
defined and stabilized. The department expects to complete the data
warehouse in the first quarter of fiscal year 2014.
Figure 4 depicts the program timeline, from program proposal through
deployment of the SAM, IFAS, and data warehouse components.
Figure 4: FLITE Timeline:
[Refer to PDF for image: timeline]
Program initiation activities:
FLITE proposed: FY 2005, Q4;
PDO established: FY 2007, Q2;
IT PMO established: FY 2007, Q3.
SAM:
Pilot implementation: FY 2009, Q3 to FY 2010, Q4;
Acquisition planning and source selection: FY 2009, Q3 to FY 2010, Q4;
Beta: FY 2010, Q3 to FY 2012, Q1;
National deployment: FY 2012, Q1 to FY 2013, Q3.
IFAS: FMS replacement:
Acquisition planning and source selection: FY 2009, Q2 to FY 2009, Q4;
Pilot: FY 2010, Q1 to FY 2010, Q4;
National deployment: FY 2011, Q1 to FY 2012, Q4.
IFAS: IFCAP replacement:
Development: FY 2010, Q1 to FY 2011, Q1;
Pilot: FY 2011, Q1 to FY 2011, Q2;
Beta: FY 2011, Q2 to FY 2011, Q4;
National deployment: FY 2011, Q4 to FY 2014, Q1.
Data warehouse:
Acquisition planning and source selection: FY 2009, Q4 to FY 2011, Q1;
Development: FY 2011, Q1 to FY 2012, Q1;
Pilot/Beta: FY 2012, Q1 to FY 2012, Q3;
Deployment: FY 2012, Q3 to FY 2014, Q1.
Source: GAO analysis of VA data.
[End of figure]
In 2009, the program office undertook various activities, including
issuing the IFAS request for proposals (February), awarding a program
management support contract (March), awarding the SAM pilot project
contract and beginning work (April), issuing a request for proposals
for independent verification and validation support (July), and
initiating planning for the data warehouse (September).
According to program officials, as of September 2, 2009, the department
had spent approximately $90.8 million on FLITE. This amount included
$73.0 million for about 40 contract actions on behalf of the program
office: $28.5 million for program management and technical support,
$27.8 million for software licenses, $10.9 million for the SAM project,
$5.5 million for analyses (e.g., requirements analyses), and $0.3
million for other program activities (e.g., training).
Prior GAO and OIG Reviews of FLITE:
Both we and VA's OIG have previously reported on the FLITE initiative.
In a September 2008 report, we noted that key planning documents
related to the initiative lacked specificity and detail, and that VA
had not addressed all the findings in the CoreFLS findings repository.
[Footnote 14] We recommended that VA add more specificity and details
to key planning documents, such as the concept of operations and work
breakdown structure, and address all findings in the CoreFLS findings
repository to minimize risk to the successful implementation of FLITE.
In response to our report, as of September 2009, VA had updated key
planning documents and reported that it had taken actions that
addressed all of the findings identified in the repository.
In September 2009, VA's OIG reported on VA's effectiveness in managing
the FLITE program.[Footnote 15] The office noted, among other things,
that although program managers had taken steps toward addressing the
CoreFLS findings, deficiencies similar to those found in CoreFLS were
also evident in FLITE. For example, OIG reported that FLITE program
functions were not fully staffed.
Pilot Project Is Behind Schedule:
VA and its contractor have begun one of the two planned pilot systems--
the SAM component. Specifically, in April 2009, the department
contracted with General Dynamics Information Technology Inc. to
implement Maximo at the VA Medical Center in Milwaukee, Wisconsin.
Among the activities the contractor is expected to perform are
analyzing business processes, documenting requirements, configuring
Maximo, and performing system tests.
As of mid-September 2009, VA reported that, with the contractor only 5
months into the 1-year time period planned to complete the pilot, the
project had fallen 2 months behind schedule. This 2-month schedule slip
was a consequence of the contractor falling behind in its efforts to
perform tasks and deliver products that are necessary to implement the
pilot system. Specifically, of the 34 tasks planned to be undertaken by
mid-September, the contractor reported that 11 had not yet been
started--including conducting a security assessment and predeployment
testing--and that of 23 tasks that had been initiated, 16 were behind
schedule. For example, among the tasks that the contractor noted as
behind schedule were analysis of security requirements, business
process analysis, and system configuration. Regarding the seven
remaining tasks, two had reportedly been completed and five were
identified as being on schedule. The contractor reported that it had
completed a requirements traceability matrix and was on schedule with
respect to starting up a project management office, performing
organizational change management activities, and developing quality
assurance and control programs.
Further, with respect to the delivery of products, the contractor
reported that it had delivered only 7 of 37 products due by mid-
September. The SAM project management plan and the requirements
management plan were among the products that were delivered. Products
that had not yet been delivered included the Maximo system
configuration document, intended to provide detailed instructions to
enable a trained Maximo administrator to incorporate all VA
configuration requirements, and the SAM system security plan.
VA attributed the project being 2 months behind schedule to a shortage
of FLITE program office human capital resources and poor project
management by the contractor. Specifically, according to the program
director, the program did not have the personnel it needed during the
initial months of the SAM pilot project to provide the contractor with
the information it needed to make planned progress. Regarding the
contractor's project management, VA stated that the contractor:
* provided a project manager who did not possess the skills necessary
to deliver quality and timely products,
* delayed hiring a project scheduler and used an initial project
scheduling approach that was incorrect,
* used an ineffective and inefficient approach to analyzing VA's
business processes and underestimated the time needed to obtain a
thorough understanding of the processes, and:
* underestimated the effort necessary to configure a database server
used in the pilot's development environment.
In mid-September, the FLITE program director stated that the department
had filled almost all of the program office vacancies and that the
contractor had begun to improve its project management weaknesses.
Nevertheless, according to the program director, while the department
does not expect any further delays in completing the SAM pilot, it does
not expect to recover the 2-month schedule slippage that has already
occurred. As a result, the department projected completion of the pilot
in 14 months, instead of 12 months as originally planned.
Additionally, activities are under way to initiate the IFAS pilot.
Specifically, the department issued a request for proposals for a pilot
contractor in February 2009. A contract for the IFAS pilot is planned
for award in late October 2009.
VA Has Recently Filled FLITE Program Staff Positions but Has Not Yet
Fully Established Other Program Management Capabilities:
VA has taken steps to institute effective management of the FLITE
program; however, the department has not yet fully established key
capabilities needed to ensure that system components will be
implemented as planned. The department recently made progress toward
filling program office staff vacancies. Nonetheless, more work is
needed to fully establish program management capabilities in areas that
are important to the development of its integrated financial and
logistics system. Until VA completes efforts to develop and reconcile
its cost estimate; comply with EVM system standards; implement
performance measures for its schedule; include all relevant federal and
system requirements; and perform effective, independent verification
and validation, it will have increased risk that FLITE will experience
cost overruns and schedule delays and will not provide the capabilities
that users need.
VA Recently Filled Vacant Program Office Positions:
Our past work has found that the success of federal programs depends on
having effective strategic human capital management and, in particular,
having the right number of people with the right mix of knowledge and
skills.[Footnote 16] VA has recently taken steps to fill long-standing
vacancies in the FLITE program that have adversely impacted the
program's ability to maintain schedules. Specifically, in mid-
September, the program acquired 36 staff, filling 111 of 112 required
positions. According to the Acting Assistant Secretary for the Office
of Management, vacant FLITE program positions were filled by
individuals who were reassigned, detailed, or newly hired when the VA
Deputy Secretary became aware of the program's need for staff
resources. As a result of the department's recent actions to fill
vacant positions, the office should be better positioned to effectively
manage the program.
VA's FLITE Cost Estimate Is Incomplete:
Federal guidelines[Footnote 17] recommend that operations and
maintenance costs over the entire estimated life cycle of an investment
be included in a cost estimate. Inclusion of these costs over the time
period corresponding to the life of the investment is encouraged by the
federal government's guidance for managing capital assets because such
costs are a key element for establishing the total cost of ownership.
Further, our Cost Estimating and Assessment Guide[Footnote 18]
describes effective cost-estimating practices, including performance of
a risk and uncertainty analysis and development of an independent cost
estimate that provides an unbiased test of whether the program's
estimate is reasonable. Typically, the two estimates are reconciled.
In August 2008, the FLITE program office developed a program cost
estimate of $608.7 million for fiscal years 2007 through 2014--when
FLITE systems are expected to achieve full operational capability.
However, the office did not project operations and maintenance costs
over the entire estimated life of the FLITE investment, and it did not
perform a risk and uncertainty analysis as encouraged by best
practices. Program officials stated that they did not consider life-
cycle operations and maintenance costs in their estimate because they
wanted to capture only the cost for developing the FLITE system up to
its full operational capability. Also, rather than perform a risk and
uncertainty analysis of their own, the program office planned to rely
on risk analyses by an outside entity, the Department of the Navy Space
and Naval Warfare Systems Command (SPAWAR), that the department engaged
to generate a risk adjusted independent cost estimate.
Completed in April 2009, the SPAWAR estimate identified costs totaling
$1.899 billion for the life of the program and included $1.061 billion
of estimated operations and maintenance costs for fiscal year 2015
through fiscal year 2024, which represented the entire estimated life
of the initiative. According to SPAWAR officials, they used our Cost
Estimating and Assessment Guide as the method for developing the
independent estimate.[Footnote 19] Also, to align with VA's estimate,
SPAWAR used standardized cost elements and definitions to develop a
probability-based estimate of $837.8 million for fiscal years 2007
through 2014. This estimate was $229.1 million higher than the
department's estimate for this period. The department's estimate was
not derived based on standardized cost elements and probability-driven
risk and uncertainty costs assessments.
VA has not yet reconciled its cost estimate with SPAWAR's estimate.
According to department officials, a significant number of end-of-
fiscal-year procurement requests and the department's prioritization of
IT acquisitions had affected the timing of plans to reconcile the
estimates. Program officials stated that they intend to incorporate
federal polices and requirements, as well as address funding,
budgetary, or contractual issues necessitated by the reconciliation.
According to the officials, the department plans to initiate this work
in December 2009 and to complete it in March 2010. Until the
reconciliation is completed, effective administration of FLITE program
planning, budgeting, acquisition, and performance management activities
could be jeopardized if accurate cost data are not available to guide
the execution of these functions. Completion of the reconciliation,
which should include estimated operations and maintenance costs for the
life of the program, is essential to increase the reliability of the
FLITE cost estimate and reduce the risk that acquisition plans,
budgets, and performance management activities will be unsuccessful or
inefficient.
VA Has Not Performed Key Actions Necessary for Reliable EVM for FLITE:
OMB and department policies require major programs to use EVM to
measure and report program progress.[Footnote 20] EVM is a tool for
measuring program progress by comparing the value of work accomplished
with the amount of work expected to be accomplished. Such a comparison
permits actual performance to be evaluated, based on variances from the
cost and schedule baselines--collectively referred to as a performance
measurement baseline. Identification of significant variances and
analysis of their causes helps program managers determine the need for
corrective actions. Before EVM analysis can be reliably performed,
developing a credible cost estimate is necessary to provide program
managers with a clear definition of the cost, schedule, and risks
associated with the scope of work planned. These inputs are then used
to create a performance measurement baseline for EVM analysis. In
addition, federal policy requires that systems used to collect and
process EVM data be compliant with the industry standard developed by
the American National Standards Institute (ANSI) and Electronic
Industries Alliance (EIA), ANSI/EIA Standard 748.[Footnote 21]
Program officials have recognized the importance of reliable EVM and
finalized the FLITE Program Measurement Earned Value Management Plan in
August 2009. The plan identified roles and responsibilities, applicable
policy and guidance, and the program's EVM implementation approach.
According to program officials, programwide earned value reporting that
will include government, program management support, and SAM project
work activities is expected to begin in October 2009.
However, while VA plans to begin reporting earned value performance in
October 2009, a reliable cost estimate, which is necessary for EVM
reporting, is not expected to be completed by that time. Specifically,
as noted earlier, the department has not reconciled its cost estimate
for the program with SPAWAR's independent cost estimate. Program
officials do not expect reconciliation of the cost estimate to begin
until 2 months after earned value reporting is scheduled to begin.
Additionally, VA officials have not yet ensured that all EVM systems
for FLITE are certified for compliance with ANSI/EIA Standard 748.
[Footnote 22] These compliance assessments are necessary to demonstrate
the capability of providing reliable cost and schedule information for
earned value reporting. Specifically, the compliance assessment for the
SAM pilot contractor's system has not yet been completed. While program
officials did not provide information that explained why a compliance
assessment of the contractor's EVM system had not yet been completed,
they stated that the contractor has a plan to obtain system
certification. This activity is not expected to be complete until
January 2010, 3 months after earned value reporting for the program is
scheduled to begin.
Until the agency has completed reconciling its cost estimate and
ensured that contractors comply with EVM system industry standards, VA
will have an increased risk of reporting and managing the program based
on unreliable performance data.
VA Has Not Established a Reliable Program Schedule for FLITE:
GAO's Cost Estimating and Assessment Guide[Footnote 23] states that the
success of a program depends in part on having a reliable schedule that
realistically depicts the program's work activities to a specific
degree of detail, reasonably indicates when those work activities will
occur, estimates how long they will take to complete, and shows how the
work activities are related to each other. For example, a reliable
schedule would indicate when one work activity depends upon the
completion of another before it can start and that required resources
(e.g., labor and materials) are assigned to all activities. Overall,
the schedule provides the road map for the orderly execution of a
program, helps identify and address potential problems, provides a
baseline to gauge progress, and promotes accountability.
VA has not yet established a schedule for the program that is reliable.
Program officials stated that they baselined (i.e., formally
established) an integrated master schedule in January 2009. However, in
the program's August and September 2009 Risk & Issues reports, program
officials noted that the integrated master schedule was not complete
and did not represent all program requirements. The reports also
identified that the SAM pilot schedule (a key component of the overall
program schedule) did not include sufficient detail to trace project
tasks to contract requirements.
Our analysis also concluded that the schedule was unreliable and noted
that, in addition to issues VA identified with the program schedule,
the integrated master schedule did not include key program management
activities for reconciling the program cost estimate and implementing
EVM, nor did it identify resources assigned to activities already under
way or expected to start in the near future. Further, the schedule did
not identify all dependencies and activities and did not break down all
dependencies and activities to a sufficient level of detail to measure
performance. Program officials acknowledged these deficiencies and
stated that program management staffing shortages and delays in
receiving a reliable project schedule from the SAM contractor have
affected their ability to produce a reliable schedule for the program.
They stated that in July 2009, they began working with stakeholders to
address schedule issues and plan to improve the reliability of their
schedule by finalizing a revised integrated master schedule by October
2009.
Until VA completes a revised integrated master schedule that includes
all key program activities broken down to a sufficient level of detail
and identifies all resources and dependencies, the program's efforts to
measure progress and identify potential problems will be impaired, and
the program will have increased risk of missing critical milestones for
system delivery.
VA Has Not Identified and Effectively Managed All FLITE Requirements:
According to SEI guidance, the requirements for a system should
describe the functionality needed to meet user needs and perform as
intended in the operational environment.[Footnote 24] Federal agencies
also must ensure that their financial management systems comply with
federal standards mandated by the Federal Financial Management
Improvement Act of 1996.[Footnote 25] Also according to SEI guidance,
an organization can ensure system requirements are based on business
requirements by tracking the requirements from inception of the project
and agreement on a specific set of business requirements to development
of the system requirements, detailed design, code implementation, and
test cases necessary for validating the requirements. Requirements must
be traceable forward and backward (i.e., bidirectional traceability)
through the development life cycle. Traceability helps reduce the risks
of fielding a system that does not meet the needs of its users,
incurring schedule delays, and increasing costs.
VA has developed an initial set of 1,700 requirements that need to be
addressed in the development of the SAM and IFAS components. FLITE
requirements consist of core financial and procurement requirements
related to the IFAS project, as well as inventory, supply, and real
property requirements related to the SAM project. To develop the
initial set of requirements for FLITE, program officials stated that
they analyzed VA's current and planned financial and asset management
business processes and researched the Financial Systems Integration
Office's (FSIO)[Footnote 26] Core Financial System Requirements and
Inventory, Supplies, and Materials System Requirements publications.
[Footnote 27] The initial set of requirements was further defined and
refined by obtaining input from consultants and VA financial and asset
management experts. The department included all mandatory core
financial system requirements in its IFAS requirements but did not
include all mandatory inventory, supplies, and materials requirements
in its SAM requirements. For example, our analysis showed that VA did
not include requirements for recording whether goods and services are
accepted or rejected and for performing a systematic review and follow-
up of overdue in-transit items.[Footnote 28] Program officials
explained that they did not include these requirements because they had
not determined whether the requirements were applicable to the SAM
project. The officials agreed to incorporate the missing requirements.
VA is also in the process of finalizing its real property requirements
for the SAM beta phase and still plans to develop additional
requirements related to procurement for IFAS. Further, the department
is identifying data analysis and reporting requirements for the data
warehouse.[Footnote 29]
Regarding requirements traceability, SAM project officials acknowledge
that mapping system requirements to the related business requirements
is fundamental to effective requirements management. However, according
to FLITE officials, they made a business decision not to establish
bidirectional traceability between the business and system requirements
included in the SAM pilot request for proposals. Instead, they decided
to require the pilot contractor to establish traceability between the
business and system requirements after the contractor analyzes and
refines the requirements. According to the officials, the contractor
plans to complete these tasks by December 2009. In addition, program
officials stated that they plan to establish bidirectional traceability
between IFAS business and system requirements under the IFAS
implementation contract scheduled to be awarded in October 2009. In
this regard, the IFAS request for proposals states that the
implementation contractor will be required to finalize IFAS
requirements, as well as maintain and document the traceability of all
requirements to design, develop, integrate, and test specifications.
As the department develops its requirements, it is important that all
relevant and applicable federal financial management system
requirements be identified and incorporated into the program's
requirements to ensure its planned financial management systems meet
users' needs and comply with applicable federal laws. Further, until
they have established traceability between the business and system
requirements, VA will not be positioned to know whether the system
requirements are complete and effectively address each business
requirement.
VA Has Begun Independent Verification and Validation of the FLITE
Program, but All Findings Have Not Been Addressed in a Timely Manner:
According to recognized industry standards[Footnote 30] and our prior
reports,[Footnote 31] the purpose of independent verification and
validation is to provide an independent review of system processes and
products to ensure that quality standards are being met. As we have
previously noted, the use of independent verification and validation is
a recognized best practice for large and complex system development and
acquisition programs such as FLITE and involves an independent
organization conducting unbiased reviews of processes, products, and
results to verify and validate that they meet stated requirements and
standards. VA policy recognizes the importance of addressing
independent verification and validation results in a timely manner.
Recognizing the importance of independent verification and validation,
the department's Systems Quality Assurance Service was tasked with
performing independent verification and validation activities for the
FLITE program.[Footnote 32] In April 2009, this organization developed
a Software Quality Assurance Plan[Footnote 33] to guide independent
verification and validation activities for the program. The plan was
developed consistent with industry standards and generally contained
the required elements. The plan also outlined reviews that would be
performed by the Systems Quality Assurance Service, including product
(e.g., program and project deliverables), process, internal controls,
test readiness, and production readiness reviews. In addition, the
Systems Quality Assurance Service is responsible for advising and
assisting with the program's implementation of a suite of tools to
support requirements management, change management, risk, and test
management.
Independent verification and validation of the FLITE program has been
focused primarily on the review of program and project deliverables.
According to program officials, as of September 2009, the Systems
Quality Assurance Service had reviewed 30 FLITE work products and
provided findings and recommendations to document owners.[Footnote 34]
Out of 1,064 total findings, 947 (approximately 89 percent) had been
fully addressed by the program or had been identified as obsolete by
the Systems Quality Assurance Service. Of the 117 remaining findings,
59 had been addressed but had not yet been reflected in revised
documents, and 58 required additional attention. Of the 58 findings and
recommendations that remained open, the SAM pilot site readiness plan
accounted for 18 that were identified in December 2008. According to
the Systems Quality Assurance Service, these findings focused on the
need for consistency with other project documentation, clarity in the
timing of site activities, and incorporation of planned site-level
activities into the program work breakdown structure. In addition,
according to department officials, the FLITE acquisition strategy has
two findings and recommendations that were identified in December 2008
and that remain to be addressed. These findings are related to VA's
approach for acquiring SAM and IFAS integration support and the
program's focus on front-end acquisition activities, rather than full
life cycle acquisition processes. Unknown or incomplete system
integration requirements may result in significant rework and adversely
impact the program's cost, schedule, and quality.
According to FLITE program officials, they have not had the human
capital resources they need to address all the independent verification
and validation findings and recommendations in a timely manner. As a
result, independent verification and validation findings that highlight
important program issues (e.g., determining an approach for integrating
SAM and IFAS) have not received the attention that they need. As
discussed earlier, the staff resources recently added could help
address the program's inability to focus sufficient attention on
resolving findings from initial independent verification and validation
activities. It remains unclear whether the program office will be
positioned to efficiently resolve findings raised when the scope of
independent verification and validation activities expands to include
system testing and production readiness reviews, which affect the
extent to which FLITE components will meet stated requirements and
quality standards.
Conclusions:
The pilot for VA's new asset management system has experienced a 2-
month schedule delay just 5 months after award of the contract. While
VA has recently taken steps to address the staffing shortages that have
substantially contributed to this delay, it has not yet fully
established the management capability necessary for FLITE to be
successful. For example, the department's program cost estimate did not
represent total program costs, nor has the estimate been reconciled
with an independent estimate--a process that could increase its
reliability. Further, it has not conducted EVM that is needed to ensure
the reliability of the department's programwide reporting on the
initiative. Also, VA has not yet made revisions that are needed to
increase the reliability of the program's integrated master schedule.
In addition, the requirements for the two major program systems, SAM
and IFAS, do not yet address all the functions expected of federal
asset management and financial management systems. Finally, key
findings from independent reviews of the program have not been fully
addressed on a timely basis.
As a consequence, the department is faced with significant challenges
in implementing FLITE's pilot systems as planned, while simultaneously
working to fully establish program management capabilities. Program
officials recognize the importance of reconciling their cost estimate,
ensuring compliance with EVM system standards, establishing a reliable
schedule, ensuring all relevant federal and system requirements are
identified and traceable, and addressing all independent verification
and validation findings. Further, they have stated that they plan to
take such actions. However, just as program officials needed the
department's support in filling long-standing program office vacancies,
the full support of the department's top management is critical to
ensuring that planned actions are executed. If the program is not
effective in addressing its management weaknesses, the department
increases the risk of repeating its unsuccessful earlier attempt to
modernize the department's financial and logistics systems.
Recommendations for Executive Action:
To help guide and ensure successful completion of FLITE, the Secretary
of Veterans Affairs should direct and ensure that the Assistant
Secretary for Management and the Assistant Secretary for Information
and Technology take the following five actions:
* Improve the reliability of the program cost estimate by ensuring that
the estimate includes system operations and maintenance costs and that
the estimate is reconciled with the independent cost estimate.
* Improve the reliability of program earned value management reporting
by ensuring that contractor earned value management systems comply with
industry standards.
* Complete a revised integrated master schedule that includes all key
program activities, including reconciliation of the program cost
estimate and implementation of earned value management, and identifies
all resources and dependencies.
* Ensure that all relevant and applicable federal financial management
system requirements are included in FLITE's requirements and establish
and maintain requirements traceability.
* Ensure that all comments from independent verification and validation
reviews are addressed.
Agency Comments and Our Evaluation:
The VA Chief of Staff provided written comments on a draft of this
report. In its comments, the department concurred with our
recommendations and described actions to address them. For example, the
department stated that it plans to reconcile the FLITE program cost
estimate with the independent cost estimate by the second quarter of
fiscal year 2010; ensure that future contractors' EVM systems comply
with industry standards and begin an independent review of the
program's EVM compliance by the first quarter of 2010; and include the
reconciled program cost estimate in the integrated master schedule by
the third quarter of fiscal year 2010. Further, the department stated
that it plans to validate the completeness of FLITE requirements by mid-
November 2009 and ensure that outstanding comments from independent
verification and validation reviews are addressed by mid-December 2009.
If the recommendations are properly implemented, they should better
position VA to effectively manage the FLITE program.
The department also provided a technical comment, which we have
addressed in the report as appropriate. The department's written
comments are reproduced in appendix II.
As agreed with your offices, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days
from the date of this letter. At that time, we will send copies of the
report to interested congressional committees, the Secretary of
Veterans Affairs, and other interested parties. In addition, the report
will be available at no charge on our Web site at [hyperlink,
http://www.gao.gov].
If you or your staffs have questions about this report, please contact
Valerie C. Melvin at (202) 512-6304 or melvinv@gao.gov, or Kay L. Daly
at (202) 512-9095 or dalykl@gao.gov. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last
page of this report. Key contributors to this report are listed in
appendix III.
Signed by:
Valerie C. Melvin:
Director, Information Management and Human Capital Issues:
Signed by:
Kay L. Daly:
Director, Financial Management and Assurance:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
As requested, the objectives of our study were to (1) determine the
status of the Financial and Logistics Integrated Technology
Enterprise‘s (FLITE) pilot system development and (2) evaluate key
program management processes, including the Department of Veterans
Affairs‘ (VA) efforts to institute effective human capital management,
develop a reliable program cost estimate, use earned value management,
establish a realistic program schedule, employ effective requirements
development and management, and perform independent verification and
validation.
To determine the status of the pilot system development, we:
* obtained and analyzed program documentation, including program
management plans, contracts, schedules, briefing slides, meeting
minutes, and project status reports to identify from these reports the
planned FLITE pilot activities and deliverables and determined to what
extent these tasks had been completed; and;
* supplemented department program documentation and our analyses by
interviewing department and contractor officials, such as the program
director, and observing project status meetings.
We also evaluated VA‘s progress toward implementing our prior
recommendations related to adding specificity and details to key
planning documents by comparing updated documents, including the
Program Management Plan and Strategic Asset Management (SAM) Concept of
Operations to prior versions.
To evaluate key program management processes, we:
* compared program staffing plans with the program‘s staffing resource
reports to determine the extent to which program human capital needs
have been met;
* compared the program cost estimate and estimating activities to
Office of Management and Budget guidance and GAO‘s Cost Estimating and
Assessment Guide[Footnote 35] to determine the estimate‘s completeness
and the effectiveness of the estimating activities;
* reviewed department documentation, such as the program‘s plan for
earned value management implementation, and compared them to federal
policy and GAO‘s Cost Estimating and Assessment Guide to determine the
department‘s preparedness for conducting reliable earned value
management;
* reviewed the program schedule and compared it to planned activities,
deliverables, and practices described in GAO‘s Cost Estimating and
Assessment Guide to assess the schedule‘s reliability;
* analyzed program documentation, including the department‘s business
requirements, concept of operations for FLITE, traceability matrix, and
requirements management plan, to determine the extent to which they
reflect practices such as those recognized by SEI[Footnote 36] and
include federal financial management system requirements; and;
* reviewed program documentation, such as the software quality
assurance plan, quality management plan, and technical review reports,
to determine the extent to which the program has addressed independent
verification and validation findings.
We conducted this performance audit at VA headquarters in Washington,
D.C., from November 2008 through October 2009 in accordance with
generally accepted government auditing standards. Those standards
require that we plan and perform the audit to obtain sufficient,
appropriate evidence to provide a reasonable basis for our findings and
conclusions based on our audit objectives. We believe that the evidence
obtained provides a reasonable basis for our findings and conclusions
based on our audit objectives.
[End of section]
Appendix II: Comments from the Department of Veterans Affairs:
The Secretary Of Veterans Affairs:
Washington:
October 19, 2009:
Ms. Valerie Melvin:
Director:
Information Management and Human Capital Issues:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Ms. Melvin:
The Department of Veterans Affairs (VA) has reviewed the Government
Accountability Offices (GAO) draft report, Information Technology;
Actions Needed to Fully Establish Program Management Capability for
VA's Financial and Logistics Initiative (GAO-10-40) and generally
agrees with GAO's conclusions and concurs with GAO's recommendations to
the Department.
The enclosure specifically addresses GAO's recommendations and provides
additional comments to the draft report. VA appreciates the opportunity
to comment on your draft report. A similar response has been sent to
Ms. Kay Daly.
Sincerely,
Signed by:
John R. Gingrich:
Chief of Staff:
Enclosure:
[End of letter]
The Secretary Of Veterans Affairs:
Washington:
October 19, 2009:
Ms. Kay Daly:
Director:
Financial Management and Assurance:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Ms. Daly:
The Department of Veterans Affairs (VA) has reviewed the Government
Accountability Office's (GAO) draft report, Information Technology:
Actions Needed to Fully Establish Program Management Capability for
VA's Financial and Logistics Initiative (GAO-10-40) and generally
agrees with GAO's conclusions and concurs with GAO's recommendations to
the Department.
The enclosure specifically addresses GAO's recommendations and provides
additional comments to the draft report. VA appreciates the opportunity
to comment on your draft report. A similar response has been sent to
Ms. Valerie Melvin.
Sincerely,
Signed by:
John R. Gingrich:
Chief of Staff:
Enclosure:
[End of letter]
Enclosure:
Departments of Veterans Affairs (VA) Comments to Government
Accountability Office (GAO) Draft Report, Information Technology:
Actions Needed to Fully Establish Program Management Capability for
VA's Financial and Logistics Initiative (GAO-10-40):
GAO recommendation:
To help guide and ensure successful completion of FLITE, the Secretary
of Veterans Affairs should direct and ensure that the Assistant
Secretary for Management and the Assistant Secretary for Information
and Technology take the following five actions:
Recommendation 1: Improve the reliability of the program cost estimate
by ensuring that the estimate includes system operations and
maintenance costs and that the estimate is reconciled with the
independent cost estimate.
VA response: Concur. Financial and Logistics Integrated Technology
Enterprise (FLITE) completed a program manager's cost estimate that
formed the basis of the e300 submission in December 2008 and
subsequently issued a short-term contract for an independent cost
estimate. The Department of Veterans Affairs (VA) fully intends to
complete the cost estimation work for the ELITE program by reconciling
it with the independent cost estimate provided by SPAWAR in July 2009.
VA is contracting for a follow-on independent cost analysis which will
include reconciliation of the Program Managers and independent cost
estimates to validate assumptions. The Government expects to complete
the cost estimate reconciliation by Q2 FY 2010.
Recommendation 2: Improve the reliability of program earned value
management reporting by ensuring that contractor earned value
management systems comply with industry standards.
VA response: Concur. VA is committed to continually improving its
program and project management activities, to include the reliability
of earned value management (EVM) reporting. To increase this
reliability, VA will:
* Conduct thorough validation activities to ensure all future FLITE
contractors' EVM systems comply with the applicable American National
Standards Institute (ANSI) standards, specifically ANSI/EIA Standard
748 and VA Earned Value Application guide and Directive 6061.
* On a consistent basis, conduct integrated baseline reviews on all
ELITE contracts with an EVM requirement.
* Obtain an independent compliance review of the FLITE program's
implementation of EVM. This activity will begin in Q1 FY 2010.
* Ensure all Requests for Proposals issued by the FLITE program include
mandatory EVM compliance requirements.
* Ensure that when awarding contracts, there is formal proof of a
contractors compliance with ANSI standards for use of an EVM system.
* Ensure agreement with any applicable contractor, before and after
award as appropriate, in order to verify the Performance Measurement
Baseline (PMB). The PMB is the basis against which all performance will
be measured and EVM data generated.
* Maintain a change management process to ensure integrity of the cost
and schedule baseline, This includes submission of formal change
requests, based upon pre-determined thresholds and parameters, which
will be reviewed by the overall FLITE Change Control Board.
Recommendation 3: Complete a revised integrated master schedule that
includes all key program activities, including reconciliation of the
program cost estimate and implementation of earned value management and
identifies all resources and dependencies.
VA response: Concur. VA has included tasks for creating and maintaining
integrated master schedules (IMS) in all its applicable contracts. The
FLITE program has an existing overall IMS, which is supported by the
project levels schedules. This IMS includes all key program activities
and is fully developed.
The FLITE program IMS will include the reconciled program cost estimate
by Q3 FY 2010.
Recommendation 4: Ensure that all relevant and applicable federal
financial management system requirements are included in FLITE's
requirements and establish and maintain requirements traceability.
VA response: Concur. The Strategic Asset Management (SAM) system
Project Office believes that the applicable federal financial
management system requirements were included when the original SAM
business requirements were developed. However, to ensure and validate
this assumption, the SAM Project Office will cross reference these
requirements with SAM requirements.
Any requirement that may have been omitted will be added to the SAM
Business Requirements document following established change control
procedures. Any federal financial management system requirement
determined not to be applicable to VA will be documented as to the
reason why. The validation process will be accomplished by November 13,
2009.
Recommendation 5: Ensure that all comments from independent
verification and validation reviews are addressed.
VA response: Concur. The 59 findings that were addressed are being
included into the appropriate documents during their planned updates.
The 58 findings requiring additional attention are being reviewed for
document revision update. The planned updates and the review process
will be completed by December 15, 2009.
In addition, VA's Systems Quality Assurance Service will provide a
biweekly status report for use by the FLITE program and project
managers. The report will outline program and project quality
assurance, testing activities and status. In addition all program and
project level product reviews will be included in a current period
activity table and a cumulative status tracking format which will allow
for easy progress assessment for follow-up action.
[End of section]
Appendix III: GAO Contacts and Staff Acknowledgments:
GAO Contacts:
Valerie C. Melvin, (202) 512-6304 or melvinv@gao.gov:
Kay L. Daly, (202) 512-9095 or dalykl@gao.gov:
Staff Acknowledgments:
In addition to the contacts named above, key contributions to this
report were made by Mark T. Bird, Assistant Director; Michael S.
LaForge, Assistant Director; Heather A. Collins; Neil J. Doherty;
Rebecca Eyler; David A. Hong; Jacqueline K. Mai; Yvonne D. Moss; Robert
L. Williams, Jr.; and Leonard E. Zapata.
[End of section]
Footnotes:
[1] Investments that are designated as high risk require special
attention from the highest level of agency management and oversight
authorities due to size, complexity, or nature of the risk of the
project.
[2] VA currently does not have standardized systems or processes for
managing physical assets, supply inventories, and related work order
management across the department. While VA uses some common inventory,
procurement, and asset management applications, each VA facility
operates a customized version of each system.
[3] The material weakness in financial management system functionality
is linked to VA‘s outdated legacy financial systems, impacting VA‘s
ability to prepare, process, and analyze financial information that is
reliable, timely, and consistent. Legacy system deficiencies
necessitated significant manual workarounds and a large number of
general ledger adjustments, increasing the risk of processing errors
and misstatements in the financial statements.
[4] VA OIG, Issues at VA Medical Center Bay Pines, Florida and
Procurement and Deployment of the Core Financial and Logistics System
(CoreFLS), 04-01371-177 (Washington, D.C., Aug. 11, 2004).
[5] Carnegie Mellon SEI, Report of the Independent Technical Assessment
of the Department of Veterans Affairs CoreFLS Program (June 2004).
[6] VA OIG, Issues at VA Medical Center Bay Pines, Florida and
Procurement and Deployment of the Core Financial and Logistics System
(CoreFLS), 04-01371-177.
[7] VA MQAS, VA Lessons Learned: Findings and Recommendations Summary
from CoreFLS Reviews, 07-04-SAD002 (Austin, Tex., Aug. 1, 2007).
[8] The aggregated list of 141 lessons learned included 80 lessons
learned from SEI, 22 from OIG, and 39 from MQAS. VA officials
subsequently reduced the total number of lessons learned to 103 by
eliminating duplicate findings.
[9] The use of lessons learned is a principal component of an
organizational culture committed to continuous improvement. Sharing
such information serves to communicate acquired knowledge more
effectively and to ensure that beneficial information is factored into
planning, work processes, and activities. Lessons learned can be based
on positive experiences or on negative experiences that result in
undesirable outcomes.
[10] VA developed their initial cost estimate for FLITE in August 2008.
In April 2009, an independent cost estimate adjusted for program risks
and uncertainties estimated FLITE costs to be $837.8 million.
[11] Maximo is a Web-based asset management suite used to manage the
complete life cycle of strategic assets, including planning,
procurement, deployment, tracking, maintenance, and retirement.
[12] IFCAP is a decentralized procurement, funds control, and front-end
accounting system that complements the FMS functionality. It is used at
the VA Medical Centers and certain Regional and Administrative Offices.
IFCAP integrates functions of Fiscal Service, Acquisition and
Logistics, and other VA Medical Center services that request supplies
and services for VA. There are more than 150 stand-alone instances of
IFCAP across VA.
[13] SAM and IFAS are intended to be transactional systems but will not
be optimized for query and reporting activities, which is the role that
the data warehouse is intended to fill.
[14] GAO, Veterans Affairs: Additional Details Are Needed in Key
Planning Documents to Guide the New Financial and Logistics Initiative,
[hyperlink, http://www.gao.gov/products/GAO-08-1097] (Washington, D.C.:
Sept. 22, 2008).
[15] VA OIG, Department of Veterans Affairs: Audit of FLITE Program
Management‘s Implementation of Lessons Learned, 09-01467-216 (Sept. 16,
2009).
[16] For example, our prior work has shown negative cost and schedule
implications for complex services acquisitions at the Department of
Homeland Security that did not have adequate staff. See GAO, Department
of Homeland Security: Better Planning and Assessment Needed to Improve
Outcomes for Complex Service Acquisitions, [hyperlink,
http://www.gao.gov/products/GAO-08-263] (Washington, D.C.: Apr. 22,
2008).
[17] Executive Office of the President, Office of Management and
Budget, Circular No. A-11, Part 7, Planning, Budgeting, Acquisition,
and Management of Capital Assets (Washington, D.C., June 2008). The OMB
guidelines state that risk adjusted life cycle costs include the
overall estimated cost over the time period corresponding to the life
of the investment, including periodic and continuing costs of
operations and maintenance.
[18] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for
Developing and Managing Capital Program Costs, [hyperlink,
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009).
[19] To develop probability-based estimated costs leading to a risk-
adjusted independent cost estimate for the FLITE program, SPAWAR used a
risk-based cost-estimating technique called Latin Hypercube sampling.
Latin Hypercube sampling is a sampling technology designed to ensure
high forecast efficiency and accurately recreate the input distribution
through sampling. Sampling is forced to represent values in each
interval and, thus, is forced to recreate the input probability
distribution. For the independent cost evaluation, SPAWAR used 50,000
iterations of the model to generate estimates from the model to enhance
statistical efficiency. SPAWAR also obtained additional inputs from the
FLITE PMO and obtained specific responses to independent cost
evaluation team questions from the VA PMO, and subsequently
reconsidered and modified some cost-related assumptions that VA used in
creating their estimate.
[20] OMB issued policy guidance (M-05-23) to agency chief information
officers on improving technology projects that includes requirements
for reporting performance to OMB using EVM (August 2005). VA, VA Earned
Value Management System, VA Directive 6061, (February 2006).
[21] See OMB, Capital Programming Guide, II.2.4, Establishing an Earned
Value Management System. Reflected in the Federal Acquisition
Regulation (FAR), 48 C.F.R. subpart 34.2.
[22] Typically, an independent organization conducts the compliance
review of an EVM system. Upon successful completion of the review,
system acceptance should be documented, showing how each of the 32
ANSI/EIA Standard 748 guidelines have been satisfied.
[23] [hyperlink, http://www.gao.gov/products/GAO-09-3SP].
[24] SEI is a federally funded research and development center whose
objective is to provide leadership in software engineering and the
transition of new software engineering technology into practice.
[25] Specifically, the Federal Financial Management Improvement Act of
1996 states that financial management systems must comply substantially
with financial management systems requirements located in a series of
publications entitled Federal Financial Management Systems
Requirements, which specifies the mandatory functional and technical
requirements that agencies‘ systems must meet.
[26] FSIO is responsible for coordinating the work related to federal
financial management systems requirements.
[27] FSIO, Core Financial System Requirements, OFFM-NO-0106
(Washington, D.C., January 2006) and Inventory, Supplies, and Materials
System Requirements, JFMIP-SR-03-02 (Washington, D.C., August 2003).
[28] Recording the acceptance or rejection of goods and services, as
well as resolving overdue in-transit items, are useful when evaluating
the performance of suppliers and in reconciling asset management data
with financial data such as payment and general ledger posting
activities. Additionally, recording the date goods and services are
accepted is a critical data element for ensuring an agency complies
with the Prompt Payment Act. (Prompt Payment Act codified at 31 U.S.C.
§§ 3901-3904 and implemented at 5 C.F.R. 1315.)
[29] According to the VA FLITE Real Property Strategy Working Paper, VA
needs to perform additional analysis of VA business needs to determine
if more real property business requirements are needed. Additionally,
the FLITE program office plans to develop the additional requirements
related to procurement, data analyses, and reporting during the IFCAP
Replacement Development phase of the IFAS project and the data
warehouse scheduled to begin in October 2009 and September 2010,
respectively.
[30] Institute of Electrical and Electronics Engineers, Inc., IEEE
Standard for Software Verification and Validation, IEEE Std 1012-2004
(New York, N.Y., June 8, 2005).
[31] GAO, Office of Personnel Management: Improvements Needed to Ensure
Successful Retirement Systems Modernization, [hyperlink,
http://www.gao.gov/products/GAO-08-345] (Washington, D.C.: Jan. 31,
2008) and Homeland Security: First Phase of Visitor and Immigration
Status Program Operating, but Improvements Needed, [hyperlink,
http://www.gao.gov/products/GAO-04-586] (Washington, D.C.: May 11,
2004).
[32] VA‘s Systems Quality Assurance Service is organizationally
independent of the FLITE Program and reports directly to the Office of
Business Oversight and Office of Management for the escalation of
significant issues.
[33] Standards have been developed by the Institute of Electrical and
Electronics Engineers (IEEE) to guide the development of software
quality assurance plans, which typically include information on roles
and responsibilities, tasks, metrics, software reviews, system tests,
and independent verification and validation tools that will be utilized
to support the software quality assurance process.
[34] The document owner is responsible for managing the document review
process; this includes identifying appropriate reviewers, consolidating
adjudicated comments, and determining the acceptability of work
products.
[35] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for
Developing and Managing Capital Program Costs, [hyperlink,
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009).
[36] SEI is a federally funded research and development center whose
objective is to provide leadership in software engineering and in the
transition of new software engineering technology into practice.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
