Information Technology
Management Improvements Are Essential to VA's Second Effort to Replace Its Outpatient Scheduling System Gao ID: GAO-10-579 May 27, 2010The Department of Veterans Affairs (VA) provides medical care, disability compensation, and vocational rehabilitation to veterans. The Veterans Health Administration (VHA)--a component of VA--provides care to over 5 million patients in more than 1,500 facilities. VHA relies on an outpatient scheduling system that is over 25 years old. In 2000, VHA began the Scheduling Replacement Project to modernize this system as part of a larger departmentwide modernization effort called HealtheVet. However, in February 2009, VA terminated a key contract supporting the project. GAO was asked to (1) determine the status of the Scheduling Replacement Project, (2) determine the effectiveness of VA's management and oversight of the project, and (3) assess the impact of the project on VA's overall implementation of its HealtheVet initiative. To do so, GAO reviewed project documentation and interviewed VA and contractor officials.
After spending an estimated $127 million over 9 years on its outpatient scheduling system project, VA has not implemented any of the planned system's capabilities and is essentially starting over. Of the total amount, $62 million was expended for, among other things, project planning, management support, a development environment, and equipment. In addition, the department paid an estimated $65 million to the contractor selected to develop the replacement scheduling application. However, the application software had a large number of defects that VA and the contractor could not resolve. As a result, the department terminated the contract, determined that the system could not be deployed, and officially ended the Scheduling Replacement Project on September 30, 2009. VA began a new initiative that it refers to as HealtheVet Scheduling on October 1, 2009. As of April 2010, the department's efforts on this new initiative had largely consisted of evaluating whether to buy or custom build a new scheduling application. VA's efforts to successfully complete the Scheduling Replacement Project were hindered by weaknesses in several key project management disciplines and a lack of effective oversight that, if not addressed, could undermine the department's second effort to replace its scheduling system: (1) VA did not adequately plan its acquisition of the scheduling application and did not obtain the benefits of competition. (2) VA did not ensure requirements were complete and sufficiently detailed to guide development of the scheduling system. (3) VA performed system tests concurrently, increasing the risk that the system would not perform as intended, and did not always follow its own guidance, leading to software passing through the testing process with unaddressed critical defects. (4) VA's project progress and status reports were not reliable, and included data that provided inconsistent views of project performance. (5) VA did not effectively identify, mitigate, and communicate project risks due to, among other things, staff members' reluctance to raise issues to the department's leadership. (6) VA's various oversight boards had responsibility for overseeing the Scheduling Replacement Project; however, they did not take corrective actions despite the department becoming aware of significant issues. The impact of the scheduling project on the HealtheVet initiative cannot yet be determined because VA has not developed a comprehensive plan for HealtheVet that, among other things, documents the dependencies among the projects that comprise the initiative. VA officials stated that the department plans to document the interdependencies, project milestones, and deliverables in an integrated master schedule as part of a project management plan that is expected to be completed by June 2010. In the absence of such a plan, the impact of the scheduling project's failure on the HealtheVet program is uncertain. Secretary of Veterans Affairs direct the Chief Information Officer to take six actions to improve key processes, including acquisition management, system testing, and progress reporting, which are essential to the department's second outpatient scheduling system effort. In written comments on a draft of this report, VA generally concurred with GAO's recommendations and described actions to address them.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Valerie C. Melvin Team: Government Accountability Office: Information Technology Phone: (202) 512-6304GAO-10-579, Information Technology: Management Improvements Are Essential to VA's Second Effort to Replace Its Outpatient Scheduling System
This is the accessible text file for GAO report number GAO-10-579
entitled 'Information Technology: Management Improvements Are
Essential to VA's Second Effort to Replace Its Outpatient Scheduling
System' which was released on June 28, 2010.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Ranking Member, Committee on Veterans' Affairs, U.S.
Senate:
United States Government Accountability Office:
GAO:
May 2010:
Information Technology:
Management Improvements Are Essential to VA's Second Effort to Replace
Its Outpatient Scheduling System:
GAO-10-579:
GAO Highlights:
Highlights of GAO-10-579, a report to the Ranking Member, Committee on
Veterans‘ Affairs, U.S. Senate.
Why GAO Did This Study:
The Department of Veterans Affairs (VA) provides medical care,
disability compensation, and vocational rehabilitation to veterans.
The Veterans Health Administration (VHA)”a component of VA”provides
care to over 5 million patients in more than 1,500 facilities. VHA
relies on an outpatient scheduling system that is over 25 years old.
In 2000, VHA began the Scheduling Replacement Project to modernize
this system as part of a larger departmentwide modernization effort
called HealtheVet. However, in February 2009, VA terminated a key
contract supporting the project. GAO was asked to (1) determine the
status of the Scheduling Replacement Project, (2) determine the
effectiveness of VA‘s management and oversight of the project, and (3)
assess the impact of the project on VA‘s overall implementation of its
HealtheVet initiative. To do so, GAO reviewed project documentation
and interviewed VA and contractor officials.
What GAO Found:
After spending an estimated $127 million over 9 years on its
outpatient scheduling system project, VA has not implemented any of
the planned system‘s capabilities and is essentially starting over. Of
the total amount, $62 million was expended for, among other things,
project planning, management support, a development environment, and
equipment. In addition, the department paid an estimated $65 million
to the contractor selected to develop the replacement scheduling
application. However, the application software had a large number of
defects that VA and the contractor could not resolve. As a result, the
department terminated the contract, determined that the system could
not be deployed, and officially ended the Scheduling Replacement
Project on September 30, 2009. VA began a new initiative that it
refers to as HealtheVet Scheduling on October 1, 2009. As of April
2010, the department‘s efforts on this new initiative had largely
consisted of evaluating whether to buy or custom build a new
scheduling application.
VA‘s efforts to successfully complete the Scheduling Replacement
Project were hindered by weaknesses in several key project management
disciplines and a lack of effective oversight that, if not addressed,
could undermine the department‘s second effort to replace its
scheduling system:
* VA did not adequately plan its acquisition of the scheduling
application and did not obtain the benefits of competition.
* VA did not ensure requirements were complete and sufficiently
detailed to guide development of the scheduling system.
* VA performed system tests concurrently, increasing the risk that the
system would not perform as intended, and did not always follow its
own guidance, leading to software passing through the testing process
with unaddressed critical defects.
* VA‘s project progress and status reports were not reliable, and
included data that provided inconsistent views of project performance.
* VA did not effectively identify, mitigate, and communicate project
risks due to, among other things, staff members‘ reluctance to raise
issues to the department‘s leadership.
* VA‘s various oversight boards had responsibility for overseeing the
Scheduling Replacement Project; however, they did not take corrective
actions despite the department becoming aware of significant issues.
The impact of the scheduling project on the HealtheVet initiative
cannot yet be determined because VA has not developed a comprehensive
plan for HealtheVet that, among other things, documents the
dependencies among the projects that comprise the initiative. VA
officials stated that the department plans to document the
interdependencies, project milestones, and deliverables in an
integrated master schedule as part of a project management plan that
is expected to be completed by June 2010. In the absence of such a
plan, the impact of the scheduling project‘s failure on the HealtheVet
program is uncertain.
What GAO Recommends:
GAO is recommending that the Secretary of Veterans Affairs direct the
Chief Information Officer to take six actions to improve key
processes, including acquisition management, system testing, and
progress reporting, which are essential to the department‘s second
outpatient scheduling system effort. In written comments on a draft of
this report, VA generally concurred with GAO‘s recommendations and
described actions to address them.
View [hyperlink, http://www.gao.gov/products/GAO-10-579] or key
components. For more information, contact Valerie Melvin at (202) 512-
6304 or melvinv@gao.gov.
[End of section]
Contents:
Letter:
Background:
VA Ended the Outpatient Scheduling System Project without Delivering
Expected Capabilities and Has Begun a New Initiative:
Scheduling Replacement Project Was Hindered by Weaknesses in Key
Management Capabilities:
Impact of Scheduling Replacement Project Failure on HealtheVet Program
is Uncertain:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Comments from the Department of Veterans Affairs:
Appendix III: GAO Staff Contact and Acknowledgments:
Figure:
Figure 1: Timeline of Scheduling Replacement Project Events:
Abbreviations:
ANSI: American National Standards Institute:
CASU: Cooperative Administrative Support Units:
CIO: Chief Information Officer:
COTS: commercial off-the-shelf:
CPI: cost performance index:
EVM: earned value management:
EIA: Electronic Industries Alliance:
FAR: Federal Acquisition Regulation:
GSA: General Services Administration:
IT: information technology:
OED: Office of Enterprise Development:
OI&T: Office of Information & Technology:
OMB: Office of Management and Budget:
PMAS: Program Management Accountability System:
SPI: schedule performance index:
SwRI: Southwest Research Institute:
VA: Department of Veterans Affairs:
VHA: Veterans Health Administration:
VistA: Veterans Health Information Systems and Technology Architecture:
VISN: Veterans Integrated Service Network:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
May 27, 2010:
The Honorable Richard Burr:
Ranking Member:
Committee on Veterans' Affairs:
United States Senate:
Dear Senator Burr:
The Department of Veterans Affairs (VA) is responsible for providing a
variety of services to veterans, including medical care, disability
compensation, and vocational rehabilitation. The Veterans Health
Administration (VHA)--a component of VA--manages one of the largest
health care systems in the United States, providing health care to
more than 5 million patients in more than 1,500 facilities. To carry
out its daily operations in providing health care to patients, VHA
relies on the Veterans Health Information Systems and Technology
Architecture (VistA), an information system comprised of multiple
applications that include health provider applications; management and
financial applications; registration, enrollment, and eligibility
applications; health data applications; and information and education
applications.
As part of VistA, VHA operates an electronic outpatient scheduling
system that automates all aspects of the outpatient appointment
process, including the scheduling of patients and the generation of
managerial reports. However, this system is over 25 years old, is
inefficient in coordinating care between different sites, and has
contributed to increasing wait times for appointments as the number of
VA patients has grown in recent years. In 2000, VHA began an
initiative to modernize the system--the Scheduling Replacement
Project--with the goal of creating an outpatient scheduling
application that would improve veterans' access to health care. The
Scheduling Replacement Project was to result in the first system to be
deployed as part of a larger, departmentwide initiative to modernize
the department's health information system, known as HealtheVet.
[Footnote 1] However, after 9 years of attempting to produce a new
outpatient scheduling system, VA terminated a key contract supporting
the Scheduling Replacement Project in February 2009 and ended the
project in September 2009. According to the program manager, the
department then began a new project to develop a scheduling system in
October 2009.
At your request, we conducted a review of VA's efforts toward
replacing its scheduling system. Specifically, our objectives were to
(1) determine the status of the scheduling project, (2) determine the
effectiveness of VA's management and oversight of the project, and (3)
assess the impact of the project on the department's overall
implementation of its health information system modernization
initiative--HealtheVet.
To accomplish our objectives, we reviewed relevant project
documentation and interviewed appropriate VA and contractor officials.
Specifically, to determine the status of the project, we reviewed the
project management plan and project status reports. To determine the
effectiveness of VA's management and oversight of the project, we
compared VA's plans and activities in key areas of management controls
to best practices, as well as the department's own policies and
guidance. To assess the impact of the scheduling project on VA's
overall implementation of its health information system modernization
initiative, we reviewed documentation such as briefings from
HealtheVet planning meetings and interviewed officials about the
status of the HealtheVet initiative.
We conducted this performance audit from May 2009 through May 2010 in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives. See appendix I
for a more complete description of our objectives, scope, and
methodology.
Background:
As part of VA's mission, VHA is to serve the needs of America's
veterans and their families (spouses and children) by providing
primary care, specialized care, and related medical and social support
services. VHA provides health services through more than 1,500 sites
of care, including 153 hospitals, 995 outpatient clinics, 135
community living centers, and 232 Vet Centers. It employs more than
15,000 physicians and serves more than 5 million patients at these
sites of care each year.
To carry out its daily operations in providing health care to veterans
and their families, VHA relies on an outpatient appointment scheduling
system that is part of the department's current electronic health
information system, known as VistA. However, according to the
department, the current scheduling system has a number of limitations
that impede its effectiveness, including:
* Appointment activity resides at multiple medical centers, making it
difficult to retrieve all of a patient's health care history.
* Clinicians must maintain multiple calendars to account for the
various services they provide.
* Appointments and ancillary services are not linked, resulting in the
inability to associate medical data with appointments.
* Access to multiple sites is required to make appointments, resulting
in inefficient coordination of care between facilities.
Accordingly, in 2000, VHA initiated a project to replace the existing
scheduling system. In doing so, it envisioned that the new scheduling
system would provide benefits for the department, including:
* a single enterprise database that would allow all appointments to be
viewed, regardless of the point of care;
* calendars that would include sequential appointment settings;
* long-term appointment lists that would track and remind staff of
future appointments; and:
* ancillary service links that would allow for automated updates to
appointment cancellations.
VA originally planned to deploy the new outpatient scheduling system
to an initial site by December 2004 and nationally by June 2006. In
August 2002, the department had estimated that the total cost to
develop and deploy the new system across all VHA facilities would be
about $59 million.
History of VA's Scheduling System Initiative:
VHA began the scheduling replacement initiative in October 2000, at
which time it began to identify business requirements for the new
system. It also issued a request for proposals, seeking interested
Veterans Integrated Service Networks (VISN)[Footnote 2] to partner
with its Office of Information to conduct a business process
reengineering effort and replace the VistA scheduling system with a
commercial off-the-shelf (COTS) application. In January 2001, VHA
selected VISNs 16 and 17, representing Texas and the south-central
United States, respectively, to perform these tasks. Additionally, the
Muskogee, Oklahoma medical center, part of VISN 16, was the planned
location for the initial deployment of the new scheduling system.
The VISNs used a pre-existing Cooperative Administrative Support Units
(CASU)[Footnote 3] contract to obtain the services of the Southwest
Research Institute (SwRI) to support the project.[Footnote 4] The
statement of work included tasks to develop business information flow
models and information system technical documents, and to select a
COTS product to integrate into the system. However, according to
project officials, in April 2002, VHA's Chief Information Officer
(CIO) determined that using a COTS solution would result in excessive
costs and make the department dependent on a vendor for a core
business function. Thus, the CIO directed the VISNs to redirect their
efforts and funding to develop a scheduling application instead of
purchasing a COTS application. VA issued a new statement of work for
SwRI to design, build, and test the scheduling application. The
department planned to deliver the new outpatient scheduling system
first to the location in Muskogee, referred to as the alpha
deployment, by December 2004. Once successfully tested and deployed at
this location, the system was to be deployed within VISN 16 and 17 for
testing, then nationally to all VHA facilities.
In 2004, issues integrating the application with HealtheVet components
and funding reductions led to a delay in the alpha deployment date,
pushing it back to October 2006. In an effort to meet the new date, VA
decided in April 2005 to descope the alpha version of the scheduling
application by removing certain planned capabilities. Simultaneously,
the department and SwRI began treating a separate version that was to
retain all planned capabilities as a distinct development effort,
referred to as the beta version. Nevertheless, delays in correcting
defects, conducting tests, and changing the code in response to
infrastructure modifications, resulted in six more extensions of the
target alpha deployment date (over 2 ½ years beyond the October 2006
planned date).
Further, in an attempt to expedite the project, in September 2008, the
Principal Deputy Under Secretary for Health directed the project team
to focus its efforts on a national deployment of the new scheduling
system by the end of 2009, rather than on the single-site alpha
deployment. However, in January 2009, the project team determined that
the product that had been developed for alpha deployment would not be
suitable for national deployment by the end of 2009; thus, in February
2009, the department terminated its contract for the replacement
scheduling application. VA subsequently ended the entire Scheduling
Replacement Project in September 2009. Figure 1 depicts a timeline of
key project events from its initiation through its termination.
Figure 1: Timeline of Scheduling Replacement Project Events:
[Refer to PDF for image: timeline]
Request for Proposals to VISNs: October 2000.
Project awarded to VISNs 16 and 17: January 2001.
Project redirected from COTS to development: April 2002.
Slipped alpha deployment: December 2004.
Project descoped alpha and beta version split: April 2005.
Original planned national deployment: June 2006.
Slipped alpha deployment: October 2006.
Slipped alpha deployment: June 2007.
Slipped alpha deployment: September 2007.
Slipped alpha deployment: March 2008.
Slipped alpha deployment: June 2008.
Project redirected to implement national deployment of scheduling
application by the end of 2009: August 2008.
Slipped alpha deployment: September 2008.
Key contract for scheduling application terminated: February 2009.
Slipped alpha deployment: May 2009.
Project termination: October 2009.
Source: GAO analysis of VA data.
[End of figure]
Governance of the Scheduling Replacement Project:
Several organizations within VA were responsible for governance of the
Scheduling Replacement Project:
* In July 2000, VHA established a project management office to
coordinate all efforts and monitor project activities to ensure
success of the Scheduling Replacement Project. The project management
office was to ensure achievement of milestones, evaluate project
success, and report to VHA senior level executives.
* In June 2001, VHA established the Scheduling Replacement Board of
Directors to guide the overall direction of the project. According to
its charter, the board was to review project activities on a quarterly
basis, provide key decisions at major project milestones, confirm the
achievement of project milestones, and evaluate project success.
* In February 2003, VA established an Enterprise Information Board as
its executive decision-making body for information technology (IT)
capital planning and investment control. The board was to provide
oversight in the selection, management, and control of IT investments
such as the scheduling system.
In February 2007, the Secretary of Veterans Affairs approved a
centralized IT management structure for the department.[Footnote 5] As
part of this realignment, staff from the project management office
with responsibility for the Scheduling Replacement Project were
transferred from VHA to the Office of Enterprise Development (OED)
within VA's Office of Information and Technology (OI&T).
Also in 2007, VA issued a governance plan[Footnote 6] to enable the
department to better align its IT strategy to its business strategy,
manage investments, and reconcile disputes regarding IT. The
governance structure established by the plan included three governance
boards for IT projects, such as the Scheduling Replacement Project:
* The Budgeting and Near-Term Issues Board is to identify, review,
recommend, and advocate projects and programs across the department.
[Footnote 7] The board's responsibilities include monitoring projects'
achievement of results.
* The Programming and Long-Term Issues Board is to oversee portfolio
development and evaluate program execution by conducting milestone
reviews and program management reviews of IT investments.[Footnote 8]
* The Information Technology Leadership Board is responsible for
adjudicating all unresolved resource issues forwarded by the Budgeting
and Near-Term Issues Board and forwarding recommendations to the
department's Strategic Management Council.[Footnote 9]
Prior Reviews of HealtheVet and the Scheduling Replacement Project:
We and VA's Office of Inspector General have both issued reports
concerning the HealtheVet initiative and the Scheduling Replacement
Project. Specifically, in a June 2008 report, we raised concerns about
VA's HealtheVet initiative.[Footnote 10] We noted that the eight major
software development projects comprising the initiative (which
included the Scheduling Replacement Project) were in various stages of
development, and that none had yet been completed. We noted that while
VA had established interim dates for completing the component
projects, it had not developed a detailed schedule or approach for
completing the overall HealtheVet initiative. Further, the department
had not yet implemented a complete governance structure; several key
leadership positions within the development organization had not been
filled or were filled with acting personnel; and the departmental
governance boards had not scheduled critical reviews of HealtheVet
projects. We concluded that, without all elements of governance and
oversight in place, the risk to the success of the HealtheVet
initiative and, therefore, its component initiatives (such as the
Scheduling Replacement Project) was increased. Accordingly, we
recommended that VA develop a comprehensive project management plan
and schedule, as well as a governance structure, to guide the
development and integration of the many projects under this complex
initiative. Subsequent to our 2008 report, VA reported that it had
begun to formulate a project management plan, an integrated schedule
of projects, and a governance plan for the HealtheVet initiative.
Further, in reporting on the development of the replacement scheduling
application in August 2009, the Office of Inspector General noted,
among other things, that VA did not have staff with the necessary
expertise to execute large-scale IT projects. The report also noted
that there was minimal oversight of the contracting processes on the
project and that the department had made no attempt to find a
contracting officer with experience for this multi-year, complex
project. The Inspector General suggested that VA develop effective
oversight processes, develop in-house staff with the expertise to
manage and execute complex integrated IT programs, and expand the
number of contracting officers with experience on large projects. In
response, the department consolidated IT procurements under the Office
of Acquisition, Logistics, and Construction and established the
Technology Acquisition Center to administer future OI&T contracts.
VA Ended the Outpatient Scheduling System Project without Delivering
Expected Capabilities and Has Begun a New Initiative:
After spending an estimated $127 million over 9 years (from fiscal
years 2001 through 2009) on its outpatient scheduling system project,
VA has not yet implemented any of the system's expected capabilities.
According to the department, of the total amount, $62 million was
expended for, among other things, project planning, management
support, a development environment, and equipment. In addition, the
department paid an estimated $65 million to SwRI to develop the
replacement scheduling application. However, VA and SwRI were not able
to resolve a large number of system defects, and the department
terminated the contract in February 2009. Subsequently, the department
determined that the application was not viable (i.e., did not meet its
needs), and officially ended the Scheduling Replacement Project on
September 30, 2009.
The department began a new initiative on October 1, 2009, which it
refers to as HealtheVet Scheduling.[Footnote 11] However, as of early
April 2010, it had completed only limited tasks for the new
initiative. Specifically, the department's efforts consisted of
analyzing alternatives and briefing VA's CIO on the analysis.
Officials told us that they had not yet developed a project plan or
schedule for the initiative, but intended to do so after determining
whether to build or buy the new application.
Scheduling Replacement Project Was Hindered by Weaknesses in Key
Management Capabilities:
The success of large IT projects is dependent on agencies' possessing
management capabilities to effectively conduct acquisitions, manage
system requirements, perform system tests, measure and report project
performance, and manage project risks. In addition, effective
institutionalized oversight is necessary to ensure that projects are,
in fact, demonstrating these management capabilities and achieving
expected results. However, the Scheduling Replacement Project had
weaknesses in these areas that, if not addressed, could derail the
department's current attempt to deliver a new scheduling system.
Contracting for the Scheduling System Was Inconsistent with
Fundamental Acquisition Management Principles:
The Federal Acquisition Regulation (FAR) requires preparation of
acquisition plans,[Footnote 12] and our prior work evaluating major
system acquisitions has found that planning is an essential activity
to reduce program risk.[Footnote 13] According to the FAR, an
acquisition plan must address, among other things, how competition
will be sought, promoted, and sustained throughout the course of the
acquisition, or cite the authority and justification for why full and
open competition cannot be obtained.[Footnote 14] Competition can help
save taxpayer money, improve contractor performance, and promote
accountability for results. Agencies are generally required to obtain
full and open competition, except in certain specified situations such
as modifications within the scope of the existing contract.[Footnote
15] Orders placed against a federal supply schedule are considered to
be issued using full and open competition if the applicable procedures
are followed.[Footnote 16] We have also found that having a capable
acquisition workforce is a necessary element of properly conducting
acquisitions that will meet agency needs.
VA did not develop an acquisition plan until May 2005, about 4 years
after the department first contracted for a new scheduling system.
Thus, formative decisions with implications for the scheduling
project's success, such as what the contractor was to do, the type of
contract to be used, and how competition would be promoted and, if
not, why, were made in an ad hoc fashion (i.e., not subject to a
deliberative planning process). Further, VA did not promote
competition in contracting for its scheduling system. Specifically,
rather than performing activities that are intended to promote
competition (e.g., announcing the requirement, issuing a solicitation,
and evaluating proposals), VA issued task orders against an existing
CASU contract that the department had in place for acquiring services
such as printing, computer maintenance, and data entry. Later, when
the department changed its strategy to acquire a custom-built
scheduling application instead of pursuing COTS integration--a
fundamental change to the development approach and contract scope--the
department again did not seek to obtain the benefits of competition.
Instead, the project team directed the change through a letter to the
existing contractor and a substantially revised statement of work.
In August 2004, VA determined that it would no longer support the CASU
agreement, and in response, the project team sought to use a General
Services Administration (GSA) schedule contract to retain the services
of its existing contractor. However, VA did not follow required
ordering procedures when it transitioned to the GSA schedule contract.
Specifically, VA did not solicit price quotes from at least three
schedule vendors, as required by the FAR.[Footnote 17] Instead, at the
direction of the program office, the department provided a statement
of work only to the incumbent contractor, which responded with a
proposal and price quote. As a result, VA increased the risk that it
was not selecting a contractor that would provide the best approach.
Further, VA did not assess whether the purchase of commercial services
under this schedule was the most suitable means for developing a
custom-built scheduling application.
These weaknesses in VA's acquisition management for the scheduling
system project reflect the inexperience of the department's personnel
in administering major IT contracts. In this regard, VA's Inspector
General identified the lack of VA personnel who are adequately trained
and experienced to plan, award, and administer IT contracts as a major
management challenge for the department and specifically cited the
scheduling system acquisition as an example. Also, VA's contracting
officer told us that the contracting office did not have prior
experience in the award or administration of contracts for IT system
development.
According to the HealtheVet Scheduling program manager, going forward,
the scheduling system project team plans to use VA's Technology
Acquisition Center within the Office of Acquisition, Logistics, and
Construction to administer future contracts. Established in March
2009, in an effort to improve the department's IT acquisition
management, the center is comprised of experienced acquisition staff
members who are to provide exclusive contracting support to the Office
of Information and Technology. According to the Executive Director,
the Technology Acquisition Center includes technical specialists who
can offer assistance with refining statements of work and contractual
requirements. Also, representatives from the Office of General Counsel
are colocated with the center to facilitate reviews for compliance
with applicable federal laws and regulations.
Although VA has taken positive actions to improve its IT acquisition
management, these actions do not ensure that the department will not
repeat the pattern of failing to seek and promote competition and
other weaknesses that it demonstrated in contracting for the
scheduling system. Until the department ensures that it has adequately
planned for the future acquisition of a scheduling system, including
whether and how it will provide for competition or otherwise comply
with federal contracting requirements, it cannot ensure that it will
be effective in acquiring a system that meets user needs at a
reasonable cost and within a reasonable time frame.
VA Did Not Ensure Requirements Were Complete and Sufficiently Detailed
to Guide Development of the Scheduling System:
According to recognized guidance, using disciplined processes for
defining and managing requirements can help reduce the risks of
developing a system that does not meet user needs, cannot be
adequately tested, and does not perform or function as intended.
[Footnote 18] Requirements should serve as the basis for a shared
understanding of the system to be developed. Among other things,
effective practices for defining requirements include analyzing
requirements to ensure that they are complete, verifiable, and
sufficiently detailed to guide system development. In addition,
maintaining bidirectional traceability from high-level operational
requirements through detailed low-level requirements to test cases is
an example of a disciplined requirements management practice. Further,
in previous work, we have found that requirements development
processes should be well-defined and documented so that they can be
understood and properly implemented by those responsible for doing so.
[Footnote 19]
VA did not adequately analyze requirements to ensure they were
complete, verifiable, and sufficiently detailed to guide system
development. For example, in November 2007, VA determined that
performance requirements were missing and that some requirements were
not testable. Further, according to project officials, some
requirements were vague and open to interpretation. For example,
although the requirement to sort appointment requests to be processed
was included, it required clarification on how those appointments
should be sorted. Also, requirements for processing information from
systems on which the scheduling application depended were missing. For
example, in June 2008, several requirements for processing updates to
a patient's eligibility had to be added. The incomplete and
insufficiently detailed requirements resulted in a system that did not
function as intended.
In addition, VA did not ensure that requirements were fully traceable.
As early as October 2006, an internal review of the scheduling
project's requirements management noted that the requirements did not
trace to business rules or to test cases. Yet, almost 2 years later,
in August 2008, VA documentation continued to reflect this problem--
stating that not every lower-level requirement traced back to one or
more of the higher-level functional requirements and down to test
cases. By not ensuring requirements traceability, the department
increased the risk that the system could not be adequately tested and
would not function as intended.
According to scheduling project officials, requirements were
incomplete, in part, because they depended on information from other
related systems that had not yet been fully defined. In addition, VA
did not develop a requirements management plan for the Scheduling
Replacement Project until October 2008. Our analysis of this plan
found it to be generally consistent with leading practices. However,
the project team's use of the requirements management plan was
precluded by the department's decision to end the project. According
to the Scheduling program manager, the project team expects to further
develop the requirements management plan, dependent upon the
department's yet-to-be-selected alternative for proceeding with the
current effort, HealtheVet Scheduling. Nevertheless, the department
has not yet demonstrated its capability to execute effective
requirements management practices.
Without well-defined and managed requirements, VA and its contractor
lacked a common understanding of the system to be developed and
increased the risk that the system would not perform as intended.
Going forward, effective requirements development and management will
be essential to ensuring that this risky situation, which could
endanger the success of VA's new scheduling system project, is not
repeated.
VA's Approach to Performing System Tests Increased Risk that the
System Would Not Perform as Intended:
Best practices in system testing indicate that testing activities
should be performed incrementally, so that problems and defects
[Footnote 20] with software versions can be discovered and corrected
early, when fixes generally require less time and fewer resources.
VA's guidance on conducting tests during IT development projects is
consistent with these practices and specifies four test stages and
associated criteria that are to be fulfilled in order to progress
through the stages.[Footnote 21] For example, defects categorized as
critical, major, and average severity that are identified in testing
stage one (performed within the development team) are to be resolved
before testing in stage two (performed by the testing services
organization) is begun.[Footnote 22]
Nonetheless, VA took a high-risk approach to testing the scheduling
system by performing tests concurrently rather than incrementally.
Based on information provided by project officials, the department
began stage two testing on all 12 versions of the scheduling
application before stage one testing had been completed. On average,
stage two testing began 78 days before stage one testing of the same
version had been completed. In two of these cases, stage two testing
started before stage one testing had begun. Compounding the risk
inherent in this concurrent approach to testing, the first alpha
version to undergo stage two testing had 370 defects that were of
critical, major, or average severity even though the department's
criteria for starting stage two testing specified that all such
defects are to be resolved before starting stage two testing. While
stage two testing was ongoing, VA made efforts to reduce the number of
defects by issuing additional task orders for defect repair to its
contractor and by hiring an additional contractor whose role was to
assist in defect resolution. However, almost 2 years after beginning
stage two testing, 87 defects that should have been resolved before
stage two testing began had not been fixed.
Scheduling project officials told us that they ignored their own
testing guidance and performed concurrent testing at the direction of
Office of Enterprise Development senior management in an effort to
prevent project timelines from slipping. In addition, project
officials told us they made a conscious decision to conduct concurrent
testing in an effort to promote early identification of software
defects. However, because the department did not follow its guidance
for system testing and, instead, performed concurrent testing, it
increased the risk that the scheduling project would not perform as
intended and would require additional time and resources to be
delivered.
If VA is to be successful in its new initiative to provide an
outpatient scheduling system, it is critical that the department
adhere to its own testing guidance for ensuring the resolution of
problems in a timely and cost-effective manner. Not doing so lessens
the usefulness of results from its testing activities and increases
the risk of additional system development failures.
VA's Progress Reporting Based on Earned Value Management Data Was
Unreliable:
Office of Management and Budget (OMB) and VA policies require major
projects to use earned value management (EVM) to measure and report
progress.[Footnote 23] EVM is a tool for measuring project progress by
comparing the value of work accomplished with the amount of work
expected to be accomplished. Such a comparison permits actual
performance to be evaluated, based on variances from the cost and
schedule baselines.[Footnote 24] Identification and reporting of
variances and analysis of their causes help program managers determine
the need for corrective actions. In addition, the cost performance
index (CPI) and schedule performance index (SPI) are indicators of
whether work is being performed more or less efficiently than planned.
[Footnote 25] Like the variances, reporting of CPI and SPI can provide
early warning of potential problems that need correcting to avoid
adverse results. For a complete view of program status and an
indication of where problems exist, performance data should be
reported for both current (generally the most recent month) and
cumulative periods. In addition, federal policy requires that systems
used to collect and process EVM data be compliant with the industry
standard developed by the American National Standards Institute (ANSI)
and Electronic Industries Alliance (EIA), ANSI/EIA Standard 748.
[Footnote 26] Such compliance is necessary to demonstrate the
capability to provide reliable cost and schedule information for
earned value reporting.
Although VA submitted monthly reports to the department's CIO based on
earned value data for the scheduling project, the reliability of the
data on which the reports were based was questionable and the reports
included data that provided inconsistent views of project performance.
Specifically regarding data reliability, department officials did not
ensure that the EVM reporting systems for the scheduling project had
been certified for compliance with ANSI/EIA Standard 748.[Footnote 27]
According to the former program manager, the department did not seek
to determine whether its development contractor's system was compliant
because SwRI entered cost and schedule data directly into the
department's EVM system. Although department officials asserted that
this EVM system was compliant with ANSI/EIA Standard 748, the
department could not provide documentation of such compliance. Because
VA had not demonstrated compliance with the standard, it could not
ensure that the data resulting from its EVM system and used for
progress reporting were reliable.
Regarding EVM reporting, in January 2006, the scheduling project
management office began providing monthly reports to the department's
CIO that were based on EVM data. However, in addition to being based
on data from EVM systems that had not been assessed for compliance
with the applicable standard, the progress reports also included
contradictory information about project performance. Specifically, the
reports featured stoplight (i.e., green for "in control," yellow for
"caution," or red for "out of control") indicators, based on the
cumulative CPI and SPI.[Footnote 28] These indicators frequently
provided a view of project performance that was inconsistent with the
reports' narrative comments. For example, the September 2006 report
identified cost and schedule performance as green, even though
supporting narrative comments stated that the project schedule was to
be extended by 9 months due to a delay in performing testing and the
need for additional time to repair system defects. The June 2007
report also identified project cost and schedule performance as green,
despite the report noting that the project budget was being increased
by $3 million so that the development contract could be extended to
accommodate schedule delays. Further, the December 2007 report
identified cost and schedule performance as green, while at the same
time stating that the development contract was to be extended again
and that a cost variance would be reported in the near future. This
pattern of inconsistent progress reporting continued until October
2008, when the report for that month and all others through August
2009 showed cost and schedule performance as red, which was consistent
with the actual state of the project.
In discussing this matter, the former program manager stated that the
Scheduling Replacement Project complied with the department's EVM
policies, but noted that the department performed EVM for the
scheduling project only to fulfill the OMB requirement and that the
data were not used as the basis for decision making because doing so
was not a part of the department's culture.[Footnote 29] Because VA's
scheduling project was not managed in accordance with EVM methods that
could provide a widely recognized means of reliably determining and
reporting cost and schedule performance, the department was not
positioned to detect performance shortfalls and initiate timely
corrective actions that might have prevented the project's failure.
Having EVM reporting that provides a reliable measure of progress will
be essential as the department moves forward with its new scheduling
project.
Major Risks of the Scheduling Project Were Not Identified and Reported:
Managing project risks means proactively identifying circumstances
that increase the probability of failure to meet commitments and
taking steps to prevent them from occurring. Federal guidance and best
practices advocate risk management.[Footnote 30] To be effective, risk
management activities should include identifying and prioritizing
risks as to their probability of occurrence and impact, documenting
them in an inventory, and developing and implementing appropriate risk
mitigation strategies. By performing these activities, potential
problems can be avoided before they become actual cost, schedule, and
performance shortfalls.
VA established a process for managing the scheduling project's risks
that was consistent with relevant best practices. Specifically,
project officials developed a risk management plan for managing risks
to the scheduling project. The plan defined five phases of the risk
management process--risk identification, risk analysis, risk response
planning, risk monitoring and control, and risk review. The plan also
defined risk-related roles and responsibilities for the scheduling
project staff and tools to be used to capture identified risks, track
their status, and communicate them. In addition, project officials
captured identified risks to the scheduling project in an automated
tracking tool. Examples of risks identified in the tool included the
risk that hardware at sites where the system was to be deployed was
incompatible with the new application and another related to SwRI's
failure to meet deliverable dates.
However, while the department had established a process for managing
risks to the scheduling project, it did not have a comprehensive list
of risks because it did not take key project risks into account. As
previously discussed, we identified problems in VA's approach to
managing the project in four major areas--acquisition management,
requirements management, system testing, and earned value management.
Nevertheless, VA did not identify as risks its weaknesses in the
following three project management practices: (1) using a
noncompetitive acquisition approach, (2) conducting concurrent testing
and initiation of stage two testing with significant defects, and (3)
reporting unreliable project cost and schedule performance
information. Any one of these risks alone had the potential to
adversely impact the outcome of the project. The three of them
together dramatically increased the likelihood that the project would
not succeed. Since these project management weaknesses were not
identified as risks, VA was unable to estimate the significance of
their occurrence and decide what steps should be taken to best manage
them.
Senior project officials indicated that staff members were often
reluctant to raise risks or issues to leadership in the Office of
Enterprise Development due to the emphasis on keeping the project on
schedule. Further, the scheduling program manager recognized that the
project management office was inadequately staffed to implement a
disciplined risk management process and stated that, in September
2008, a full-time risk manager was added to the staff.
As VA continues with its latest scheduling effort, it will be critical
that the department identify a comprehensive list of risks so that
threats to the project can be detected and mitigated in a timely
manner.
VA Did Not Conduct Oversight of the Scheduling Replacement Project for
2 Years after Major Problems Occurred:
GAO and OMB guidance call for the use of institutional management
processes to control and oversee IT investments.[Footnote 31] Critical
to these processes are activities to track progress of IT projects,
such as milestone reviews that include mechanisms to identify
underperforming projects, so that timely steps can be taken to address
deficiencies. These reviews should track project performance and
progress toward predefined cost and schedule goals, as well as monitor
project benefits and exposure to risks. Moreover, these activities
should be conducted by a department-level investment review board (or
comparable entity) composed of senior executives from the IT office
and business units with appropriate authority to address issues when
projects are not meeting cost, schedule, and performance goals.
VA's Enterprise Information Board was established in February 2003 to
provide oversight of IT projects through in-process reviews when
projects experience problems or variances outside of tolerance levels.
Similarly, the Programming and Long-Term Issues Board, established in
June 2007 as a result of the IT realignment, is responsible for
performing milestone reviews and program management reviews of
projects. However, between June 2006 and May 2008, the department did
not provide oversight of the Scheduling Replacement Project, even
though the department had become aware of significant issues
indicating that the project was having difficulty meeting its schedule
and performance goals.[Footnote 32] Specifically, in June 2006, the
project team found that a delivery of software from SwRI included over
350 defects, leading the office to delay the system deployment by 9
months, from October 2006 to July 2007, to mitigate the defects. A May
2007 report from an independent contractor stated that VA's project
management team did not have a clear understanding of the status of
the project in terms of progress being made on those defects. Further,
a July 2007 review by the Software Engineering Institute found that a
test environment had not been developed and that the schedule for
testing did not include sufficient time to identify and correct all
infrastructure issues. Based on the results of these reviews, the
project management office recommended the project be stopped and
reevaluated before moving forward.
Despite indications of problems with the project, neither the
Enterprise Information Board nor the Programming and Long-Term Issues
Board conducted reviews between June 2006 and May 2008 that could have
identified corrective actions for the Scheduling Replacement Project.
In June 2008, the Director of the Office of Enterprise Development
requested an operational test readiness review of the replacement
scheduling application by the Programming and Long-Term Issues Board
to determine if the application was ready for deployment. That review
identified issues, including significant critical defects in the
application and a lapse in a contract to resolve defects. According to
the chairman of the Programming and Long-Term Issues Board, it did not
conduct reviews of the scheduling project prior to June 2008 because
it was focused on developing the department's IT budget strategy.
In June 2009, VA's Assistant Secretary for Information and Technology,
who serves as the department's CIO, began establishing a new process
for planning and managing its IT projects--the Program Management
Accountability System (PMAS). According to the CIO, this process is
intended to promote near-term visibility into troubled programs,
allowing the department to take corrective actions earlier and avoid
long-term project failures. PMAS is expected to improve oversight of
IT projects through strict adherence to project milestones and
imposing strong corrective measures if a project misses multiple
milestones.
According to the CIO, under PMAS, projects will be expected to deliver
smaller, more frequent releases of new functionality to customers. In
addition, specific program resources and documentation are to be in
place before development begins, and approved processes are to be used
during the system development life cycle. This approach is intended to
ensure that customers, project members, and vendors working on a
project are aligned, accountable, and have access to the resources
necessary to succeed before work begins. For a program to be approved
for investment under PMAS, the program must have, among other things,
an established customer sponsor, a qualified incremental program plan,
requirements for three delivery milestones, and documented success
criteria.
According to the HealtheVet Scheduling program manager, the department
expects to develop plans for the new scheduling initiative, required
under PMAS, once a strategy for the initiative is selected. However,
the department has not yet demonstrated that it can sustain the
wholesale change in management of IT projects that PMAS represents or
that this new approach will be sufficiently robust to prevent or
correct weaknesses such as those that contributed to the Scheduling
Replacement Project's failure. Until the department has fully
established and effectively implemented the project management
controls that are expected to be a component of PMAS, it remains to be
seen whether this new approach will be effective in providing
oversight to ensure the success of the department's new scheduling
effort.
Impact of Scheduling Replacement Project Failure on HealtheVet Program
is Uncertain:
While the Scheduling Replacement Project was one of many components of
VA's HealtheVet initiative, the impact of the project's termination on
the initiative is currently unclear. The impact is unclear because the
relationships (i.e., interdependencies) among the various projects
under HealtheVet have not been determined.
As described in VA's budget submission for 2011, HealtheVet is the
most critical IT development program for medical care, and is expected
to enhance and supplement the legacy VistA system using highly
integrated health care applications, such as the capability to
schedule outpatient appointments. However, the department's efforts
have not yet resulted in a finalized plan that outlines what needs to
be done and when. As of March 2010, the department had not completed
its comprehensive plan and integrated schedule to guide the
development and integration of the many projects that make up this
departmentwide initiative. According to officials in VA's Office of
Information and Technology, the department plans to document the
interdependencies, project milestones, and deliverables in an
integrated master schedule as part of a project management plan that
is expected to be completed by June 2010.
In the absence of an overall comprehensive plan for HealtheVet that
incorporates critical areas of system development and considers all
dependencies and subtasks and that can be used as a means of
determining progress, it is difficult to determine how scheduling and
other applications will be integrated into this larger HealtheVet
system. Likewise, without such a plan, the impact of the terminated
scheduling project on the completion of the HealtheVet initiative
cannot be determined.
Conclusions:
After almost a decade of effort, VA has not accomplished what it set
out to achieve in replacing its patient scheduling system. A broad
range of managerial weaknesses plagued the project from beginning to
end and increased the project's risk of failure. Specifically, because
the department did not develop and execute an acquisition plan, its
acquisition activities were ad hoc and it did not seek to obtain the
benefits of competition. Additionally, in defining and managing system
requirements, the department did not perform critically important
activities such as ensuring that the requirements were complete and
sufficiently detailed. Further, the department's decision to
concurrently conduct tests contributed to an increased risk that the
application would not perform as intended, and its earned value
management data did not serve as a reliable indicator of project
performance. Moreover, even though the department had a plan and
process for managing project risks, it did not identify key risks
mentioned or take steps to mitigate them. Finally, although the
department was aware of major issues with the project through several
external reviews, the lack of effective institutional oversight
allowed the project to continue unchecked and, ultimately, to fail.
Given this situation, the department is starting over and is in the
process of analyzing alternative strategies, which will be the basis
for a project plan that is to be developed. At the same time, the
department is instituting a new approach that is intended to manage
and control IT system projects and avoid project failures, such as
what has occurred with the Scheduling Replacement Project. Finally,
while the scheduling system project was to result in the first
component of VA's larger HealtheVet initiative to modernize the
department's health information system, the specific impact of the
project's failure on this initiative is unclear because HealtheVet
plans have not been completed.
Until the department effectively implements measures that prevent the
types of management weaknesses that plagued its earlier efforts, it
risks incurring similar weaknesses in its latest scheduling
replacement effort, which could again prevent VA from delivering this
important capability for serving the health care needs of veterans and
their families.
Recommendations for Executive Action:
To enhance VA's effort to successfully fulfill its forthcoming plans
for the outpatient scheduling system replacement project and the
HealtheVet program, we recommend that the Secretary of Veterans
Affairs direct the CIO to make certain the following six actions are
taken:
* Ensure acquisition plans document how competition will be sought,
promoted, and sustained or identify the basis of authority for not
using full and open competition.
* Ensure implementation of a requirements management plan that
reflects leading practices for requirements development and
management. Specifically, implementation of the plan should include
analyzing requirements to ensure they are complete, verifiable, and
sufficiently detailed to guide development, and maintaining
requirements traceability from high-level operational requirements
through detailed low-level requirements to test cases.
* Adhere to the department's guidance for system testing including (1)
performing testing incrementally and (2) resolving defects of average
and above severity prior to proceeding to subsequent stages of testing.
* Ensure effective implementation of EVM by making certain that the:
(1) EVM reporting systems for the scheduling project are certified for
compliance with ANSI/EIA Standard 748 and data resulting from the
systems are reliable; (2) project status reports based on EVM data are
reliable in their portrayal of the project's cumulative and current
cost and schedule performance; and (3) officials responsible for
managing and overseeing the project use earned value data as an input
to their decision-making processes.
* Identify risks related to the scheduling project moving forward and
prepare plans and strategies to mitigate them.
* Ensure that the policies and procedures VA is establishing to
provide meaningful program oversight are effectively executed and that
they include (1) robust collection methods for information on project
costs, benefits, schedule, risk assessments, performance metrics, and
system functionality to support executive decision making; (2) the
establishment of reporting mechanisms to provide this information in a
timely manner to department IT oversight control boards; and (3)
defined criteria and documented policies on actions the department
will take when development deficiencies for a project are identified.
Agency Comments and Our Evaluation:
The VA Chief of Staff provided written comments on a draft of this
report. In its comments, the department generally agreed with our
conclusions, concurred with five of our six recommendations, and
described actions to address them. For example, the department stated
that it will work closely with contracting officers to ensure future
acquisition plans clearly identify an acquisition strategy that
promotes full and open competition. In addition, the department stated
that its new IT project management approach, PMAS, will provide near-
term visibility into troubled programs, allowing the Principal Deputy
Assistant Secretary for Information and Technology to provide help
earlier and avoid long-term project failures.
The department concurred in principle with one of our recommendations:
that it ensure effective implementation of EVM. In this regard, the
department noted that PMAS requires monthly analysis and reporting of
project performance, in addition to VA's project status reporting to
OMB and the public. However, the department did not describe its
actions to ensure the reliability of project performance data and
reports, nor did it explain how it would ensure the use of reliable
performance data in managing and overseeing the project under PMAS.
Unless the department fully addresses this recommendation, VA may not
be positioned to reliably detect performance shortfalls and initiate
timely corrective actions that could prevent future project failure.
The department also provided technical comments, which we have
incorporated in the report as appropriate. The department's written
comments are reproduced in appendix II.
As agreed with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days
from the date of this letter. At that time, we will send copies of the
report to interested congressional committees, the Secretary of
Veterans Affairs, and other interested parties. In addition, the
report will be available at no charge on the GAO Web site at
[hyperlink, http://www.gao.gov].
If you or your staff have questions about this report, please contact
me at (202) 512-6304 or melvinv@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. Key contributors to this report are
listed in appendix III.
Sincerely,
Signed by:
Valerie C. Melvin:
Director, Information Management and Human Capital Issues:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
The objectives of our study were to (1) determine the status of the
Scheduling Replacement Project, (2) determine the effectiveness of the
Department of Veterans Affairs (VA) management and oversight of the
project, and (3) assess the impact of the project on VA's overall
implementation of its health information system modernization
initiative--HealtheVet.
To determine the status of the Scheduling Replacement Project, we
reviewed status briefings on VA's assessment of alternatives for its
new scheduling initiative, as well as the department's fiscal year
2011 budget submission. We supplemented these reviews with interviews
with the scheduling program manager, the Director of the Office of
Enterprise Development, and the Veterans Health Administration
Enterprise Systems Manager for the project.
To determine the effectiveness of the department's management and
oversight of the project, we evaluated its acquisition management,
system requirements management, system test management, use of earned
value management, management and mitigation of risks, and project
oversight and governance processes.
To evaluate VA's approach to contracting for the scheduling system, we
reviewed and analyzed program documentation, including the Scheduling
Replacement Project acquisition plans, contract task orders,
statements of work, sole source justifications, and a contracting
white paper to determine the extent to which the agency's practices
were consistent with relevant planning and competition requirements in
the Federal Acquisition Regulation.
Regarding system requirements management, we compared project
requirements management practices described in system requirements
documents such as the software requirements specification and project
status briefings to recognized requirements management guidance, such
as those included in the Software Engineering Institute's Capability
Maturity Model Integration.[Footnote 33] We also assessed the
scheduling project requirements management plan and examined the
degree to which it was consistent with leading requirements management
practices such as the Software Engineering Institute's Capability
Maturity Model Integration.
To determine the effectiveness of VA's test management, we reviewed
the department's guidance for performing system tests and compared
project testing activities to this guidance and associated best
practices. Specifically, we reviewed documentation of test results to
determine the dates testing occurred and the number and severity of
defects identified.
To review VA's use of earned value management (EVM) to assess and
report project performance, we reviewed Office of Management and
Budget Memorandum M-05-23, as well as VA standard operating procedures
related to EVM to identify requirements for effective execution of
this discipline in assessing project performance. We compared the
Scheduling Replacement Project's approach to EVM with recognized
practices as described in GAO's Cost Estimating and Assessment Guide,
such as the American National Standards Institute (ANSI) and
Electronic Industries Alliance (EIA), ANSI/EIA Standard 748.[Footnote
34] We reviewed scheduling project reports on earned value performance
that were provided to management to determine the level to which these
reports provided complete and meaningful cost and schedule performance
trends to department management.
To determine the effectiveness of the management and mitigation of
scheduling project risks, we consulted industry guidance on risk
mitigation and management, including Software Engineering Institute's
Capability Maturity Model Integration.[Footnote 35] In addition, we
reviewed the scheduling project's risk management plan and process,
including the Scheduling Replacement Project Risk Management plan, and
determined the level to which the department's plans and processes met
industry best practices and were executed to identify risks. Further,
we also examined the department's risk inventory to determine whether
project risks we found during our review had been identified and
considered by VA.
To assess the effectiveness of scheduling project oversight and
governance, we reviewed GAO guidance on effective project oversight,
including our Information Technology Investment Management
Framework;[Footnote 36] analyzed documentation from department
oversight entities that existed over the course of the project,
including the Enterprise Information Board and the Programming and
Long-Term Issues Board; and determined the extent to which these
bodies performed effective oversight of the project.
In addition to the actions just described, we supplemented our
analysis by interviewing cognizant VA and contractor officials
including the VA Chief Information Officer, current and former program
managers, project team members, representatives from the Veterans
Health Administration, the department's contracting officer for the
project, and the Director of the Office of Enterprise Development.
To assess the impact of the scheduling project on VA's overall
implementation of its health information system modernization
initiative, we reviewed documentation such as briefings from
HealtheVet planning meetings and interviewed cognizant officials,
including the Medical Care Program Executive Officer in the Office of
Information and Technology and the Director of Health Information
Systems in the Office of Enterprise Development, about the status of
the HealtheVet initiative.
We conducted this performance audit at VA headquarters in Washington,
D.C., from May 2009 through May 2010, in accordance with generally
accepted government auditing standards. Those standards require that
we plan and perform the audit to obtain sufficient, appropriate
evidence to provide a reasonable basis for our findings and
conclusions based on our audit objectives. We believe that the
evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.
[End of section]
Appendix II: Comments from the Department of Veterans Affairs:
Department of Veterans Affairs:
Office of the Secretary:
May 11, 2010:
Ms. Valerie C. Melvin:
Director, Information Management and Human Capital Issues:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Ms. Melvin:
The Department of Veterans Affairs (VA) has reviewed the Government
Accountability Office's (GAO) draft report, Information Technology:
Management Improvements Are Essential to VA's Second Effort to Replace
Its Outpatient Scheduling System (GA0-10-579) and generally agrees
with GAO's conclusions and concurs with five and concurs in principle
with one of GAO's recommendations to the Department.
The enclosure addresses GAO's recommendations and provides technical
comments to the draft report. VA appreciates the opportunity to
comment on your draft report.
Sincerely,
Signed by:
John R. Gingrich:
Chief of Staff:
Enclosure:
[End of letter]
Enclosure:
Department of Veterans Affairs (VA) Comments to Government
Accountability Office (GAO) Draft Report: Information Technology:
Management Improvements Are Essential to VA's Second Effort to Replace
Its Outpatient Scheduling System (GA0-10-579):
GAO recommendation: To enhance VA's efforts to successfully fulfill
its forthcoming plans for the outpatient scheduling system replacement
project and the HealtheVet program, we recommend that the Secretary of
Veterans Affairs direct the CIO to make certain the following six
actions are taken:
Recommendation 1: Ensure acquisition plans document how competition will
be sought, promoted, and sustained or identify the basis of authority
for not using full and open completion.
Response: Concur. VA will work closely with contracting officers to
ensure future acquisition plans clearly identify an acquisition
strategy that promotes, and gives preference to, full and open
competition. Specifically, the Technical Acquisition Center (TAC) will
be central to all acquisition efforts.
Recommendation 2: Ensure implementation of a requirements management
plan that reflects leading practices for requirements development and
management. Specifically, implementation of the plan should include
analyzing requirements to ensure they are complete, verifiable, and
sufficiently detailed to guide development, and maintaining
requirements traceability from high-level operational requirements
through detailed low-level requirements to test cases.
Response: Concur. Involving the business and technical communities, VA
initiated a complete review of all business requirements and is
working towards ensuring the requirements are complete, verifiable and
sufficiently detailed to guide development. All requirements
(functional and technical) are closely managed and documented using
documented, repeatable, processes located in Office of Enterprise
Development (OED) ProPath. OED ProPath is a Web-based, comprehensive
map mandating standardized, end-to-end, repeatable processes and
project documentation for all IT development projects.
Recommendation 3: Adhere to the department's guidance for system
testing including (1) performing testing incrementally and (2)
resolving defects of average and above severity prior to proceeding to
subsequent stages of testing.
Response: Concur. All testing is closely managed and documented using
documented, repeatable, processes located in OED ProPath.
Recommendation 4: Ensure effective implementation of EVM by making
certain that the (1) EVM reporting systems for the scheduling project
are certified for compliance with ANSI/EIA Standard 748 and data
resulting from the systems are reliable; (2) project status reports
based on EVM data are reliable in their portrayal of the project's
cumulative and current cost and schedule performance; and (3)
officials responsible for managing and overseeing the project use
earned value data as an input to their decision-making processes.
Response: Concur in principle. Each project in the Office of
Information and Technology is transitioning to an incremental delivery
model. VA's Program Management Accountability System (PMAS)
incorporates industry best practices relating to software development.
As such, VA intends to adopt the industry best practices for
management, metrics, and reporting that are inherent to these new,
risk-reducing software development methods. PMAS related processes in
VA's implementation of ProPath require project management analysis of
performance as well as examining both performance and examining
performance variances in monthly reports to Executive Management. VA
is already engaged in fully transparent reporting of project status to
the Office of Management and Budget and the public. VA is also looking
for additional approaches to enhance transparency to the public and
strengthen stewardship of public funds.
Recommendation 5: Identify risks related to the scheduling project
moving forward and prepare plans and strategies to mitigate them.
Response: Concur. VA developed a comprehensive Analysis of
Alternatives (AoA) to examine the benefits and risks associated with
several alternatives for meeting the patient scheduling needs of the
business community. The AoA closely examined acquisition, development,
deployment, and sustainment risks and identified mitigation strategies
for consideration as part of the VA decision process. Additionally,
the HealtheVet Scheduling Program Office employs a full-time Risk
Manager and has implemented a formal, active, and thorough Risk
Management Program. The Risk Management Program includes regular
program reviews to evaluate and manage risks.
Recommendation 6: Ensure that the policies and procedures VA is
establishing to provide meaningful program oversight are effectively
executed and that they include (1) robust collection methods for
information on project costs, benefits, schedule, risk assessments,
performance metrics, and system functionality to support executive
decision making; (2) the establishment of reporting mechanisms to
provide this information in a timely manner to department IT oversight
control boards; and (3) defined criteria and documented policies on
actions the department will take when development deficiencies for a
project are identified.
Response: Concur. VA instituted PMAS as a rigorous management approach
to addressing performance shortcomings. PMAS delivers smaller, more
frequent releases of new functionality to customers, ensuring
customers, project managers, and vendors working on a project are
aligned, accountable, and have access to necessary resources before
work begins. PMAS mandates that specific program resources and
documentation be in place before development begins and mandates that
approved processes be used during the system development life cycle
(SDLC).
PMAS provides near-term visibility into troubled programs, allowing
the Principal Deputy Assistant Secretary for Information and
Technology to provide help earlier and avoid long-term project
failures. Frequent software deliveries allow customers to provide
earlier feedback on system functionality, eliminates "big bang"
program/project failures, and increases the probability of
successfully developing and deploying VA IT systems.
To address shortfalls in project documentation and process controls,
VA identified and documented key SDLC processes in OED ProPath. OED
ProPath is a Web-based, comprehensive map mandating standardized, end-
to-end, repeatable processes and project documentation for all IT
development projects.
[End of section]
Appendix III: GAO Staff Contact and Acknowledgments:
GAO Contact:
Valerie C. Melvin, (202) 512-6304 or melvinv@gao.gov:
Staff Acknowledgments:
In addition to the contact named above, key contributions to this
report were made by Mark T. Bird, Assistant Director; Carol Cha; Shaun
Byrnes; Neil Doherty; Rebecca Eyler; Michele Mackin; Lee McCracken;
Constantine J. Papanastasiou; Michael W. Redfern; J. Michael Resser;
Sylvia Shanks; Kelly Shaw; Eric Trout; Adam Vodraska; and Merry Woo.
[End of section]
Footnotes:
[1] See GAO, Veterans Affairs: Health Information System Modernization
Far from Complete; Improved Project Planning and Oversight Needed,
[hyperlink, http://www.gao.gov/products/GAO-08-805] (Washington, D.C.:
June 30, 2008). In this report, we highlighted VA's progress toward
modernizing its medical information system, developing plans for
completing the project, and providing oversight of the project.
However, we noted that VA lacked a comprehensive project management
plan to guide remaining work and a complete governance structure for
HealtheVet.
[2] In 1995, VA shifted management authority from its headquarters to
new regional management structures--VISNs. VA created 22 VISNs, each
led by a director and a small staff of medical, budget, and
administrative officials. The VISNs have been configured around
historic referral patterns to VA's tertiary care medical centers.
These networks have substantial operational autonomy and now perform
the basic decision-making and budgetary duties of the VA health care
system. The network office in each VISN oversees the operations of the
medical centers in its area and allocates funds to each of them.
[3] CASUs are a network of federal entrepreneurial organizations that
provide a full range of "best value" administrative support services
to federal agencies throughout the United States and overseas on a
cost-reimbursable basis.
[4] SwRI is an independent, nonprofit applied research and development
organization.
[5] This centralization was to provide greater authority and
accountability over the department's resources by centralizing IT
management under the department-level Chief Information Officer and
standardizing operations and systems development across the department
using new management processes based on industry best practices.
[6] Department of Veterans Affairs, VA IT Governance Plan, version
8.3, March 12, 2007.
[7] This board became operational in May 2007 and was originally named
the Business Needs and Investment Board. This board is chaired by the
VA Principal Deputy Assistant Secretary for Information and Technology.
[8] This board became operational in June 2007 and was originally
named the Planning, Architecture, Technology, and Services Board. This
board is chaired by the VA Deputy Assistant Secretary for Information
and Technology Enterprise Strategy, Policy, Plans, and Programs.
[9] This board became operational in June 2007 and is chaired by the
VA Assistant Secretary for Information and Technology.
[10] [hyperlink, http://www.gao.gov/products/GAO-08-805].
[11] VA's fiscal year 2010 budget estimate for HealtheVet Scheduling
is $10 million.
[12] See FAR, subpart 7.1. See also FAR 34.004.
[13] See FAR 39.102 and GAO, Defense Acquisitions: Managing Risk to
Achieve Better Outcomes, GAO-10-374T (Washington, D.C.: Jan. 20, 2010).
[14] See FAR 7.105 b(2).
[15] See FAR, part 6.
[16] See FAR 8.404.
[17] See FAR 8.405-2, et seq., as added by Federal Acquisition
Circular 2001-24, FAR Case 1999-603; Item V, 69 Fed. Reg. 34231 (June
18, 2004).
[18] See FAR 39.102 and Carnegie Mellon Software Engineering
Institute, Capability Maturity Model® Integration for Development,
Version 1.2 (Pittsburgh, Pa., August 2006), and Software Acquisition
Capability Maturity Model (SA-CMM) version 1.03, CMU/SEI-2002-TR-010
(Pittsburgh, Pa., March 2002).
[19] GAO, Secure Border Initiative: DHS Needs to Address Significant
Risks in Delivering Key Technology Investment, [hyperlink,
http://www.gao.gov/products/GAO-08-1086] (Washington, D.C.: Sept. 22,
2008).
[20] Defects are system problems that require a resolution and can be
due to a failure to meet the system specifications.
[21] According to VA testing documentation, these stages are (1)
testing within the VA development team, (2) testing services, (3)
field testing, and (4) final review and acceptance testing.
[22] VA's Defect Control Process identifies four severity levels: (1)
critical, serious errors that cause system crashes, loss of data, or
loss of overall system functionality without a workaround; (2) major,
serious errors that impair major system function for which no
workaround is available or for which only workarounds of more than
three user steps are available; (3) average, errors in daily
operations, serious errors with workarounds with less than three user
steps; and (4) minor, cosmetic, and documentation errors such as
misspellings, field alignments, and missing fly-over text.
[23] OMB issued policy guidance (M-05-23) to agency CIOs on improving
technology projects that includes requirements for reporting
performance to OMB using EVM (August 2005). VA, VA EVM System, VA
Directive 6061, (February 2006). VA, Standard Operating Procedures
(SOP) for EVM Reporting and Analysis, EVM System SOP 7, (February
2007), VA, Primavera SOP, SOP 015: EVM, (February 2005). See also FAR,
Subpart 34.2.
[24] Cost variances compare the value of the completed work (i.e., the
earned value) with the actual cost of the work performed. Schedule
variances are also measured in dollars, but they compare the earned
value of the completed work with the value of the work that was
expected to be completed. Positive variances indicate that activities
cost less or are completed ahead of schedule. Negative variances
indicate activities cost more or are falling behind schedule.
[25] CPI is the ratio of earned value to actual costs; SPI is the
ratio of earned value to planned value.
[26] See OMB, Capital Programming Guide, II.2.4, Establishing an
Earned Value Management System. Reflected in FAR, subpart 34.2.
[27] Typically, an independent organization conducts the compliance
review of an EVM system. Upon successful completion of the review,
system acceptance should be documented, showing how each of the 32
ANSI/EIA Standard 748 guidelines has been satisfied.
[28] According to relevant best practices, indexes that are in control
are indicated by the color green. If the project goes into the caution
area, it is indicated by yellow and the project manager needs to get
involved to prevent the project from entering the out of control area,
which is designated by the color red. A project that stays in the red
area is considered to be out of control. In the reports to management,
VA set the tolerances for CPI and SPI to be green if the index is 0.96
through 1.04, yellow if it is 0.90 through 0.95 or 1.05 through 1.10,
and red if it is less than 0.90 or greater than 1.10.
[29] See GAO, Information Technology: Agencies Need to Improve the
Implementation and Use of Earned Value Techniques to Help Manage Major
System Acquisitions, [hyperlink, http://www.gao.gov/products/GAO-10-2]
(Washington, D.C.: Oct. 8, 2009). In this report, we assessed VA's
earned value management approach for its VistA-Foundations
Modernization program, which addressed the need to transition the
department's electronic medical record system to a new architecture.
We found that the program's EVM reports did not offer adequate detail
to provide insight into data reliability issues and earned value data
was not used for decision making.
[30] OMB Circular A-130 (Nov. 30, 2000) and Carnegie Mellon Software
Engineering Institute, Capability Maturity Model Integration for
Development, Version 1.2 (Pittsburgh, Pa., August 2006).
[31] GAO, Information Technology Investment Management: A Framework
for Assessing and Improving Process Maturity, [hyperlink,
http://www.gao.gov/products/GAO-04-394G] (Washington, D.C.: March
2004) and Office of Management and Budget, Capital Programming Guide:
Supplement to Circular A-11, Part 7, Planning, Budgeting, and
Acquisition of Capital Assets (Washington, D.C., June 2006).
[32] Oversight of the project was the responsibility of the Scheduling
Replacement Board of Directors from 2000-2004 and occurred through
annual reviews of the project.
[33] Carnegie Mellon Software Engineering Institute, Capability
Maturity Model Integration for Development, Version 1.2 (Pittsburgh,
Pa., August 2006).
[34] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for
Developing and Managing Capital Program Costs, [hyperlink,
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009).
[35] Carnegie Mellon Software Engineering Institute, Capability
Maturity Model Integration for Development, Version 1.2 (Pittsburgh,
Pa., August 2006).
[36] GAO, Information Technology Investment Management: A Framework
for Assessing and Improving Process Maturity, [hyperlink,
http://www.gao.gov/products/GAO-04-394G], version 1.1 (Washington,
D.C.: March 2004).
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
