Information Technology
Greater Use of Best Practices Can Reduce Risks in Acquiring Defense Health Care System Gao ID: GAO-02-345 September 26, 2002This report examines the acquisition of the Composite Health Care System (CHCS) II. It is one in a series of reports reviewing the Department of Defense's use of best practices in acquiring information technology systems. CHCS II is expected to cost about $1 billion to deliver full capability to almost 1,100 health facilities worldwide by 2008. GAO's objectives were to determine (1) what progress has been made against project commitments, (2) whether the system has been economically justified, and (3) whether effective technical and management controls are in place.
CHCS II is envisioned as a state-of-the-art automated medical information system allowing improved health-care decisions and lower medical and system costs. An expected highlight is computer-based patient records that doctors and other health service providers will be able to access from any military treatment facility, irrespective of location. While early CHCS II progress was limited, clear improvement has been evident over the past 2 years, as Defense has begun to embrace industry best practices. The first incremental release of the system, with a deployment decision scheduled for September 2002, is set to contain more capability than was originally planned. The current schedule is nevertheless 3 years beyond the initial estimate, due in part to major program changes and a system redesign; benefits are in question since measurement has not yet begun; and costs to date are about 2= times the 1998 estimate for deploying the first increment to one region. Until recently, Defense's basis for investing in this system has been an outdated cost/benefit analysis that did not reflect important changes in assumptions and, further, justified all system increments beyond the first solely on an economic analysis of the entire system. Officials now, however, are finalizing an updated analysis and have stated they plan to adopt an incremental approach to justifying investment in CHCS II, a best practice that successful organizations have been following for years. The department is moving forward in ensuring that effective acquisition controls are in place, but further progress can be made. Technical and management controls are largely in effect in several areas, including testing and risk management. But performance-based contracting is not being followed, resulting in the risk that CHCS II will take longer to acquire and cost more than necessary.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-02-345, Information Technology: Greater Use of Best Practices Can Reduce Risks in Acquiring Defense Health Care System
This is the accessible text file for GAO report number GAO-02-345
entitled 'Information Technology: Greater Use of Best Practices Can
Reduce Risks in Acquiring Defense Health Care System' which was
released on September 26, 2002.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
United States General Accounting Office:
GAO:
Report to the Chairman and Ranking Minority Member, Subcommittee on
Readiness and Management Support, Committee on Armed Services, U.S.
Senate:
September 2002:
Information Technology:
Greater Use of Best Practices Can Reduce Risks in Acquiring Defense
Health Care System:
GAO-02-345:
GAO Highlights:
Highlights of GAO-02-345, a report to the Chairman and Ranking Minority
Member, Subcommittee on Readiness and Management Support, Committee on
Armed Services, U.S. Senate.
Why GAO Did This Study:
This report examines the acquisition of the Composite Health Care
System (CHCS) II. It is one in a series of reports reviewing the
Department of Defense‘s use of best practices in acquiring information
technology systems. CHCS II is expected to cost about $1 billion to
deliver full capability to almost 1,100 health facilities worldwide by
2008. GAO‘s objectives were to determine (1) what progress has been
made against project commitments, (2) whether the system has been
economically justified, and (3) whether effective technical and
management controls are in place.
What GAO Found:
CHCS II is envisioned as a state-of-the-art automated medical
information system allowing improved health-care decisions and lower
medical and system costs. An expected highlight is computer-based
patient records that doctors and other health service providers will be
able to access from any military treatment facility, irrespective of
location (see figure). While early CHCS II progress was limited, clear
improvement has been evident over the past 2 years, as Defense has
begun to embrace industry best practices. The first incremental release
of the system, with a deployment decision scheduled for September 2002,
is set to contain more capability than was originally planned. The
current schedule is nevertheless 3 years beyond the initial estimate,
due in part to major program changes and a system redesign; benefits
are in question since measurement has not yet begun; and costs to date
are about 2½ times the 1998 estimate for deploying the first increment
to one region.
Until recently, Defense‘s basis for investing in this system has been an
outdated cost/benefit analysis that did not reflect important changes in
assumptions and, further, justified all system increments beyond the
first solely on an economic analysis of the entire system. Officials
now, however, are finalizing an updated analysis and have stated they
plan to adopt an incremental approach to justifying investment in CHCS
II, a best practice that successful organizations have been following
for years.
The department is moving forward in ensuring that effective acquisition
controls are in place, but further progress can be made. Technical and
management controls are largely in effect in several areas, including
testing and risk management. But performance-based contracting is not
being followed, resulting in the risk that CHCS II will take longer to
acquire and cost more than necessary.
Figure: Creating a Computer-based Patient Record:
[See PDF for image]
This figure is an illustration of the creation of a computer-based
patient record. The illustration depicts the following information:
Patient arrives:
* Diagnostics (blood pressure, heart rate, temperature, etc.);
* Triage (nurse asks about reason for visit, history of illness);
* Physical exam (doctor sees patient).
CHCS II workstation:
Date from Diagnostics, Triage, and Physical exam is entered, producing
a computer-based patient record, which is stored in a server.
Source: GAO, based on DOD data.
[End of figure]
What GAO Recommends:
GAO is making several recommendations to the Secretary of Defense,
including ensuring expanded use of best practices in managing CHCS II by
(1) modifying the project‘s investment strategy to justify investment
in each system release before beginning development, and measuring
return on investment; and (2) employing performance-based contracting
practices where possible on all future delivery orders. Defense
generally agreed with our recommendations.
This is a test for developing highlights for a GAO report. The full
report, including GAO's objectives, scope, methodology, and analysis,
is available at [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-02-
45]. For additional information about the report, contact Randolph C.
Hite (202-512-3439). To provide comments on this test highlights,
contact Keith Fultz (202-512-3200) or email HighlightsTest@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
DOD Has Made Recent Progress Since Missing Early Project Commitments:
DOD Has Not Adequately Justified Investments in CHCS II Releases, but
Improvements Are Under Way and Planned:
Important Technical and Management Controls Now Being Followed, but
Opportunities Exist For Further Use of Best Practices:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Comparison of CHCS and CHCS II Capabilities:
Appendix III: Detailed Explanation of How CHCS II Will Operate:
Appendix IV: CHCS II Release 1 Software Packages:
Appendix V: CHCS II Capabilities Planned for Releases 3-7, and Expected
Release Dates:
Appendix VI: GAO Contact and Staff Acknowledgments:
GAO Contact:
Acknowledgments:
Tables:
Table 1: CHCS II Releases 1 and 2 Capabilities and Expected Deployment
Dates:
Table 2: Division of CHCS II Management Responsibilities and Functions
Among DOD Units:
Table 3: MHS Enterprise Architecture Satisfaction of DOD Architecture
Framework Essential Products:
Table 4: Simplified Part of Information Exchange Matrix:
Table 5: Examples of MHS and CHCS II Mapping:
Table 6: CHCS II Release 1 Software Packages by Location:
Figures:
Figure 1: Simplified DOD Health Affairs Organization Chart:
Figure 2: Release 1 Interfaces with Four Existing Systems:
Figure 3: Time Line of CHCS II‘s Initial and Current Schedules:
Figure 4: CHCS II Expenditures to Date:
Figure 5: Number of Priority 1, 2, and 3 Defects Opened June 2001
through July 2002:
Figure 6: Unresolved Priority 1, 2, and 3 Defects Opened April 2001
through July 2002:
Figure 7: A Simplified Operational High-Level Graphic:
Figure 8: Creating a Computer-based Patient Record:
Figure 9: CHCS II Tri-level Architecture:
Figure 10: Example of CHCS II Regional Communications:
Abbreviations:
ASD(C3I): Assistant Secretary of Defense for Command, Control,
Communications, and Intelligence:
CIO: chief information officer:
CDR: clinical data repository:
COTS: commercial-off-the-shelf:
CHCS: Composite Health Care System:
CPR: computer-based patient record:
DISA: Defense Information Systems Agency:
DOD: Department of Defense:
GOTS: government-off-the-shelf:
IT: information technology:
MHS: Military Health System:
MTF: military treatment facilities:
SEI: Software Engineering Institute:
[End of section]
United States General Accounting Office:
Washington, DC 20548:
September 26, 2002:
The Honorable Daniel K. Akaka:
Chairman:
The Honorable James M. Inhofe:
Ranking Minority Member:
Subcommittee on Readiness and Management Support:
Committee on Armed Services:
United States Senate:
This report responds to your request that we review the Department of
Defense‘s (DOD) acquisition of the Composite Health Care System (CHCS)
II, an automated medical information system to be deployed to about
1,100 health facilities at 142 military installations worldwide.
According to the department, CHCS II will facilitate the worldwide
delivery of health care to active-duty service members, their
dependents, and other eligible beneficiaries, when care is received in
military facilities. DOD has invested about $320 million to acquire an
initial version of CHCS II, and expects to invest about $1 billion to
acquire and deploy the full system and an additional $3.2 billion to
operate and maintain the system over its useful life. DOD began the
last phase of testing of the initial version of the system in May 2002,
plans to make a deployment decision for this version in September 2002,
and plans to deploy it to all sites by October 2005. Over the next 6
years, DOD plans to acquire and deploy a series of more capable
versions of the system, delivering full capability to all health
facilities at all installations by 2008.
This report is one in a series examining DOD‘s use of best practices in
acquiring information technology (IT) systems. [Footnote 1] As agreed
with your offices, the objectives of our review were to determine (1)
what progress DOD has made against CHCS II project commitments,
including required system capabilities, expected system benefits, and
estimated project costs and milestones; (2) whether DOD has
economically justified its investment in the system; and (3) whether
DOD has effective technical and management controls in place for
acquiring the system. The controls reviewed were requirements
management, test management, architecture development and alignment,
risk management, and contract management.
Results in Brief:
Details of our objectives, scope, and methodology are included in
appendix I. Concerning progress to date, DOD did not meet its May 1998
commitment to deliver initial CHCS II system capabilities and
associated mission benefits in April 1999; and, since it did not
estimate the cost of delivering the initial system capabilities
(release 1), it does not have a cost commitment against which to
measure progress. Reasons for missing delivery of the capabilities
include initial use of a Web-based architecture that did not meet
system performance requirements, initial requirements that were ill-
defined, a later influx of requirements changes, and unexpected budget
cuts that forced changes in the project‘s scope and approach. By July
2000, 26 months later, the department had redefined its plans for CHCS
II to include adopting a new technical architecture, establishing a
means for controlling changes to requirements, committing to
incremental releases [Footnote 2] of system capabilities, and delaying
the decision date for deploying release 1 to January 2001, which was 21
months later than its May 1998 commitment. It did not, however,
similarly redefine benefits, or provide a cost commitment for the
initial version.
Since July 2000, DOD has made progress in delivering release 1‘s system
capabilities, although precisely which capabilities will be included in
this release have continued to change, and the deployment decision date
was delayed another 20 months, to September 2002. Progress against CHCS
II‘s benefits commitment is not yet known, because the project has yet
to measure the accrual of actual benefits even though versions of
release 1 have been used at test sites for about 2 years. Similarly,
progress against a cost commitment has not been measured because DOD
did not develop a cost estimate for release 1 until April 2002, only 5
months before this release is to be deployed. Available measures of
whether release 1 meets revised commitments [Footnote 3] indicate that
the system is maturing, but questions still exist as to the system‘s
operational efficiency. For example, while DOD has corrected all of the
most serious defects, about 50 defects affecting system efficiency
remain unresolved.
Second, DOD has not economically justified its investment in CHCS II
releases, but it plans to do so from this point forward. It initially
developed a single economic justification for CHCS II in 1998, which
has been used as the basis for its past and ongoing investment in
releases 1, 2, and 3. However, this justification had known limitations
in the cost estimate and was not updated to reflect material changes in
planned system capabilities, costs, and milestones. Further, although
DOD has adopted an incremental approach to acquiring and deploying
system releases, it did not treat investment in releases 2 and 3 as
separate investment decisions. This approach is not consistent with
best practices and federal requirements. As such, it has invested
hundreds of millions of dollars in the system over 4 years without
knowing whether the cost of a particular release is outweighed by its
benefits. Recently, DOD updated its analysis and reported that releases
1 and 2 are economically justified. CHCS II project officials have
acknowledged that incremental investment is a best practice that should
be followed, and stated that they will change their strategy for future
releases to include economically justifying each release before
investing in it and verifying each release‘s benefits and costs after
deployment.
Finally, DOD has appropriately given attention and priority to employing
certain acquisition best practices, including those related to managing
system requirements and test events, aligning the system architecture
with the DOD military health system enterprise architecture, [Footnote
4] and proactively managing project risks. Such practices better
position the department for successfully acquiring CHCS II.
Nevertheless, management can be improved upon. For example, performance-
based contracting methods are not being used to ensure contractor
accountability. Until this is changed, acquisition risks will remain
higher than they need to be.
DOD‘s recent progress on CHCS II is due in part to its attention and
commitment to adopting certain acquisition management best practices,
such as those governing requirements management. We are making
recommendations to the Secretary of Defense for similarly pursuing
improvements in CHCS II investment and other acquisition management
controls.
DOD provided what it termed ’official oral comments“ on a draft of this
report. In its comments, DOD generally agreed with the report‘s message
and recommendations. Its comments are discussed more fully in the
Agency Comments and Our Evaluation section of this report.
Background:
DOD operates a nationwide health care program, including overseas
facilities, for its health care beneficiaries. This program is just one
of the responsibilities of the Office of the Under Secretary of Defense
for Personnel and Readiness, along with recruiting, training,
educating, and a range of morale, welfare, and recreation services.
Within the Office of the Under Secretary is the Office of the Assistant
Secretary for Health Affairs, as well as assistant secretaries or
deputy undersecretaries for force management, reserve affairs,
readiness, and program integration.
The Assistant Secretary for Health Affairs is responsible for the
military health system (MHS) program (see fig. 1). The MHS program has
two missions: wartime readiness (maintaining the health of service
members and treating wartime casualties), and peacetime care (providing
for the health care needs of the families of active-duty members,
retirees, their families, and survivors). The Assistant Secretary for
Health Affairs establishes policy regarding health care for all DOD
beneficiaries and also plans and budgets for health care operations and
maintenance. At the same time, each military service has its own
medical department, headed by a surgeon general, which operates medical
facilities and recruits and funds military medical personnel. Health
Affairs, through the surgeons general, provides policy direction,
oversight, and resource support to these medical facilities, referred
to as military treatment facilities.
Figure 1: Simplified DOD Health Affairs Organization Chart:
[See PDF for image]
This figure is an organizational chart, depicting the following
relationships:
Top level:
Deputy Secretary of Defense;
Second level:
Under Secretary of Defense (Personnel and Readiness);
* Assistant Secretary of Defense (Health Affairs);
- Military Health System Program.
Assistant Secretary of Defense (Command, Control, Communications, and
Intelligence).
Secretary of the Army;
* Army Medical Service;
* Army Surgeon General (interfaces with the Military Health System
Program);
- Army Medical Facilities.
Secretary of the Navy;
* Bureau of Medicine and Surgery;
* Navy Surgeon General (interfaces with the Military Health System
Program);
- Navy Medical facilities.
Secretary of the Air Force;
* Air Force Medical Service;
* Air Force Surgeon General (interfaces with the Military Health System
Program);
- Air Force Medical facilities.
Source: GAO based on DOD data.
[End of figure]
MHS: A Large Health Care Provider:
Today, over 8 million people are eligible to receive care from MHS
facilities”about 80 percent retirees and dependents and about 20 percent
active-duty personnel. Reflecting the magnitude of this patient
population, MHS‘s worldwide operations are significant: about 130,000
personnel at about 1,000 Army, Navy, and Air Force medical facilities
divided into 12 domestic plus European, Pacific, and Latin American
regions. The fiscal year 2002 MHS budget is almost $24 billion, with
about $5 billion of this amount funded by the three services‘ budgets
for military medical personnel.
DOD provides about 75 percent of MHS services through these military
facilities, supplementing this by contracting for health services with
civilian providers. Active-duty members are required to obtain care at
military treatment facilities if such are available; in contrast,
retirees and dependents may obtain care at either military facilities,
on a space-available basis, or through civilian contract providers.
Since 1968, DOD has pursued the goal of providing computer support to
its hospitals and clinics. During fiscal years 1976 to 1984, DOD spent
about $222 million to acquire, implement, and operate various health-
care computer systems. The original CHCS, begun by DOD in 1988 and first
deployed in 1993, is the primary DOD medical information system now
used in all MHS facilities worldwide. This system supports DOD hospitals
and clinics in registering patients, documenting inpatient activity, and
providing laboratory, radiology, pharmacy, drug-interaction, and other
functions. Other medical information systems include the Ambulatory
Data System, Preventive Health Care Application, and the Nutrition
Management Information System. [Footnote 5] According to a 1996
analysis of the MHS mission, the existing medical information systems
cannot meet DOD‘s needs. The department thus initiated CHCS II in 1997
with the goal of assisting clinicians in making health-care decisions
and lowering medical and systems costs through, for example,
replacement of each of the above-cited systems. (Appendix II shows a
comparison of the capabilities provided by CHCS and CHCS II.)
CHCS II Design: A Brief Description:
CHCS II is designed to be a multi-level, open system based on a client-
server model. [Footnote 6] (Appendix III describes the system in
greater detail.) The system is to consist of user [Footnote 7]
workstations and application and security network computing devices,
located at military treatment facilities. DOD plans to divide the CHCS
II acquisition into seven releases. The decision to deploy release 1 is
scheduled for September 2002; release 2 for July 2003. Remaining
deployments are scheduled to begin at about 1-year intervals.
DOD‘s description of release 1 shows users creating and storing
computer-based patient records (CPRs) using 30 CHCS II workstation- and
computer-based software packages (21 commercial, 9 government-owned;
see app. IV for more information on these packages). DOD plans to
connect each facility‘s workstations and servers via each installation‘s
local or wide area networks. Each installation is also to be connected
through a wide area network [Footnote 8] to a defense computing center,
where the CPRs will be stored in a database known as the clinical data
repository. As release 1 is deployed to more installations, DOD intends
that medical providers will be able to access a patient‘s CPR from any
military treatment facility, no matter where the patient is being or
has been treated.
Release 1 is not intended to meet all needs, and thus additional system
capabilities are to be added over several years, beginning in July 2003.
DOD plans for release 1 to provide system security; user and system
interface controls; and various functions such as appointment
management, order-entry/management, reporting, preventive health care,
immunization tracking, and CPR management. Release 1 will target
ambulatory beneficiaries (out-patients). Other service categories, such
as inpatient support, are planned for future releases. Release 1 will
also replace the Ambulatory Data System and the Preventive Health Care
Application.
Release 1 is planned to interface with four systems: CHCS; the Defense
Enrollment Eligibility Reporting System, which receives and responds to
CHCS II requests for eligibility and immunization information; the
Third-party Outpatient Collection System, which receives third-party
billing data from CHCS II; and the Corporate Executive Information
System, which receives patient information from CHCS II that is used
for executive reporting (see fig. 2). Two additional interfaces are
planned for release 2, one to a new patient eligibility system and one
to a new primary care manager system. [Footnote 9]
Figure 2: Release 1 Interfaces with Four Existing Systems:
[See PDF for image]
This figure illustrates the Release 1 interface with four existing
systems as follows:
CHCS II:
Interface with CHCS: Global data files and order entry requests and
responses;
Interface with Defense Enrollment Eligibility Reporting System:
Eligibility and immunization requests and responses;
Interface with Corporate Executive Information System: Patient data for
data mining and reporting goes to CEIS;
Interface with Third-part Outpatient Collection System: Billing data
goes to third-party payers.
Source: GAO based on DOD data.
[End of figure]
The acquisition plan calls for release 1 to rely on certain existing
CHCS capabilities, including patient appointment/scheduling, radiology,
pharmacy, and laboratory; plans call for future releases to gradually
replace CHCS, allowing it to be shut down by 2008. Table 1 provides
details on releases 1 and 2 deployment dates and capabilities. Appendix
V provides details on capabilities and deployment dates for later
releases.
Table 1: CHCS II Releases 1 and 2 Capabilities and Expected Deployment
Dates:
Release number and expected deployment date: 1, September 2002;
Capabilities included:
* Automated health evaluation/assessment questionnaire;
* Beneficiary population health reports;
* Centralized health record repository;
* CHCS, eligibility, executive information, and third-party collection
system interfaces;
* Clinical outpatient graphical user interface;
* CPR;
* Enrollment and eligibility data;
* Immunization data;
* New ambulatory data and preventive health systems;
* Patient data transcription;
* Patient encounter documentation;
* Patient encounter results codes;
* Patient health history;
* Patient satisfaction reports;
* Patient self-assessment data entry;
* Provider profile data;
* Specialist consults results reports;
* User alerts and reminders;
* User role-based access to system.
Release number and expected deployment date: 2, July 2003;
Capabilities included:
* Automated clinical practice guidelines;
* Dental charting and documentation;
* Enhanced clinical decision support;
* Global health record repository;
* New eligibility and primary care manager systems interfaces;
* Optometric documentation and order entry;
* Theater (deployed) functions.
Source: GAO based on DOD data.
[End of table]
CHCS II Project Management:
Several DOD units within the Office of the Assistant Secretary for
Health Affairs, supported by other DOD components, are involved in
acquiring and deploying CHCS II. The MHS chief information officer
(CIO) under the Assistant Secretary is responsible for several MHS IT
projects, of which CHCS II is but one. MHS‘s clinical information
technology program office provides direct management of the project.
The Assistant Secretary of Defense for Command, Control,
Communications, and Intelligence (ASD(C3I)) is responsible for
overseeing the project and deciding whether it can proceed to the next
acquisition cycle milestone. Table 2 shows how management
responsibility for CHCS II is divided among DOD units.
Table 2: Division of CHCS II Management Responsibilities and Functions
Among DOD Units:
Entity: MHS CIO;
Responsibility/function: Oversees the MHS information management and
technology program.
Entity: Program Executive Office;
Responsibility/function: Directs all centrally managed MHS IT system
acquisitions, including oversight of procurement, development,
implementation, deployment, maintenance, and operations.
Entity: Clinical IT Program Office (includes the CHCS II project team);
Responsibility/function: Acquires and deploys CHCS II; monitors
contract performance; performs testing and training on use of the
product delivered by the contractor.
Entity: Other DOD organizations;
Responsibility/function: Service military treatment facilities upgrade
power needs, validate data conversions, and uninstall and remove older
system hardware. Another MHS program office provides communications
systems infrastructure. The Office of Program Analysis and Evaluation
is responsible for verifying and validating the reliability of economic
analyses for major projects such as CHCS II. The Army Test and
Evaluation Command is responsible for conducting the operational test.
The Navy Center for Cost Analysis is responsible for validating the
CHCS II life-cycle cost estimate.
Entity: ASD (C3I);
Responsibility/function: Approves the project to proceed through its
acquisition cycle on the basis of a review of key documents – an
independently evaluated life-cycle cost/benefit estimate, a component
cost analysis, and an acquisition strategy and project baseline. (Is
also the milestone decision authority for CHCS II, which is categorized
as a major automated information system initiative.)[a};
Entity: Joint Requirements Oversight Council;
Responsibility/function: Approves mission need and operational
requirements for automated information systems with joint
(multiservice) interest.
[A] A project that is either designated as such by the Assistant
Secretary, or estimated to require (a) project costs in any single year
in excess of $30 million; (b) total project costs in excess of $120
million; or (c) total life-cycle costs in excess of $360 million, all
in fiscal year 1996 constant dollars.
Source: GAO based on DOD data.
[End of table]
DOD Has Made Recent Progress Since Missing Early Project Commitments:
Best practices and related federal guidance emphasize the need for
disciplined processes and information to help ensure that projects are
implemented at acceptable costs and within reasonable and expected time
frames. [Footnote 10] They also recognize that acquisitions should
contribute to tangible, observable mission performance enhancements. In
short, projects are expected to meet the capability, schedule, benefit,
and cost commitments upon which their approval was justified.
For the last 4 years, MHS has invested about $320 million in CHCS II but
either does not know if it is meeting its commitments or has largely not
met them. The exception is in its system capability commitment, where it
is delivering more in release 1 than originally envisioned. The reasons
for missing commitments relate to a flawed initial design, user
dissatisfaction with the system prototype, additional requirements,
technical problems, such as slow network performance, and a funding cut
for the 1999 MHS budget. As a result, the deployment and concomitant
benefits have been delayed, and costs have risen substantially over the
initial costs approved for release 1. Nevertheless, release 1 is now in
the last phase of testing and available data suggest that it is largely
performing as intended, although some issues surrounding system
efficiency remain.
Overall Progress Against Project Commitments Has Been Limited, but
Recent Events Show Improvement:
During the first 2 years of the project, MHS made little progress
against the CHCS II capability, schedule, benefits, and cost
commitments it made in 1998. Progress has improved since then, however.
* Capability Commitment. CHCS II release 1 is intended to meet
foundational system functional, performance, and interface requirements,
allowing medical providers at military treatment facilities to exchange,
display, and place data into beneficiaries‘s medical records regardless
of where the patient is being treated or where past treatments occurred.
Release 1 is also expected to deliver more capabilities than the
original commitment due to the requirements added in 2001 in response
to user demands. Examples of added requirements (originally planned for
future releases) include Windows 2000, pathology order entry, patient
self-assessment, and certain patient medical charting features. MHS
reports that these additional capabilities added about $7 million to
the cost of release 1, and about 6 months to the schedule.
* Schedule Commitment. MHS plans to make a deployment decision for
release 1 in September 2002––over 3 years late. When the project was
approved in May 1998, MHS envisioned deploying a prototype system
beginning in October 1998 and beginning deployment of the initial
version about April 1999. The initial deployment has been delayed for
several reasons. The prototype architecture, which employed a Web-based
user interface, [Footnote 11] was found to cause slow system
performance. Further, medical personnel were not satisfied with other
system functions, such as the ability to view and print patient
documentation, causing requirements redefinition and a system redesign;
and the Joint Requirements Oversight Council added more CHCS II
requirements in 2000, which added 15 months to the schedule. DOD also
cut the MHS budget for fiscal year 1999 that MHS reports caused the
initial deployment to be delayed 6 months. System users demanded a
number of additional requirements in 2001, and system testing found
technical problems with the implementation, such as slow network
performance. This combination of the additional user requirements and
technical problems added another 20 months to the schedule, for a total
delay of 41 months past the original April 1999 estimate. Figure 3
provides a time line of CHCS II‘s initial and current schedules through
2003.
Figure 3: Time Line of CHCS II‘s Initial and Current Schedules:
[See PDF for image]
This figure is a time line of CHCS II‘s initial and current schedules,
as follows:
Initial plan:
Prototype development starts: early, 1998;
Prototype operational testing: late, 1998;
Prototype deployment: late, 1998;
Release 1 operational testing: mid-1999;
Release 1 initial operational capability: mid-2000;
Today: last, 2002.
Actual/Future plan:
Prototype development starts: early, 1998;
Prototype operational testing: early, 1999;
System assessment/prototype deployment canceled: mid-1999;
System redesign/release 1 development starts: last, 1999;
Release 2 development starts: late 1999;
Release 1 preliminary installation acceptance testing: late, 2000;
Release 3 development starts: late, 2001;
Release 1 final installation acceptance testing: early, 2002;
Release 1 operational testing: mid-2002;
Release 1 deployment: late 2002;
Release 1 initial operational capability: early, 2003;
Release 2 deployment: mid-2003.
Source: GAO based on DOD data.
[End of figure]
* Benefits Commitment. The 1998 economic analysis estimated CHCS II
benefits at about $5.7 billion in 1998 dollars, including $86 million in
benefits between fiscal years 2000 and 2002. A May 2002 update of this
analysis estimates benefits of about $6.6 billion in 1998 dollars, but
accrual of benefits is not expected to begin until fiscal year 2003.
Thus, MHS has not met its original benefit commitments. In addition, it
has yet to begin verifying whether its benefit expectations are being
met, and only began validating its plan for measuring benefit accrual
in March 2002, although versions of release 1, each providing more
capabilities, have been operating at test sites since June 2000.
According to project officials, they did not begin this process earlier
because users at the sites would not yet have been in a position to
determine how CHCS II affected their work.
* Cost Commitment. MHS did not prepare a baseline cost estimate for
acquiring and deploying release 1 until April 2002. However, on the
basis of the 1998 economic analysis, DOD did approve project funding of
about $109 million for acquiring and deploying CHCS II from 1997
through 1999. This included the acquisition cost for release 1, plus
costs to deploy the system to the sites in a single MHS region from
April through September 1999. As of April 2002, MHS reports it has
spent about $284 million to acquire release 1, which is about two and
one-half times the amount approved in 1998 to acquire and deploy
release 1 to a single MHS region. As with the schedule delays, MHS
reports that these additional costs are due to the problems with the
prototype, additional requirements, and technical complexity. In
addition to the release 1, MHS reports that it has spent about $35
million to date on release 2 and about $700,000 on release 3, as shown
in figure 4.
Figure 4: CHCS II Expenditures to Date:
[See PDF for image]
This figure is a pie-chart, depicting the following data:
CHCS II Expenditures to Date:
Release 1: 88.8%;
Release 2: 11.0%;
Release 3: 0.2%.
Source: GAO based on DOD data.
[End of figure]
Recent Test Results and System Defect Rates Show Initial Release Is
Maturing, But Issues Remain:
Two indicators of system maturity are the results of system testing and
the number of system defects remaining. One type of testing that both
best practices and federal guidance advocate before a system is
deployed is system acceptance testing, the purpose of which is to
verify that the system meets key technical and system requirements.
Defects are system problems that require resolution. As a system
matures, the number of defects should show a downward trend. The CHCS
II project categorizes defects by severity, ranging from 1 to 5, with 1
being the most serious and the highest priority. [Footnote 12]
According to the test plan, priority 1 and 2 defects, which jeopardize
patient safety or adversely affect mission-essential capabilities, need
to be resolved before the system can continue to operational testing
and deployment. Priority 3, 4, and 5 defects do not need to be resolved
since these defects have less effect on the system. Both the system
acceptance test results and trends in priority 1 and 2 defects show
that release 1 is maturing. Trends in priority 3 defects, which require
inefficient workarounds to overcome, do not yet show a maturing trend.
Acceptance Test Results:
The project team conducted the acceptance test to validate that the
system is effective and suitable when operating in a production-
representative military treatment facility environment, and that the
system meets all critical operational issues and key performance
parameters. An outcome of acceptance testing is a determination of the
system‘s maturity and its readiness to proceed to the final phase of
testing”operational testing and evaluation. [Footnote 13]
The acceptance test was sufficiently scoped to provide a basis for
making each of these determinations. For example, the test was
conducted at 4 sites and covered all 28 categories of release 1
functionality, [Footnote 14] and all 1,138 release 1 requirements,
including performance (e.g., system availability) and interface
requirements. While two functionality categories”telephone consults and
self-assessments”were not exercised at all test sites, this was
mitigated by the fact that ’telephone consult“ was exercised at one
site using an automated test tool, and ’self-assessment“ was tested at
one site and also used by the test team on test patients at another
site.
The acceptance test also defined 15 system maturity parameters/
indicators. [Footnote 15] According to the test results, 10 of these 15
were fully satisfied (judged to be mature) and 5 were partially
satisfied (judged to be of limited maturity). For example, one
parameter called ’functional completeness“ required that 90 percent of
the system requirements be met. According to the test results, release
1 exceeded this requirement, meeting 100 percent of the requirements.
Other examples of parameters that were fully met are ’fault closure
rate“ and ’fault severity/safety,“ meaning that defect density is
decreasing and no priority 1 and 2 defects exist, respectively.
According to the test results and our analysis of defects (discussed in
the next section of this report), both were met.
For the five parameters that were not fully satisfied and thus rated as
limited maturity, we analyzed the reasons provided for these
determinations, and believe that the parameters were sufficiently
satisfied to mitigate any concerns about whether the parameter was
sufficiently satisfied. For example, one of the five is ’key
performance parameters“ which actually consists of nine performance
requirements that must be met. Of these nine, test results show that
eight were met and one was not. However, the one that was not met is
actually a performance requirement for release 2 and not release 1.
Similarly, one of the five is ’system availability,“ which for CHCS II
is 99 percent system availability. According to the test results, this
parameter was not met because a scheduled equipment replacement at the
Defense Information Systems Agency computing center caused the system
to not be operational while the replacement occurred. However, as we
have previously reported, [Footnote 16] system availability is
generally regarded as the time that a system is operating
satisfactorily, expressed as percentage of the time that the system is
required to be operational (i.e., excluding the time that scheduled
system maintenance is occurring). Thus, the down time associated with
the scheduled equipment replacement should not have been used in
measuring system availability during the acceptance test, and according
to the test report, if the equipment replacement had occurred during
nonclinic hours, the system would have met the 99 percent requirement.
On the basis of the 15 maturity parameter/indicator ratings, the project
office determined that release 1 was mature, meaning that the testers
had high confidence that the system was ready for the final phase of
testing”operational testing and evaluation.
Trends in System Defects:
Another indicator of system maturity and quality is defect density,
which can be measured in a number of ways, including the trend in the
number of defects being reported and the trend in the number of
unresolved (uncorrected) defects. Available data on these measures show
a generally positive maturity trend, although issues of system
efficiency and progress relative to defect expectations exist. For
example, after adding requirements in June 2001, the number of defects
opened each month began to rise, as shown in figure 5. By January 2002,
however, this number began to drop and by the end of July 2002 was
about zero for priority 1, 2, and 3 defects combined.
Figure 5: Number of Priority 1, 2, and 3 Defects Opened June 2001
through July 2002:
[See PDF for image]
This figure is a multiple line graph depicting the number of priority
1, 2, and 3 defects opened June 2001 through July 2002. The vertical
axis of the graph represents number of defects from 0 to 50. The
horizontal axis represents months from June 2001 through July 2002.
Lines depict Priority 1 and 2 defects combined and Priority 3 defects.
Shaded ares indicate the system acceptance testing period and the
operational testing and evaluation period.
Source: GAO, based on DOD data.
[End of figure]
The number of unresolved defects (or defects open each month) also
began to rise after last June, as shown in figure 6. Since February,
however, this number, especially for priority 1s and 2s, has dropped.
As of the end of July 2002, all priority 1 and 2 defects had been
resolved, but 46 priority 3 defects remained open.
While not as serious as priority 1s and 2s, priority 3 defects are still
detrimental because they require workarounds of some sort, which by
definition decrease system efficiency. For example, one unresolved
priority 3 defect is when CHCS II prevents terminal access because the
user had not employed the workstation within the required time limit.
When this happens, reentering the user password should allow renewed
access, but does not. This means that the workstation is unusable until
it can be restarted, resulting in lost work time. The test plan only
requires priority 1 and 2 defects to be resolved before moving on to
operational testing and deployment. However, according to DOD
officials, they plan to correct all defects identified prior to July
31, 2002, before deploying release 1, except for 11 that are embedded
in vendor commercial-off-the-shelf software packages and thus must be
corrected by the respective vendors.
Figure 6: Unresolved Priority 1, 2, and 3 Defects Opened April 2001
through July 2002:
[See PDF for image]
This figure is a multiple line graph depicting the number of unresolved
priority 1, 2, and 3 defects opened April 2001 through July 2002. The
vertical axis of the graph represents number of defects from 0 to 90.
The horizontal axis represents months from June 2001 through July 2002.
Lines depict Priority 1 and 2 defects combined and Priority 3 defects.
Shaded ares indicate the system acceptance testing period and the
operational testing and evaluation period.
Source: GAO, based on DOD data.
[End of figure]
DOD Has Not Adequately Justified Investments in CHCS II Releases, but
Improvements Are Under Way and Planned:
Between the project‘s start in 1997 and April 2002, MHS invested
hundreds of millions of dollars in CHCS II without following all IT
investment best practices and related federal guidance that call for
separating large systems into smaller increments and making informed
return-on-investment decisions based on reliable estimates of
incremental project costs, benefits, and risks. Instead, MHS has
justified its past and ongoing investment in releases 1, 2, and 3 using
an outdated analysis of costs and benefits that, until recently, did
not reflect material changes to cost and benefit assumptions. Moreover,
MHS did not treat investment in releases 2 and 3 as separate decisions,
instead viewing these releases as being justified as part of MHS‘s
economic analysis of the entire CHCS II system (all releases). Such a
monolithic approach to analyzing and justifying a system‘s return on
investment has been abandoned by successful organizations as inherently
unreliable because it relies on predictions of many variables over many
years. According to CHCS II project officials, they will adopt an
incremental approach to investing in releases from this point forward.
Reliable Economic Analysis and Incremental Investment Are Tenets of
Effective Investment Management:
The Clinger-Cohen Act of 1996 and OMB guidance [Footnote 17] provide a
framework for IT investment management that comports with recognized
best practices. Together, they set requirements for (1) economically
justifying proposed projects on the basis of reliable analyses of
expected life-cycle costs, benefits, and risks; (2) using these
analyses throughout a project‘s life cycle as the basis for investment
selection, control, and evaluation decision-making; and (3) doing so
for large projects (to the maximum extent practical) by dividing them
into a series of smaller, incremental subprojects or releases. By doing
so, the tremendous risk associated with investing large sums of money
over many years in anticipation of delivering capabilities and expected
business value far into the future can be spread across project
fragments that are smaller, of shorter duration, and capable of being
more reliably justified and more effectively measured against cost,
benefit, and risk expectations. DOD policy also reflects these
investment principles by requiring that investments be justified by an
economic analysis and, more recently, [Footnote 18] that investment
decisions for major projects such as CHCS II be made incrementally to
better ensure that each increment delivers measurable benefit,
independent of future increments.
DOD Did Not Economically Justify Past, Ongoing Investments in CHCS II,
But Has Taken Steps to Do So:
IT investment management best practices and federal guidance advocate
economically justifying proposed investments on the basis of a net
present value benefit-to-cost ratio that is greater than 1, basing that
ratio on reliable analyses of expected life-cycle benefits, costs, and
risks, and updating the analysis when significant system changes occur.
MHS has not followed this practice. It originally justified its
investment in CHCS II in 1998, with an economic analysis that showed a
1.14 to 1 benefit-to-cost ratio for the total program and a 1.09 to 1
ratio for the initial increment (based on present value adjustments to
1998 dollars). However, in January 1999 the DOD Inspector General
reported that the cost estimate was based on assumptions that could be
flawed and result in estimate inaccuracies, and that the analysis did
not disclose this or its return-on-investment implications. [Footnote
19] Moreover, since this analysis was developed, the project has
undergone a number of changes affecting its cost and benefits profile,
as previously discussed.
MHS prepared two draft updates to its 1998 economic analysis (January
and August 2000), but it did not approve an update until May 2002.
According to project officials, the approved version was delayed because
system requirements were in a state of flux and did not become stable
until late 2000. Project officials also stated that the draft updates,
which had benefit-to-cost ratios greater than 1, provided them with
sufficient assurance that continued investment in CHCS II (releases 1,
2, and 3) was justified. However, according to the project office‘s
quarterly reports during 2000, CHCS II was still in a state of change
during and after the time of these updates. As stated in the October
2000 quarterly report, the economic analysis required revision to
reflect recent project developments and a change in the system release
strategy.
The May 2002 economic analysis estimates life-cycle benefits and costs
for the entire CHCS II system, including benefits of about $6.5 billion
and costs of about $4.3 billion (both in 1998 dollars). [Footnote 20]
When converted to present values, these estimates are approximately
$4.6 billion and $2.8 billion, respectively, producing a benefit-to-
cost ratio of 1.63 to 1 and a net present value of about $1.78 billion.
[Footnote 21] However, this analysis is still undergoing review by DOD
stakeholders. For example, the analysis‘s lifecycle cost estimate,
which was developed by MHS, is currently being validated by the Naval
Center for Cost Analysis. Because this analysis is still being
reviewed, we did not evaluate it.
Incremental Justification Planned for Future CHCS II Investments:
Incremental investment management involves three fundamental
components: (1) acquiring a large system in a series of smaller
increments; (2) individually justifying investment in each separate
increment on the basis of costs, benefits, and risks; and (3)
monitoring actual benefits achieved and costs incurred on ongoing
increments. MHS has structured CHCS II into a series of seven
increments (releases). However, until recently it had not followed best
practices in incrementally justifying investments in all system
releases. More specifically, it did not treat releases 2 and 3 as
separate investments. Further, it has yet to determine whether actual
benefit and cost expectations are being met by the first release.
According to project officials, this is because DOD policy did not
require them to do so, and the DOD CIO, who is the project‘s milestone
decision authority, approved the 1998 economic analysis and granted
approval to proceed.
Nevertheless, the May 2002 economic analysis, which was developed
during the course of our review and is currently under review within the
department, does define release 1 and 2 costs and benefits. The release
1 analysis reports life-cycle benefits and costs of $3.4 billion and
$2.8 billion, respectively (in 1998 dollars). When converted to present
values, these estimates are approximately $2.4 billion and $2.1
billion, respectively, producing a benefit-to-cost ratio of 1.12 to 1
and a net present value of about $255 million. The release 2 analysis
reports life-cycle benefits and costs of $509 million and $103 million,
respectively (in 1998 dollars). When converted to present values, these
estimates are approximately $359 million and $84 million, producing a
benefit-to-cost ratio of 4.25 to 1 and a net present value of about
$275 million. The analysis does not, however, address release 3.
According to project officials, they have recently decided to
economically justify all future system releases as separate investments
based on updates to the CHCS II economic analysis, and they plan to
determine whether return-on-investment expectations for deployed
releases are being met. The officials also stated that they would
analyze the benefits and costs of all future releases during MHS‘s
annual IT investment portfolio reviews and decide whether to fund them.
Further, they stated that actual benefits and costs from deployed
releases will be measured for each release, starting in February 2003
for release 1, and the CHCS II investment strategy will be updated to
reflect these changes.
Important Technical and Management Controls Now Being Followed, but
Opportunities Exist for Further Use of Best Practices:
MHS has generally established technical and management control best
practices in key areas: requirements management, test management,
architecture development and alignment, and risk management. This has
not been the case throughout the project‘s life, but project officials
said that they have devoted considerable management attention and given
priority to applying lessons learned and acting on the results of our
work, and they have made improvements to apply best practices.
Nevertheless, the CHCS II project office can improve its risk
management efforts, and it is still not following performance-based
contracting best practices. According to project officials, they did
not know how to apply such contracting practices to a system
acquisition, but have recently begun to develop this expertise. By not
following these practices, MHS risks paying more and taking longer to
acquire CHCS II than necessary.
Key Requirements Management Controls Have Recently Been Established:
Effectively managing requirements involves establishing and maintaining
agreement among the project team–including end users and contractors”on
what the system is to do (functionality), how well it is to do it
(performance), and how it is to interact with other systems
(interfaces). Best practices [Footnote 22] for managing requirements
include (1) adhering to a documented requirements management plan, (2)
establishing a validated set of requirements that serves as the
baseline against which changes are made, (3) controlling changes to the
baseline, (4) maintaining traceability among requirements and related
project deliverables, and (5) involving end users in developing and
changing requirements.
For CHCS II, MHS has defined products and processes that meet each of
these tenets of effective requirements management. First, there is a
documented CHCS II requirements management plan with specific written
steps governing development and maintenance of requirements. Second,
the requirements for release 1 have been validated and approved by CHCS
II users, and a baseline requirements set exists. Third, a formal change
control process has been put in place, which requires change control
board approval for baseline modifications based on the changes‘s impact
on the project. Fourth, a procedure exists for tracking requirements to
other work products by allowing changes to the contractor‘s requirements
database only from DOD‘s requirements database. Last, the process
provides for end user participation in both developing and approving
changes to existing requirements and addition of new requirements.
To confirm that established requirements management controls were
being followed, we reviewed requirements for two CHCS II capabilities–
clinical practice guidelines, which were defined in 2000, the first
year the current management controls were put in place, and patient
index, which were defined in 2001. In both cases, we determined that
defined process controls were being followed. For example, in both
cases, users were involved in developing and changing requirements and
requirements were baselined and put under change control. Also in both
cases, changes to requirements were assessed in terms of costs,
benefits, and risks before being accepted.
According to project officials, they did not effectively manage
requirements until 2000. A 1999 DOD Inspector General report affirms
early problems with requirements management, [Footnote 23] as does a
report in October 1999 from a CHCS II test team concluding that the
lack of complete requirements caused test problems. In response, MHS
redefined how requirements would be managed, adopting the current
process during the 2000-2001 timeframe. By doing so, the risks
associated with defining complete and correct system requirements, and
preventing uncontrolled changes, commonly referred to as ’requirements
creep,“ have been mitigated.
Key System Acceptance Testing Management Controls in Place:
Complete and thorough testing is essential to providing reasonable
assurance that systems perform as intended. Testing a system is not a
one time event, but rather is a series of test events throughout the
systems development and maintenance cycle, each of which addressing
different levels and aspects of the system and building on the results
of previous tests. Among the types of tests are (1) tests of small
units of software, known as unit testing; (2) tests of whether an
integrated version of the system meets defined requirements
(functional, performance, and interface), referred to as acceptance
testing; and (3) tests of whether the system meets the needs of users
in an operational setting, called operational testing.
Best practices including our own guidance [Footnote 24] recommend that
organizations implement a structured and disciplined approach to
managing each type of tests. DOD policy requires a similar approach. In
general, effectively managing a test event, such as system acceptance
testing, entails (1) developing a test plan that defines, for example,
test objectives, scope, schedules, resource needs, locations, and
documentation and reporting requirements; (2) preparing test
procedures, based on the plan, that specify test cases, steps, data,
inputs, and expected outputs, and that are traceable to requirements;
(3) defining test exit criteria, which establish the requirements that
must be met to successfully complete testing; (4) executing tests in
accordance with plans and procedures; (5) documenting test results in
accordance with plans and procedures; (6) identifying, prioritizing,
and correcting defects, and re-testing defect corrections; (7)
comparing test results with exit criteria to ensure that specified
requirements are met; and (8) reporting test results to management in
accordance with plans and procedures.
Between January and April 2002, MHS conducted system acceptance
testing. Our analysis of the management of this test event showed that
key tenets of effective test management were met. For example, MHS
prepared test plans that addressed test objectives, scope, schedules,
resource needs, locations, and documentation and reporting
requirements. Also, MHS developed detailed test procedures that
included, among other things, test steps, cases, data, inputs, and
expected outcomes. Further, our analysis showed that test procedures
were directly linked to functional, performance, and interface
requirements. Specifically, we analyzed a statistically valid sample of
59 requirements against the test procedures [Footnote 25] and found
that 57 were traceable to specific test procedures. With respect
to the two other requirements, one was a duplicate, and the other was
not a CHCS II requirement but rather a requirement for an interfacing
system that was inadvertently associated with CHCS II.
Additionally, the test plan defined acceptance test exit criteria as
closing all priority 1 and 2 defects, developing workarounds for
priority 3 defects, freezing the system baseline, and demonstrating
system interoperability and stability. Further, MHS executed the
acceptance test between January and April 2002 and documented the
results in an acceptance report.
During and after the test, DOD identified, prioritized, and corrected
defects, closing all priority 1 and 2 defects. Finally, in the
acceptance report, DOD compared test results with exit criteria and
provided the test results to management.
CHCS II and MHS Architectural Alignment Is Occurring:
Developing, maintaining, and using architectures, or blueprints, is a
best practice in engineering both individual systems and entire
enterprises. [Footnote 26] Requirements for having and using
architectures to guide and constrain IT investment decisionmaking are
also addressed in federal law and guidance, [Footnote 27] and in DOD
policy. [Footnote 28] When acquiring or developing new IT systems or
maintaining existing systems, these sources recognize that it is
very important to ensure that systems are built and modified (i.e.,
’architected“) within the context of the architecture for the
enterprise that the system supports. To do less risks having systems
that are, for example, duplicative and not integrated. In the case of
CHCS II, we found that the system and the MHS enterprise architecture
are generally aligned.
MHS Enterprise Architecture:
An enterprise architecture is a systematically derived snapshot”in
useful models, diagrams, and narrative”of a given entity‘s operations
(business and systems), including how its operations are performed, what
information and technology are used to perform the operations, where the
operations are performed, who performs them, and when and why they
are performed. The architecture describes the entity in both logical
terms (e.g., interrelated functions, information needs and flows, work
locations, systems, and applications) and technical terms (e.g.,
hardware, software, data, communications, and security). Enterprise
architectures provide these perspectives for both the entity‘s current,
or ’as is“ environment, and for its target, or ’to be“ environment;
they also provide a high-level capital investment roadmap for moving
from one environment to the other.
Managed properly, an enterprise architecture can clarify and help
optimize the interdependencies and interrelationships among a given
entity‘s business operations and the underlying systems and technical
infrastructure that support these operations. [Footnote 29] Our
experience with federal agencies has shown that attempting to define
system-level architectures (e.g., develop requirements and design
specifications) and use these systems architectures to build systems
without having an enterprise architecture and aligning its systems‘
architectures with the enterprise architecture often results in systems
that are duplicative, not well integrated, unnecessarily costly to
maintain, and limited in terms of optimizing mission performance.
MHS has developed an initial version of an enterprise architecture.
[Footnote 30] We analyzed this architecture against the requirements
for an enterprise architecture as defined by the DOD architecture
framework, [Footnote 31] which specifies the requirements that all DOD
enterprise architectures are to meet. As we have previously reported,
this framework is consistent with commercial architecture frameworks.
[Footnote 32] According to the DOD framework, enterprise architectures
must have seven essential products, including a high-level operational
concept graphic, information exchange matrix, and technical
architecture profile. Our analysis shows that the current version of
the MHS enterprise architecture has each of these products (see table
3).
Table 3: MHS Enterprise Architecture Satisfaction of DOD Architecture
Framework Essential Products:
Product: Enterprise view and summary information;
Description: Serves as planning guide and summarizes ’who, what, when,
why, and how“for architecture to be developed;
MHS architecture satisfies? Yes.
Product: Integrated dictionary;
Description: Provides a central source for definitions of all terms
used in all architecture products;
MHS architecture satisfies? Yes.
Product: High-level operational concept graphic;
Description: Shows a high-level graphic description of operational
concept, including organizations, missions, and geographic distribution
of assets;
MHS architecture satisfies? Yes.
Product: Operational node connectivity description;
Description: Identifies organizational elements that produce, process,
and consume information; need to exchange information between elements;
and characteristics of information exchanged (content, media, volume
requirements, security classification, timeliness, and interoperability
requirements);
MHS architecture satisfies? Yes.
Product: Operational information exchange matrix;
Description: Provides information exchange requirements, identifying
who exchanges what information with whom, why information is necessary,
and how it is needed;
MHS architecture satisfies? Yes.
Product: System interface description;
Description: Links operational and systems architecture views by
depicting information systems and their interfaces to organizational
elements that produce, process, and consume information;
MHS architecture satisfies? Yes.
Product: Technical architecture profile;
Description: Establishes a set of rules governing system implementation
and operation; references existing technical guidance and discusses how
that guidance has been or needs to be implemented.
MHS architecture satisfies? Yes.
Source: GAO, based on DOD data.
[End of table]
For example, MHS‘s operational high-level graphic illustrates four
business areas: access to care, provision of health services, population
health management, and manage the business. (See fig. 7 for a simplified
version of the operational high-level graphic.) These business areas are
then decomposed into activities. The activities, in part or whole, are
carried out by MHS systems.
Figure 7: A Simplified Operational High-Level Graphic:
[See PDF for image]
This figure is an illustration of a Simplified Operational High-Level
Graphic. The illustration contains two concentric circles. The core
circle is the Military Health System. Surrounding it is a circle
divided into four quadrants, as follows:
Population health management:
* Develop population management practices;
* Evaluate;
* Implement tools/manage processes;
* Define/assess population.
Provision of health services:
* Plan health services;
* Deliver health services;
* Assess beneficiary health status;
* Coordinate and integrate health services;
* Ensure quality of health;
* Manage information/manage documentation.
Manage the business:
* Deliver worldwide logistics;
* Patient financial management;
* Manage finances;
* Manage human resources;
* Review/improve business management;
* Support managed care contracting;
* Perform medical management.
Access to care:
* Manage patient movement/encounter;
* Perform assessment and plan for care;
* Assess effectiveness of access to care;
* Manage enrollment and eligibility;
* Support community outreach;
* Schedule services and check-in;
* Support beneficiary services.
Source: GAO, based on DOD data.
[End of figure]
As another example, DOD‘s framework requires an information exchange
matrix that identifies who exchanges what information with whom, why
the information is necessary, and how it is needed. The MHS enterprise
architecture has this matrix, and for each information exchange, it
identifies the information source and destination, criticality and
timeliness, and frequency and format requirements. (See table 4.)
Table 4: Simplified Part of Information Exchange Matrix:
Information exchange: Eligibility determination;
Source (Who?): Enrollment/eligibility management staff;
Destination (Who?): Enterprise-wide registration staff;
Timeliness (Why?): Seconds-hours-days;
Criticality (Why?): Critical;
Frequency (How?): Event-driven;
Media type (How?): Data, text.
Information exchange: Customer data release agreement;
Source (Who?): Enrollment/eligibility management staff;
Destination (Who?): Primary care manager;
Timeliness (Why?): Seconds-hours-days;
Criticality (Why?): High;
Frequency (How?): Event-driven;
Media type (How?): Data, text.
Information exchange: Encounter (administrative) data;
Source (Who?): Enterprise-wide registration staff;
Destination (Who?): Enterprise-wide scheduling staff
Timeliness (Why?): Seconds-minutes;
Criticality (Why?): Routine.
Frequency (How?): Event-driven;
Media type (How?): Data.
Source: GAO, based on DOD data.
[End of table]
A third example pertains to DOD‘s requirement that component
architectures contain a profile of related DOD technical standards.
[Footnote 33] MHS‘s enterprise has such a standards profile, and this
profile is aligned with the relevant DOD technical standards. For
example, DOD requires a standard graphic data interchange, specific
Windows management standards, and use of the federal information
processing standard for password usage. The MHS architecture specifies
each of these standard requirements in its standards profile.
CHCS II System Architecture:
MHS has also developed a CHCS II architecture to define a level of
system detail and specificity that is not found in an enterprise
architecture, but which is needed in order to build the system. The
system architecture is articulated in a number of system engineering
documents, such as the system requirements document, the system design
specifications, a data dictionary and database description, and
commercial-off-the-shelf application and system software descriptions.
According to MHS officials, the CHCS II architecture is fully aligned
with the MHS architecture. They attributed this alignment largely to
the fact that the two architectures were developed at the same time by
the same individuals.
To verify this alignment, we compared the two architectures at both the
logical and the technical levels. Our analysis showed that the CHCS II
system architecture is aligned with the MHS enterprise architecture. At
the logical level, we identified 52 capabilities that CHCS II is to
perform in support of the MHS mission. Of these 52 capabilities, we
judgmentally selected 13, or 25 percent, from 3 of the 4 MHS business
areas that relate to CHCS II, and compared them with activities defined
in MHS‘s enterprise architecture. All 13 were aligned with the MHS
enterprise architecture. (See table 5 for three examples.)
Table 5: Examples of MHS and CHCS II Mapping:
MHS business area: Access to Care;
MHS activity: Check-in patient;
CHCS II capability: Patient registration.
MHS business area: Population Health Management;
MHS activity: Educate patients concerning new practices;
CHCS II capability: Patient education.
MHS business area: Provision of Health Services;
MHS activity: Document care plans and delivery;
CHCS II capability: Clinical documentation.
Source: GAO based on DOD data.
[End of table]
At the technical level, we compared the MHS technical standards profile
and the CHCS II system architecture to determine if they specified the
same standards. MHS‘s profile identifies 61 technical standards that are
appropriate for CHCS II. Of the 61 technical standards, 59 are
specified in the CHCS II system architecture. For instance, DOD
requires that medical systems conform to the defense information
infrastructure common operating environment. MHS and CHCS II plans show
that CHCS II is to be compliant with this standard in release 2. In
addition, DOD requires that medical systems follow certain file
transfer standards, and CHCS II architecture documentation specifies
these standards. Further, DOD requires that a standard database
language be used for medical systems, and CHCS II architecture
documentation specifies this standard language.
The two standards in the MHS standards profile that are not specified in
the CHCS II system architecture relate to document interchange. These
standards are not currently applicable because the interfaces they
apply to are for future releases.
Risk Management Controls in Place, but Application of Controls Can Be
Improved:
Managing project risk means proactively identifying facts and
circumstances that increase the probability of failing to meet project
commitments, and taking steps to prevent this from occurring. Best
practices and federal guidance34 advocate risk management. To be
effective, risk management activities should be (1) based on documented
policies and procedures and (2) executed according to a written plan
that provides for identifying and prioritizing risks, developing and
implementing appropriate risk mitigation strategies, and tracking and
reporting on progress in implementing the strategies. By doing so,
potential problems can be avoided before they manifest themselves into
cost, schedule, and performance shortfalls.
MHS has largely implemented a process for managing CHCS II risks that
meets risk management best practices. Using a documented risk
management policy and associated procedures, project officials have
developed and are implementing a written plan for managing CHCS II
risks. Under this plan, identified risks are captured in a database,
and each risk is assigned a priority of 1 to 5, with 1 being the most
serious and the highest priority. [Footnote 35] Further, a strategy for
mitigating each risk is defined and its implementation is managed,
including assigning accountability for the risk, tracking status of the
risk, and reporting on progress in implementing the risk mitigation
strategy.
Using its risk management approach, MHS has identified 99 risks over the
life of the project, including 27 priority 1 and 2 risks for release 1
and release 2. To verify that the established risk management approach
was being followed, we reviewed the history and status of these
priority 1 and 2 risks and found that they were generally being managed
in a manner consistent with the process. For example, of the five open
priority 1 and 2 risks associated with release 1 of the system, all had
risk mitigation strategies that were being implemented and tracked
through formal status reporting. However, of the five priority 1 and 2
risks associated with release 2, four were being reported as active but
no mitigation actions were being reported over the last year for any of
the four. According to project officials and documentation, this was
because the four were actually inactive but the risk database had not
been updated. The project office has since updated the database.
In addition, our analysis led us to question why one priority 1
risk”user acceptance of the system”had been closed and was no longer
being actively managed since user acceptance is a common risk of
commercial-off-the-shelf-based system solutions and, given the status
of CHCS II deployment, user interaction with the system to date has
been extremely limited (i.e., confined largely to only the
approximately 100 medical personnel who have participated in system
acceptance testing). In response, the project office returned this risk
to active status, and it is now being formally managed using a
mitigation strategy and tracked via formal reporting.
Project officials attributed these instances to lapses in updating the
risk database. The quality of this database is important, as it defines
where limited resources will be focused to thwart the emergence of real
problems with real consequences. Without an accurate and complete
inventory of risks, their status, and the progress of strategies to
address them, the chances of system cost, schedule, and performance
problems occurring are increased.
Performance-based Contract Management Practices Not Being Used:
Performance-based contracting is a recognized best practice that federal
procurement policy reflects. [Footnote 36] Specifically, this policy
requires agencies to use performance-based contracting methods to the
maximum extent practicable when acquiring services. Moreover, the CHCS
II acquisition strategy calls for performance-based contracting
practices.
To qualify as performance-based under federal regulations, an
acquisition should have (1) requirements that define the work to be
performed in measurable terms; (2) standards (quality or timeliness)
that are tied to performance requirements; (3) a quality assurance plan
that describes how the contractor‘s performance in meeting requirements
will be measured against standards; and (4) positive and negative
incentives.
In the delivery orders that it has executed to date with the CHCS II
integration contractor, the project office has not fully satisfied any
of these tenets. First, the delivery orders contain performance
requirements and the requirements are written in measurable, mission-
related terms. However, the terms are allowed to change with each
delivery order modification without holding the contractor accountable
for performance up to the point of the modification. Instead, both the
requirements and performance prior to the modification are supplanted
by the modified requirements, and contractor performance begins anew.
In light of the number of modifications that have accompanied CHCS II
delivery orders, this results in considerable contractor activity not
being managed on the basis of performance. For example, one delivery
order was modified 19 times during its 30-month life, and the
contractor‘s performance was started anew five times, although work was
behind schedule each time.
Second, the CHCS II delivery orders contain timeliness, but not quality,
performance standards for each requirement. More specifically, each
order has a defined delivery date for each deliverable, but does not
specify standards governing the quality of each deliverable. Third,
CHCS II project management does not have a quality assurance plan
defining how it will apply standards to measure contractor performance
in meeting requirements. Fourth, the orders do not specify incentives
for either positive or negative contractor performance.
According to project officials, performance-based contracting practices
have not been employed because these practices have traditionally not
been used for IT services contracts, and they did not know how to
successfully do so for a system acquisition like CHCS II, where the
government is acquiring a product rather than a service. The officials
also stated that the multi-agency contract vehicle they chose to use
offered them flexibility and was immediately available, and they have
not viewed performance-based contracting as a priority.
By not using performance-based contracting practices on CHCS II, the
project office lacks an effective means for ensuring that it pays a
fair price for what it receives. To illustrate, three times in 2001 the
project office agreed to a schedule delay for release 2 deliverables
because the contractor was not able to meet the release 1 schedule. In
each instance, the release 2 work was also behind schedule.
Nevertheless, in both instances, release 2 delivery dates were changed
and the contractor‘s performance was shown as being on schedule.
Project officials stated that they recognize the value of performance-
based contracting. To prepare them for applying these best practices on
future CHCS II releases, they stated that they have recently awarded a
performance-based contract for help-desk services for a number of MHS
systems, including CHCS II, and that another MHS project is currently
negotiating a performance-based contract for software development. They
said that they intend to use these experiences to assist them in
negotiating a performance-based contract for CHCS II release 3.
Conclusions:
Owing largely to the absence of the kind of management and technical
controls that are hallmark qualities of system investment and
acquisition best practices, CHCS II‘s early years produced little more
than lessons learned. Since then, the project‘s management team has
recognized the need to change and given priority attention to doing so.
As a result, they introduced key missing best practices and made other
improvements to the project, some of which have occurred during the
course of our review. These needed practices and improvements have
contributed to where MHS stands today: in the latter stages of having
an initial version of CHCS II that shows signs of maturation and
operational readiness, although questions about operational efficiency
due to unresolved defects remain a concern.
A larger concern, however, are unanswered questions about CHCS II‘s
investment value. These questions exist because the project‘s management
and oversight teams, to include both the MHS and DOD CIOs, have not
given implementation of incremental investment management practices
adequate priority and attention. Consequently, MHS has continued to
invest in CHCS II without sufficient economic justification for doing
so. The prospects for this changing are encouraging, given statements by
project officials, but until defining and implementing processes for
incremental investment, the risk of DOD‘s spending more on the system
than it will return in measurable benefits is increased. Opportunities
also exist for increasing the effectiveness of ongoing risk management
activities by ensuring that the risk database is complete and correct,
thereby heading off problems before they occur. Further, opportunities
exist for adopting performance-based contracting best practices to
better ensure that DOD pays a fair price for contractor deliverables.
Greater use of best practices in the areas of investment, risk, and
contract management will better position CHCS II management to ensure
that it will not only be investing in the right vehicle but that it
will be investing the right way, meaning that it will be following the
kind of proven management practices that increase the probability that
required system capabilities and expected benefits will be delivered on
time and within budget.
Recommendations for Executive Action:
To strengthen CHCS II investment, risk, and contract management
practices, and thereby increase the chances of the department‘s
investment in CHCS II‘s producing mission value commensurate with
system costs, we recommend that the Secretary of Defense, through the
Assistant Secretary of Defense for Health Affairs, direct the MHS CIO to
give expanded use of best practices in managing CHCS II the attention
and priority they deserve. At a minimum, the Assistant Secretary should
direct the MHS CIO to take the following actions:
* As part of CHCS II deployment decisions, including any request to the
DOD CIO for deployment approval, consider the aggregate impact on
defense health affairs mission performance caused by the workarounds
needed to compensate for all unresolved defects affecting the system‘s
operational efficiency.
* Define and implement incremental investment management processes to
include (1) modifying the CHCS II investment strategy to define how this
approach will be implemented; (2) justifying investment in each system
release before beginning detailed design and development of the release;
(3) requiring that such justification be based on reliable estimates of
costs, benefits, and risks; (4) measuring whether actual return-on-
investment for each deployed release is in line with justification
forecasts; and (5) using actual return-on-investment results in
deciding whether to begin detailed design and development of the next
system release.
* Verify that the CHCS II inventory of risks is complete and correct,
and report this to the Assistant Secretary for Health Affairs every 6
months, along with a report on the status of all top priority risks,
including each risk‘s probability of occurrence and impact on mission.
* Employ performance-based contracting practices on all future CHCS II
delivery orders to the maximum extent possible, including (1) defining
performance standards against which deliverables can be judged, (2)
developing and using quality assurance plans that describes how
contractor performance against the standards will be measured, and (3)
defining and using contractor incentives and penalties tied to the
quality plan.
Additionally, we recommend that the Secretary of Defense direct the
Assistant Secretary of Defense for Command, Control, Communications,
and Intelligence, who is the designated approval authority for CHCS II,
to monitor the project‘s use of best practices, including
implementation of each of the above recommendations, and use this
information to oversee the project‘s movement through its acquisition
cycle. To this end, we also recommend that the Assistant Secretary, or
other designated CHCS II approval authority, not grant any request for
deployment approval of any CHCS II release that is not justified by
reliable analysis of the release‘s costs, benefits, and risks.
Agency Comments and Our Evaluation:
DOD provided what it termed ’official oral comments“ from the Assistant
Secretary of Defense for Health Affairs on a draft of this report. In
its comments, DOD stated that it agreed with our report‘s overall
message of expanding the department‘s use of acquisition management
best practices on CHCS II. Further, it agreed with two of our three
primary recommendations and associated findings, and it partially
agreed with the third, taking exception only to two of this
recommendation‘s component elements. Each of these two elements is
discussed below.
First, DOD disagreed with the recommendation calling for using actual
return-on-investment results from an implemented system release in
deciding whether to begin detailed design and development of the next
system release, citing the impact on the product delivery cycle (i.e.,
schedule). Instead, DOD stated that it would use ’post-deployment
findings and lessons learned for a given release to minimize risks
associated with future release development.“ Assuming that DOD‘s
reference to ’post-deployment findings“ includes having at least
preliminary information on cost and benefit accrual, we do not disagree
with DOD‘s proposed alternative and do not believe that it is
inconsistent with our recommendation. If it does not, we do not believe
that the proposed alternative adequately positions the department to
make informed investment decisions on future releases because it does
not provide for knowing, at least preliminarily, whether a prior
release‘s mission value is being realized. Given that producing mission
value is a paramount reason for investing in technology, such
information about prior system releases is essential in making prudent
decisions about further investment in the system.
Moreover, by not having at least preliminary information about actual
return-on-investment, DOD would not be able to accomplish its stated
goal of using ’post-deployment findings...to minimize risks associated
with future release development“ because it would not have any
indication about whether release return-on-investment expectations are
accruing. On the contrary, DOD‘s approach would expose the program to
unnecessary risk in the name of meeting a scheduled milestone.
Restated, it would increase the chances of doing the wrong thing
faster. The intent of our recommendation is to prevent this by ensuring
that DOD has an adequate understanding of returns from prior
investments before choosing a course of action on subsequent
investments. Accordingly, we have not modified our recommendation.
Second, DOD disagreed with the recommendation element calling for
reporting on CHCS II program risks to the Assistant Secretary for
Health Affairs every 6 months, instead stating that the timing of its
reporting would comply with DOD internal milestone review requirements,
which based on program plans would result in annual reporting. We do not
believe that DOD‘s proposed alternative provides for adequate disclosure
of those items that have a high probability of significantly affecting
delivery of promised system capabilities on time and within budget to
the executive leadership sponsoring CHCS II. As our research of leading
organization IT investment management practices shows, awareness of
program risks, particularly the top priority risks covered by this
recommendation, are fundamental to ongoing investment control activities
and, among other things, should be frequently disclosed to investment
executives. Further, over extended periods of time, such as 1 year,
these risks can manifest themselves into material program cost,
schedule, and performance shortfalls. For these reasons, 6-month
reporting on these risks should be viewed as the minimum acceptable
frequency. Thus, we have not modified our recommendation.
DOD also provided a comment on our recommendation for the Assistant
Secretary of Defense for Command, Control, Communications, and
Intelligence to only approve deployment of a CHCS II release if it is
justified by reliable analysis of the release‘s cost, benefits, and
risks. Specifically, DOD stated that while the Assistant Secretary was
currently the approval authority for CHCS II deployment and related
milestone decisions, this decision authority could be delegated in the
future. We have adjusted our recommendation to recognize this
possibility.
We are sending copies of this report to the Chairman and Ranking
Minority Members of the Senate Committee on Armed Services; House
Committee on Armed Services; Subcommittee on Defense, Senate Committee
on Appropriations; the Subcommittee on Defense, House Committee on
Appropriations; the Subcommittee on Military Readiness, House Committee
on Armed Services; Senate Committee on Government Affairs; and House
Committee on Government Reform. We are also sending copies to the
Secretary of Defense and the Director, Office of Management and Budget.
Copies will be available upon request and will also be available
without charge at our Web site at [hyperlink, http://www.gao.gov].
Should you have any questions on matters discussed in this report,
please contact me at (202) 512-3439 or by e-mail at HiteR@gao.gov. Key
contributors to this report are listed in appendix VI.
Signed by:
Randolph C. Hite:
Director, Information Technology Architecture and Systems Issues:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
The objectives of our review were to determine (1) what progress the
Department of Defense (DOD) has made against the Composite Health
Care System (CHCS) II project commitments, (2) whether DOD has
economically justified its investment in CHCS II, and (3) whether DOD
has effective technical and management controls in place for acquiring
the system. Project commitments included in our scope of work were
required system capabilities, promised system benefits, and estimated
project costs and milestones; controls included requirements
management, test management, architecture development and alignment,
risk management, and contract management.
To determine the progress made against commitments, we reviewed
relevant project management documents, such as acquisition strategy and
program baseline documents, acquisition decision memoranda, and
quarterly Defense Acquisition Executive Summary reports, and we
interviewed CHCS II project, military health system (MHS) program, and
DOD oversight officials, to determine original and revised system
capabilities, expected benefits, estimated costs, and project
milestones for both system increments and the entire system. We then
reviewed program management reports and briefings and system
documentation (e.g., cost reports, defect reports, and test results),
and interviewed project, program, and oversight officials, to determine
actual (as reported) system capabilities, costs, and schedules, as well
as accrued benefits. Comparing the two, we identified variances and,
through document review and interviews, identified the causes of the
variances. We did not independently validate the status information
obtained.
To determine whether DOD had economically justified CHCS II, we
reviewed best practices governing system investment management,
including those pertaining to incremental investment practices, and we
reviewed relevant legislative requirements and federal guidance.
[Footnote 37] We then analyzed the available economic justification for
the project, an April 1998 economic analysis. [Footnote 38] We compared
this analysis to the best practices and federal guidance, and discussed
with project, program, and oversight officials, as well as CHCS II
contractor officials to determine how the analysis was prepared,
including the basis for cost and benefit estimates and assumptions. We
also requested updates of this analysis, and reviewed draft updates
provided to determine whether these updates were approved. Using data
from approved and updated draft analyses, we also calculated net
present values for CHCS II. In addition, we reviewed available
documentation and interviewed CHCS II project officials on its
portfolio-based investment analysis process and how this process was
applied to CHCS II.
To determine whether DOD has effective technical and management
controls in place for acquiring the system, we focused on whether best
practices were being employed in five key areas: requirements
management, test management, architecture development and alignment,
risk management, and contract management. We reviewed these areas
because they are critical to successfully acquiring systems. Among other
sources, these practices were derived from the work and research of
Carnegie Mellon University‘s Software Engineering Institute (SEI),
[Footnote 39] as well as our own research. [Footnote 40] Where
appropriate, we also applied relevant legislative requirements and
federal guidance that embody these best practices. [Footnote 41] We
also interviewed MHS officials, CHCS II project officials, and
contractor staff from Integic, Incorporated, the CHCS II integration
contractor. More detailed description of our scope and methodology in
each control area is provided below.
Requirements Management: To evaluate requirements management controls,
we reviewed the current CHCS II requirements management process and
compared it to best practices. We then evaluated the project‘s
compliance with its established process by analyzing selected sets of
requirements that were managed under the current process. Since the
current process was not in place until 2000, we judgmentally selected
one of the three requirements sets that were added in 2000, and one of
the five that were added in 2001. Requirements documentation that we
reviewed also included requirements descriptions, cost and impact
estimates, and the related 2000 and 2001 budget analysis documents. We
also interviewed project and MHS officials responsible for requirements
management.
Test management: To evaluate test management controls, we reviewed the
CHCS II master test plan to determine the overall test approach and
scope, including the types of testing performed and planned. Because
system acceptance testing was being planned, conducted, and completed
during the course of our review, we focused on this test event.
Specifically, we analyzed acceptance test planning documents, including
test procedures, and the actual test results, and we compared the
practices defined and followed to relevant best practices to identify
whether any variances existed. We also analyzed the scope of the test
by selecting a statistically valid sample of 59 requirements and
analyzing test procedures to determine if each requirement was
addressed. This sample size provided us with a 95 percent confidence
level. Additionally, we analyzed test results to determine whether
defined system maturity parameters were met, and if not, analyzed the
variance to determine the significance of the variance.
Architecture development and alignment: To evaluate these controls, we
first focused on the MHS enterprise architecture, determining what the
architecture framework is based on and whether the framework used is
consistent with recognized commercial frameworks. We then compared
the MHS enterprise architecture to the framework to determine if
essential architectural artifacts had been developed, and where
applicable, whether the MHS architecture was consistent with DOD-wide
architecture requirements, such as whether the technical standards
profile in the MHS architecture was consistent with the DOD Joint
Technical Architecture. [Footnote 42] Next, we obtained relevant CHCS
II architectural documents, such as requirements documents and design
specifications, and traced selected CHCS II architecture
characteristics to the MHS architecture to determine alignment. At the
business or logical level of the architecture, we compared 13 release 1
system capabilities (categories of requirements) to the business areas
and activities in the MHS architecture. We selected 13 of the 52
release 1 capabilities (25 percent) to ensure that we included
capabilities covering all MHS business areas relevant to CHCS II. We did
not plan to select more than the 13 capabilities unless we were unable
to trace any of the 13. At the technical level, we compared all of the
CHCS II technical standards to the MHS technical standards profile. In
conducting our analysis, we also interviewed MHS and project officials
and reviewed architecture development plans to determine how the
respective architectures were developed, who developed them, and how
they ensured alignment between the two.
Risk management: To evaluate risk management controls, we reviewed
the CHCS II risk management process and compared it with best practices.
To determine whether the process was being followed, we analyzed all
priority 1 and 2 risks for release 1 and release 2, including
determining whether defined steps were performed and criteria for
reporting on and closing risks were met. Additionally, we used the
results of review of CHCS II to determine whether additional risks
should be added to the existing risk inventory. In conducting our
analysis, we interviewed project officials and reviewed relevant
documentation, such as the risk management plan, risk mitigation
strategies, and risk status reports.
Contract management: In evaluating contract management controls, we
discussed with project officials their criteria and approach for
defining contract deliverables and measuring contractor performance,
and whether they had plans for changing from past practices. We also
reviewed the 11 1999 to 2001 release 1 contract delivery orders with
the CHCS II integration contractor and compared them with tenets of
performance-based contracting, which are defined in federal acquisition
regulations. [Footnote 43] Additionally, we reviewed internal reports,
such as earned value management system summary reports, which described
progress against delivery order schedules and budgets to obtain data on
contractor performance.
We conducted our work at offices of the Military Health System in Falls
Church, Virginia, and at Integic, Incorporated, offices in Chantilly,
Virginia, from August 2001 through July 2002, in accordance with
generally accepted government auditing standards.
[End of section]
Appendix II: Comparison of CHCS and CHCS II Capabilities:
Capability: Access to Care: Bed Availability Reporting;
Provided by CHCS: Yes;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Access to Care: Case Management;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Access to Care: Enrollment/Eligibility;
Provided by CHCS: Yes;
Provided by CHCS II release 1: Yes;
Provided by CHCS II when all releases complete: Yes.
Capability: Access to Care: Enterprise Member/Patient Index;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Access to Care: Enterprise Registration;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Access to Care: Enterprise Scheduling;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Access to Care: Evacuation Requests;
Provided by CHCS: Yes;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Access to Care: Global Clinical Data Repository;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Access to Care: Referral Management;
Provided by CHCS: Yes;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Access to Care: Triage and Demand Management;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Alerts and Reminders
(including Allergies and Drug Interactions);
Provided by CHCS: No;
Provided by CHCS II release 1: Yes;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Centralized Health Record
Repository;
Provided by CHCS: No;
Provided by CHCS II release 1: Yes;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Clinical Decision Support;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Clinical Documentation;
Provided by CHCS: No;
Provided by CHCS II release 1: Yes;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Dental Charting and
Documentation;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Discharge Planning;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Encounter Coding;
Provided by CHCS: No;
Provided by CHCS II release 1: Yes;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Enterprise Health Record;
Provided by CHCS: No;
Provided by CHCS II release 1: Yes;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Home-based Monitoring;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Inpatient Order
Entry/Management (Laboratory, Pharmacy, Radiology);
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Operating Room Management;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Optometric
Documentation/Order Entry;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Outpatient Order
Entry/Management (Laboratory, Pharmacy, Radiology);
Provided by CHCS: Yes;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Patient Education;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Patient Health History;
Provided by CHCS: No;
Provided by CHCS II release 1: Yes;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Pharmaceutical Profiling;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Results Reporting (Ancillary
Services);
Provided by CHCS: Yes;
Provided by CHCS II release 1: Yes;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Role-Based Security;
Provided by CHCS: No;
Provided by CHCS II release 1: Yes;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Telemedicine;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Theater Integration Related
to Provision of Health Service;
Provided by CHCS: No;
Provided by CHCS II release 1: Yes;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Transcription Services
Interface;
Provided by CHCS: No;
Provided by CHCS II release 1: Yes;
Provided by CHCS II when all releases complete: Yes.
Capability: Provisions of Health Service: Voice Recognition
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Population Health Management: Clinical Enterprise
Member/Patient Index;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Population Health Management: Clinical Look-back;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Population Health Management: Clinical Outcomes Reporting;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Population Health Management: Clinical Registries;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Population Health Management: Demand Material;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Population Health Management: Health Plan Employer Data and
Information Set Reporting;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Population Health Management: Health Risk Assessment;
Provided by CHCS: No;
Provided by CHCS II release 1: Yes;
Provided by CHCS II when all releases complete: Yes.
Capability: Population Health Management: Health Surveillance
Monitoring/Reporting;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Population Health Management: Immunization Tracking;
Provided by CHCS: No;
Provided by CHCS II release 1: Yes;
Provided by CHCS II when all releases complete: Yes.
Capability: Population Health Management: Individual Medical Readiness
Status Reporting;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Population Health Management: Interface to VA Mail Order
Pharmacy;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Population Health Management: Occupational Health
Monitoring;
Provided by CHCS: Yes;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Population Health Management: Patient Satisfaction
Reporting;
Provided by CHCS: No;
Provided by CHCS II release 1: Yes;
Provided by CHCS II when all releases complete: Yes.
Capability: Population Health Management: Patient Self-assessment Data
Entry;
Provided by CHCS: No;
Provided by CHCS II release 1: Yes;
Provided by CHCS II when all releases complete: Yes.
Capability: Population Health Management: Population Utilization
Management;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Population Health Management: Provider Profiling;
Provided by CHCS: No;
Provided by CHCS II release 1: Yes;
Provided by CHCS II when all releases complete: Yes.
Capability: Population Health Management: Radiation Health Monitoring
and Reporting;
Provided by CHCS: Yes;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Population Health Management: Rules-based Clinical
Protocols;
Provided by CHCS: No;
Provided by CHCS II release 1: No;
Provided by CHCS II when all releases complete: Yes.
Capability: Population Health Management: Theater Integration Related
to Population Health Material;
Provided by CHCS: No;
Provided by CHCS II release 1: Yes;
Provided by CHCS II when all releases complete: Yes.
Source: GAO, based of DOD data.
[End of table]
[End of section]
Appendix III: Detailed Explanation of How CHCS II Will Operate:
The initial version (release 1) of CHCS II is a medical information
system that combines existing MHS systems with commercial software to
provide a computer-based patient record (CPR) of treatment provided to
DOD health-care beneficiaries in DOD medical treatment facilities
(MTF). CHCS II is intended to allow a provider (physician, nurse,
technician, etc.) to record information gained from a patient‘s visit
to the MTF in the patient‘s CPR, as shown in figure 8.
Figure 8: Creating a Computer-based Patient Record:
[See PDF for image]
This figure is an illustration of the creation of a computer-based
patient record. The illustration depicts the following information:
Patient arrives:
* Diagnostics (blood pressure, heart rate, temperature, etc.);
* Triage (nurse asks about reason for visit, history of illness);
* Physical exam (doctor sees patient).
CHCS II workstation:
Date from Diagnostics, Triage, and Physical exam is entered, producing
a computer-based patient record, which is stored in a server.
Source: GAO, based on DOD data.
[End of figure]
CHCS II‘s architecture is an open system, client-server design of three
levels: the user (client) workstation at various MTF locations, the MTF
computers‘ (servers‘) operating system and storage hardware and
software, and a clinical data repository at a remote computing center
where the CPR is stored. DOD plans to connect all workstations at an
installation‘s hospital or clinics to the servers through the
installation‘s local or wide area network. DOD also plans for each
installation to be connected, via the Defense Information Systems
Agency‘s (DISA) unclassified wide area network, [Footnote 44] to the
Montgomery, Alabama, computing center, where the CPRs will be stored in
a database known as the clinical data repository (CDR). Figure 9 shows
how the three CHCS II levels are connected.
Figure 9: CHCS II Tri-level Architecture:
[See PDF for image]
This figure is an illustration of the CHCS II Tri-level Architecture,
depicting the following layout:
CDR level:
* CDR; connects to:
* CDR server.
Server level:
* Security server; connects to: DSIA wide area network;
* Application server; connects to: DSIA wide area network;
* DSIA wide area network: connects to CDR server at CDR level.
Client level:
* CHCS II workstations: connect to MTF local/wide area network;
* MTF local/wide area network: connects to DSIA wide area network at
server level.
Source: GAO, based on DOD data.
[End of figure]
DOD plans to deploy release 1 regionally, starting with all sites in MHS
region 2 (Virginia and North Carolina). Communications between the MTFs
and the CDR go through a single access point in each region (the
regional hub). For region 2, the access point is planned for Portsmouth,
Virginia, site of the Portsmouth Naval Hospital. The Portsmouth regional
access point plan shows a primary and secondary (for redundancy and
diversity) connection to the CDR through two separate DISA computing
centers. The communication plan shows each installation in region 2 with
an installation access point connected to Portsmouth, and a backup
connection to the CDR through a DISA computing center. Figure 10 shows
a typical communications setup for release 1, in this case for region
2. The figure uses the Ft. Bragg, North Carolina, MTF as an example of
the nonregional hub.
Figure 10: Example of CHCS II Regional Communications:
[See PDF for image]
This figure is an illustration of the CHCS II Regional Communications
setup. The following layout is depicted:
CDR: connects to CDR server at DISA computer center, Montgomery,
Alabama;
CDR server: connects to:
* DISA computer center, Columbus, Georgia; and;
* DISA computer center, Atlanta, Georgia;
* Both centers connect to: Regional access point, Portsmouth, Virginia;
* DISA, Atlanta also has a backup connection to: Ft. Bragg, North
Carolina;
* A primary connection exists between Portsmouth and Ft. Bragg;
Regional access point, Portsmouth, Virginia: contains the Portsmouth,
VA MTF, local/wide area network.
Access point, Ft. Bragg: contains Fort Bragg, NC MTF local/wide area
network.
Source: GAO, based on DOD data.
[End of figure]
[End of section]
Appendix IV: CHCS II Release 1 Software Packages:
Release 1 is composed of a number of commercial-off-the-shelf (COTS)
[Footnote 45] and government-off-the-shelf (GOTS) software packages, as
shown in table 7. The core CHCS II functionality is provided by the
existing CHCS application (nine GOTS packages), plus the two 3M
software packages, MEDCIN charting software, data dictionary, and
clinical data repository (five COTS packages). The packages are
integrated by software developed by the CHCS II contractor. In addition
to the core functionality, CHCS II is also composed of 16 other COTS
packages that provide operating system, security, data exchange, and
other system functions. The CHCS II software is located on hardware
platforms at the 142 military treatment facilities (user workstations
and application, security, interface, and CHCS servers) and the DISA
computing center (clinical data repository and server).
Table 6: CHCS II Release 1 Software Packages by Location:
Location: Client Workstation;
Product: MEDCIN;
Use: Charting;
Type: Commercial.
Location: Client Workstation;
Product: Snareworks;
Use: Security;
Type: Commercial.
Location: Client Workstation;
Product: Adobe Acrobat Reader;
Use: Form viewer;
Type: Freeware.
Location: Client Workstation;
Product: Dimension 4;
Use: Time service;
Type: Freeware.
Location: Client Workstation;
Product: Windows 2000;
Use: Operating System;
Type: Commercial.
Location: Client Workstation;
Product: Problem Knowledge Couplers;
Use: Education;
Type: Commercial.
Location: Client Workstation;
Product: McAfee Netshield;
Use: Virus protection;
Type: Commercial.
Location: Client Workstation;
Product: PARS II Data Collector;
Use: Data collection;
Type: Government.
Location: Administrator Workstation;
Product: Problem Knowledge Couplers;
Use: Education;
Type: Commercial.
Location: Administrator Workstation;
Product: Adobe Acrobat Reader;
Use: Form viewer;
Type: Freeware.
Location: Administrator Workstation;
Product: McAfee Netshield;
Use: Virus protection;
Type: Commercial.
Location: Administrator Workstation;
Product: 3M Clinical Workstation;
Use: User Interface;
Type: Commercial.
Location: Clinical Data Repository/Enterprise Security Server;
Product: 3M Software;
Use: Data dictionary, repository;
Type: Commercial.
Location: Clinical Data Repository/Enterprise Security Server;
Product: SNOMED;
Use: Data dictionary;
Type: Commercial.
Location: Clinical Data Repository/Enterprise Security Server;
Product: Netscape Unix;
Use: User registration;
Type: Commercial.
Location: Clinical Data Repository/Enterprise Security Server;
Product: HP Distributed computing;
Use: Security;
Type: Commercial.
Location: Clinical Data Repository/Enterprise Security Server;
Product: Snareworks;
Use: Security;
Type: Commercial.
Location: Clinical Data Repository/Enterprise Security Server;
Product: Oracle;
Use: Database;
Type: Commercial.
Location: Clinical Data Repository/Enterprise Security Server;
Product: Tuxedo;
Use: On-line transaction monitor;
Type: Commercial.
Location: Military Treatment Facility Security Server;
Product: Enosis Connection Manager;
Use: Software management;
Type: Commercial.
Location: Military Treatment Facility Security Server;
Product: Tardis;
Use: Time service;
Type: Shareware.
Location: Military Treatment Facility Security Server;
Product: Windows NT 4;
Use: Operating system;
Type: Commercial.
Location: Military Treatment Facility Security Server;
Product: Gradient;
Use: Middleware;
Type: Commercial.
Location: Military Treatment Facility Security Server;
Product: Snareworks;
Use: Security;
Type: Commercial.
Location: Military Treatment Facility Application Server;
Product: Active X;
Use: Operating system component;
Type: Government.
Location: Military Treatment Facility Application Server;
Product: Enosis Connection Manager;
Use: Data exchange;
Type: Commercial.
Location: Military Treatment Facility Application Server;
Product: Snareworks;
Use: Security;
Type: Commercial.
Location: Military Treatment Facility Application Server;
Product: Tardis;
Use: Time service;
Type: Shareware.
Location: Military Treatment Facility Application Server;
Product: Oracle;
Use: Database;
Type: Commercial.
Location: Military Treatment Facility Application Server;
Product: Business Objects;
Use: Reports;
Type: Government.
Location: Interface Engine Server;
Product: Datagate;
Use: Operating system; Interface: 3M to CHCS II;
Type: Commercial.
Location: CHCS Server;
Product: CHCS;
Use: CHCS application;
Type: Government.
Location: CHCS Server;
Product: Fileman;
Use: CHCS application;
Type: Government.
Location: CHCS Server;
Product: GIS;
Use: Interface software;
Type: Government.
Location: CHCS Server;
Product: HL 7;
Use: Message software;
Type: Government.
Location: CHCS Server;
Product: OV MVS;
Use: Operating system;
Type: Government.
Location: CHCS Server;
Product: MUMPS;
Use: Programming language;
Type: Government.
Location: CHCS Server;
Product: M Adaptor;
Use: Data exchange;
Type: Government.
Location: CHCS Server;
Product: M Functions;
Use: Data exchange;
Type: Government.
Source: GAO, based on DOD data.
[End of table]
[End of section]
Appendix V: CHCS II Capabilities Planned for Releases 3-7, and Expected
Release Dates:
Release number and expected release date: 3, July 2004;
Capabilities:
* Enterprise member/patient index;
* Government computerized patient record (DOD–Department of Veterans
Affairs clinical information exchange);
* Health risk assessment;
* Health surveillance monitoring and reporting;
* Individual medical readiness status reporting;
* New eligibility system interface;
* Patient self-assessment data entry (additional functionality);
* Referral management;
* Replacement ancillary system–pharmacy;
* Rules-based clinical protocols;
* Theater integration.
Release number and expected release date: 4, July 2004;
* Capabilities:
* Bed availability reports;
* Cost accounting and patient accounting interfaces;
* Dental imaging;
* Enterprisewide registration;
* Enterprisewide scheduling (includes operating room);
* Evacuation requests;
* Operating room management;
* Outpatient order entry and management;
* Pharmaceutical profiling;
* Replacement laboratory, pathology, radiology systems;
* Theater integration;
* Triage and demand management.
Release number and expected release date: 5, July 2005;
Capabilities:
* Assurance/safety;
* Case management;
* Inpatient order entry and management;
* Patient education;
* Patient safety reporting;
* Population utilization management and quality;
* Theater integration;
* Voice recognition.
Release number and expected release date: 6, July 2005;
Capabilities:
* Clinical look-back;
* Clinical outcomes reporting;
* Discharge planning;
* Enterprise health record;
* Health plan employer data reporting;
* Provider database interface;
* Theater integration.
Release number and expected release date: 7, July 2006;
Capabilities:
* Home-based monitoring;
* Nonradiation and nonlaboratory ancillary procedures interface;
* Occupational health monitoring;
* Radiation health monitoring and reporting;
* Telemedicine;
* Theater integration.
Source: GAO, based on DOD data.
[End of table]
[End of section]
Appendix VI: GAO Contact and Staff Acknowledgments:
GAO Contact:
Carl L. Higginbotham, (404) 679-1824:
Acknowledgments:
In addition to the individual named above, key contributors to this
report included Nabajyoti Barkakati, Harold J. Brumm Jr., Katherine I.
Chu-Hickman, Michael P. Fruitman, Chetna Lal, and Keith E. Steck.
[End of section]
Footnotes:
[1] U.S. General Accounting Office, DOD Systems Modernization:
Continued Investment in the Standard Procurement System Has Not Been
Justified, GAO-01-682 (Washington, D.C.: July 31, 2001); U.S. General
Accounting Office, Information Technology: DLA Should Strengthen
Business Systems Modernization Architecture and Investment Activities,
GAO-01-631 (Washington, D.C.: June 29, 2001).
[2] DOD plans to divide the CHCS II acquisition into seven releases.
Release 1 was scheduled for a deployment decision in September 2002;
release 2 in July 2003; the remaining deployments following at about 1-
year intervals.
[3] Examples include functional and performance requirements.
[4] An enterprise architecture is the operational and technical
blueprint for DOD-wide military health affairs.
[5] Each of these systems provides certain patient-related information.
For example, the Ambulatory Data System captures certain outpatient
information relating to diagnosis and treatment; the Preventive Health
Care Application contains information on preventive health services;
and the Nutrition Management Information System supports therapeutic
nutrition therapy and medical food management.
[6] Open systems conform to industry standards so that commercial
products can easily be used and support costs can be minimized. A
client is usually a desktop computing device or program that is
’served“ by one or more networked computing devices.
[7] CHCS II users are physicians, nurses, other medical personnel, and
technical or administrative support staff.
[8] The network is DOD‘s Unclassified but Sensitive Internet Protocol
Router Network.
[9] The new eligibility system is intended to replace the existing
Defense Enrollment Eligibility Reporting System and improve MHS sharing
of health care information and eliminate manual benefits
determinations. The new Primary Care Manager System is intended to
allow assignment and change of beneficiaries‘s primary care managers.
[10] Clinger-Cohen Act of 1996, Public Law 104-106 and Office of
Management and Budget (OMB) Circular A-130, Management of Federal
Information Resources (Nov. 30, 2000).
[11] A program used to connect the user to the system. The prototype
version was based on a program used to connect to the World Wide Web.
[12] Priority 1 defects prevent the accomplishment of an operational-
or mission-essential capability or jeopardize personnel safety.
Priority 2 defects adversely affect the accomplishment of a mission-
essential capability, degrading performance, for which no alternative
workaround is known. Priority 3 defects are similar to priority 2, but
workarounds are available. Priority 4 and 5 defects are less serious.
[13] Operational testing is used to independently assess effectiveness
and suitability for end users.
[14] Alerts, allergies, appointments/scheduling, assessment plan,
clinical notes, consult tracking, encounter coding, encounter
documentation, external interfaces, forms/reports, immunizations,
medications, order entry, order sets, patient demographics, patient
disposition, patient search, patient self-assessment tools, problem
list, results retrieval, screening, security, summary of care,
telephone consults, template management, templates, user preferences,
and wellness.
[15] Configuration management, data recovery and restoration, fault
closure rate, fault severity, functional baseline, functional
completeness, installation, interchangeability, key performance
parameters, network design, reliability, software, system availability,
test environment, and safety.
[16] See, for example, Weather Forecasting: Radar Ability Requirement
Not Being Met, GAO/AIMD-95-132 (Washington, D.C.: May 31, 1995) and Air
Traffic Control: FAA Plans to Replace Its Host Computer System Because
Future Availability Cannot Be Assured, GAO/AIMD-98-138R (Washington,
D.C.: May 1, 1998).
[17] Clinger-Cohen Act of 1996, Public Law 104-106; OMB Circular A-130
(Nov. 30, 2000).
[18] DOD Interim Regulation 5000.2-R and DOD Instruction 5000.2, Change
1, Operation of the Defense Acquisition System (Jan. 4, 2001).
[19] Office of the Inspector General (OIG), Department of Defense,
Acquisition Management of the Composite Health Care System II Automated
Information System, report number 99-068 (Jan. 21, 1999).
[20] The cost estimate appropriately does not include the about $320
million already spent on CHCS II because this constitutes a sunk cost
and thus is not relevant to determining current net present value.
[21] The life-cycle benefits estimate includes benefits resulting from
improvements in MHS processes ($4.4 billion) and benefits from
replacing existing MHS systems with CHCS II ($2.1 billion).
[22] See, for example, Software Acquisition Capability Maturity Model
SM (CMU/SEI-99-TR-002, April 1999).
[23] Office of the Inspector General (OIG), Department of Defense,
report number 99-068 (Jan. 21, 1999).
[24] See, for example, Year 2000 Computing Crisis: A Testing Guide
(GAO/AIMD-10.1.21, November 1998).
[25] See appendix I for statistical sampling methodology.
[26] Enterprises can be single organizations or business/mission areas
that transcend more than one organization (e.g., financial management,
combat system identification, or medical health care).
[27] Clinger-Cohen Act of 1996, P.L. 104-106; OMB Circular A-130 (Nov.
30, 2000); A Practical Guide to Federal Enterprise Architectures,
Version 1.0, Chief Information Officers Council (February 2001);
Federal Enterprise Architecture Framework, Version 1.1, Chief
Information Officers Council (September 1999).
[28] February 28, 1998, memorandum – jointly signed by the Under
Secretary of Defense for Acquisition and Technology; the Acting
Assistant Secretary of Defense for Command, Control, Communications,
and Intelligence, ASD(C3I); and the Director for Command, Control,
Communications, and Computer Systems, Joint Chiefs of Staff.
[29] For additional information on enterprise architectures, see
Information Technology: Enterprise Architecture Use across the Federal
Government Can Be Improved, GAO-02-6 (Feb. 19, 2002).
[30] DOD has recently initiated a program to develop and implement a
DOD-wide financial management enterprise architecture, which is to
include each DOD business area that triggers a financial event. MHS is
participating in this DOD-wide architecture effort, including providing
MHS architecture products and staff.
[31] DOD‘s architecture framework is called the Command, Control,
Communications, Computers, Intelligence, Surveillance, and
Reconnaissance Architecture Framework.
[32] GAO-01-631.
[33] Department of Defense, Joint Technical Architecture, version 3.1,
March 31, 2000.
[34] See, for example, Software Acquisition Capability Maturity Model
SM (CMU/SEI-99-TR-002, April 1999); OMB Circular A-130 (Nov. 30, 2000).
[35] In MHS‘s risk management plan, priority 1 risks require immediate
project changes to eliminate or reduce the risk and management
attention; priority 2 risks require some project changes, mitigation
strategies, and careful monitoring; priority 3 risks require mitigation
strategies and careful monitoring; and priority 4 and 5 risks do not
require mitigation activities.
[36] Federal Acquisition Regulation, Part 37.
[37] Clinger-Cohen Act of 1996, Public Law 104-106, and OMB Circular A-
130 (Nov. 30, 2000).
[38] CHCS II Milestone I Economic Analysis, April 15, 1998.
[39] See, for example, Software Acquisition Capability Maturity Model
SM (CMU/SEI-99-TR-002, April 1999).
[40] See, for example, Year 2000 Computing Crisis: A Testing Guide
(GAO/AIMD-10.1.21, November 1998); Information Technology Investment
Management: A Framework for Assessing and Improving Process Maturity,
Exposure Draft, GAO/AIMD-10-1.23, version 1 (Washington, D.C.: May
2000).
[41] Clinger-Cohen Act of 1996, Public Law 104-106; OMB Circular A-130,
(Nov. 30, 2000); A Practical Guide to Federal Enterprise Architectures,
Version 1.0, Chief Information Officers Council (October 2000).
[42] Department of Defense, Joint Technical Architecture, version 3.1,
March 31, 2000.
[43] Federal Acquisition Regulation, Part 37.
[44] The network is the Unclassified but Sensitive Internet Protocol
Router Network.
[End of section]
GAO‘s Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO‘s commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO‘s Web site [hyperlink,
http://www.gao.gov] contains abstracts and fulltext files of current
reports and testimony and an expanding archive of older products. The
Web site features a search engine to help you locate documents using
key words and phrases. You can print these documents in their entirety,
including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ’Today‘s Reports,“ on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
[hyperlink, http://www.gao.gov] and select ’Subscribe to daily E-mail
alert for newly released products“ under the GAO Reports heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov:
(202) 512-4800:
U.S. General Accounting Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
