Information Technology
Opportunities Exist to Improve Management of DOD's Electronic Health Record Initiative Gao ID: GAO-11-50 October 6, 2010The Department of Defense (DOD) provides medical care to 9.6 million active duty service members, their families, and other eligible beneficiaries worldwide. DOD's Military Health System has long been engaged in efforts to acquire and deploy an electronic health record system. The latest version of this initiative--the Armed Forces Health Longitudinal Technology Application (AHLTA)--was expected to give health care providers real-time access to individual and military population health information and facilitate clinical support. However, the system's early performance was problematic, and DOD recently stated that it intended to acquire a new electronic health record system. GAO was asked to (1) determine the status of AHLTA, (2) determine DOD's plans for acquiring its new system, and (3) evaluate DOD's acquisition management of the initiative. To do this, GAO reviewed program plans, reports, and other documentation and interviewed DOD officials.
After obligating approximately $2 billion over the 13-year life of its initiative to acquire an electronic health record system, as of September 2010, DOD had delivered various capabilities for outpatient care and dental care documentation. DOD had scaled back other capabilities it had originally planned to deliver, such as replacement of legacy systems and inpatient care management. In addition, users continued to experience significant problems with the performance (speed, usability, and availability) of the portions of the system that have been deployed. DOD has initiated efforts to improve system performance and enhance functionality and plans to continue its efforts to stabilize the AHLTA system through 2015, as a "bridge" to the new electronic health record system it intends to acquire. According to DOD, the planned new electronic health record system--known as the EHR Way Ahead--is to be a comprehensive, real-time health record for service members and their families and beneficiaries. The system is expected to address performance problems, provide unaddressed capabilities such as comprehensive medical documentation, capture and share medical data electronically within DOD, and improve existing information sharing with the Department of Veterans Affairs. As of September 2010, the department had established a planning office, and this office had begun an analysis of alternatives for meeting the new system requirements. Completion of this analysis is currently scheduled for December 2010. Following its completion, DOD expects to select a technical solution for the system and release a delivery schedule. DOD's fiscal year 2011 budget request included $302 million for the EHR Way Ahead initiative. Weaknesses in key acquisition management and planning processes contributed to AHLTA having fewer capabilities than originally expected, experiencing persistent performance problems, and not fully meeting the needs of users. (1) A comprehensive project management plan was not established to guide the department's execution of the system acquisition. (2) A tailored systems engineering plan did not exist to guide the technical development of the system, an effort that was characterized by significant complexity. (3) Requirements were incomplete and did not sufficiently reflect user and operational needs. (4) An effective plan was not used to improve users' satisfaction with the system. DOD has initiated efforts to bring its processes into alignment with industry best practices. However, it has not carried out a planned independent evaluation to ensure it has made these improvements. Until it ensures that these weaknesses are addressed, DOD risks undermining the success of further efforts to acquire electronic health record system capabilities. GAO is recommending that DOD take six actions to help ensure that it has disciplined and effective processes in place to manage the acquisition of further electronic health record system capabilities. In written comments on a draft of this report, DOD concurred with GAO's recommendations and described actions planned to address them.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Valerie C. Melvin Team: Government Accountability Office: Information Technology Phone: (202) 512-6304GAO-11-50, Information Technology: Opportunities Exist to Improve Management of DOD's Electronic Health Record Initiative
This is the accessible text file for GAO report number GAO-11-50
entitled 'Information Technology: Opportunities Exist to Improve
Management of DOD's Electronic Health Record Initiative' which was
released on October 6, 2010.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Ranking Member, Committee on the Budget, U.S. Senate:
United States Government Accountability Office:
GAO:
October 2010:
Information Technology:
Opportunities Exist to Improve Management of DOD's Electronic Health
Record Initiative:
GAO-11-50:
GAO Highlights:
Highlights of GAO-11-50, a report to the Ranking Member, Committee on
the Budget, U.S. Senate.
Why GAO Did This Study:
The Department of Defense (DOD) provides medical care to 9.6 million
active duty service members, their families, and other eligible
beneficiaries worldwide. DOD‘s Military Health System has long been
engaged in efforts to acquire and deploy an electronic health record
system. The latest version of this initiative-”the Armed Forces Health
Longitudinal Technology Application (AHLTA)-”was expected to give
health care providers real-time access to individual and military
population health information and facilitate clinical support.
However, the system‘s early performance was problematic, and DOD
recently stated that it intended to acquire a new electronic health
record system. GAO was asked to (1) determine the status of AHLTA, (2)
determine DOD‘s plans for acquiring its new system, and (3) evaluate
DOD‘s acquisition management of the initiative. To do this, GAO
reviewed program plans, reports, and other documentation and
interviewed DOD officials.
What GAO Found:
After obligating approximately $2 billion over the 13-year life of its
initiative to acquire an electronic health record system, as of
September 2010, DOD had delivered various capabilities for outpatient
care and dental care documentation. DOD had scaled back other
capabilities it had originally planned to deliver, such as replacement
of legacy systems and inpatient care management. In addition, users
continued to experience significant problems with the performance
(speed, usability, and availability) of the portions of the system
that have been deployed. DOD has initiated efforts to improve system
performance and enhance functionality and plans to continue its
efforts to stabilize the AHLTA system through 2015, as a ’bridge“ to
the new electronic health record system it intends to acquire.
According to DOD, the planned new electronic health record system”-
known as the EHR Way Ahead”-is to be a comprehensive, real-time health
record for service members and their families and beneficiaries. The
system is expected to address performance problems, provide
unaddressed capabilities such as comprehensive medical documentation,
capture and share medical data electronically within DOD, and improve
existing information sharing with the Department of Veterans Affairs.
As of September 2010, the department had established a planning
office, and this office had begun an analysis of alternatives for
meeting the new system requirements. Completion of this analysis is
currently scheduled for December 2010. Following its completion, DOD
expects to select a technical solution for the system and release a
delivery schedule. DOD‘s fiscal year 2011 budget request included $302
million for the EHR Way Ahead initiative.
Weaknesses in key acquisition management and planning processes
contributed to AHLTA having fewer capabilities than originally
expected, experiencing persistent performance problems, and not fully
meeting the needs of users.
* A comprehensive project management plan was not established to guide
the department‘s execution of the system acquisition.
* A tailored systems engineering plan did not exist to guide the
technical development of the system, an effort that was characterized
by significant complexity.
* Requirements were incomplete and did not sufficiently reflect user
and operational needs.
* An effective plan was not used to improve users‘ satisfaction with
the system.
DOD has initiated efforts to bring its processes into alignment with
industry best practices. However, it has not carried out a planned
independent evaluation to ensure it has made these improvements. Until
it ensures that these weaknesses are addressed, DOD risks undermining
the success of further efforts to acquire electronic health record
system capabilities.
What GAO Recommends:
GAO is recommending that DOD take six actions to help ensure that it
has disciplined and effective processes in place to manage the
acquisition of further electronic health record system capabilities.
In written comments on a draft of this report, DOD concurred with
GAO‘s recommendations and described actions planned to address them.
View [hyperlink, http://www.gao.gov/products/GAO-11-50] or key
components. For more information, contact Valerie C. Melvin at (202)
512-6304 or melvinv@gao.gov.
[End of section]
Contents:
Letter:
Background:
AHLTA Has Limited Capabilities and Continues to Experience Performance
Problems:
DOD Has Initiated Planning Activities for the EHR Way Ahead:
AHLTA Performance Was Hindered by Weaknesses in Key Acquisition
Management and Planning Processes:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Comments from the Department of Defense:
Appendix III: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Organizations Responsible for Managing and Providing
Oversight of AHLTA:
Table 2: Capabilities Planned and Delivered for Blocks 1 and 2:
Table 3: Top 10 Priorities for EHR Way Ahead:
Figure:
Figure 1: Overall AHLTA User Satisfaction Ratings between April 2005
and July 2007:
Abbreviations:
AHLTA: Armed Forces Health Longitudinal Technology Application:
CHCS: Composite Health Care System:
CIO: chief information officer:
CITPO: Clinical Information Technology Program Office:
CMMI: Capability Maturity Model Integration:
DHIMS: Defense Health Information Management System:
DOD: Department of Defense:
EHR: Electronic Health Record:
IEEE: Institute of Electrical and Electronics Engineers:
MHS: Military Health System:
SEI: Software Engineering Institute:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
October 6, 2010:
The Honorable Judd Gregg:
Ranking Member:
Committee on the Budget:
United States Senate:
Dear Senator Gregg:
This report responds to your request that we examine the Department of
Defense's (DOD) efforts to implement its military electronic health
record system known as the Armed Forces Health Longitudinal Technology
Application (AHLTA). When fully deployed, AHLTA was envisioned to
provide the department with a modernized health information system
that would generate and maintain a comprehensive, lifelong, computer-
based patient record for every soldier, sailor, airman, and marine;
their family members; and others entitled to DOD military health care.
The electronic health record was expected to give health care
providers real-time access to individual and military population
health care information, thus facilitating clinical decision support
and rationale for care rendered to U.S. service members worldwide.
However, after more than a decade of effort to deliver this system,
the department has recently begun planning for a new electronic health
record system.
At your request, we conducted a study of DOD's efforts to acquire and
implement its electronic health record system. Specifically, our
objectives were to (1) determine DOD's status in implementing AHLTA,
(2) determine the department's plans for acquiring a new system, and
(3) evaluate the department's acquisition management for its
electronic health record system.
To accomplish the objectives, we reviewed relevant program
documentation and interviewed appropriate DOD officials. Specifically,
to determine the status of the AHLTA project, we reviewed project
plans and status reports. To determine the department's plans for
acquiring a new electronic health record system, we reviewed relevant
planning documents, including an initial capabilities document. To
evaluate the department's management of its electronic health record
acquisition, we compared the department's activities for project
management planning, systems engineering management, requirements
development and management, user satisfaction feedback, and
acquisition management with DOD guidelines and industry best practices.
We conducted this performance audit from September 2009 to October
2010 at DOD offices in Falls Church, Virginia, and Bethesda, Maryland,
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives. A more
complete description of our objectives, scope, and methodology is
provided in appendix I.
Background:
DOD operates a worldwide health care program, through which it
provides medical care and assistance to 9.6 million active duty
service members, their families, and other eligible beneficiaries. Its
health care operations are significant, involving approximately
135,000 personnel in approximately 700 Army, Navy, and Air Force
medical facilities in 12 domestic regions, as well as European,
Pacific, and Latin American regions. The department's fiscal year 2010
budget for providing health care services was about $49 billion.
DOD's health care program is a responsibility of the Office of the
Undersecretary of Defense for Personnel and Readiness. Within the
Office of the Undersecretary is the Office of the Assistant Secretary
of Defense for Health Affairs, which is responsible for the
department's Military Health System (MHS) program.
MHS has two missions: wartime readiness (maintaining the health of
service members and treating wartime casualties) and peacetime care
(providing for the health care needs of the families of active-duty
members, retirees and their families, and survivors). The Assistant
Secretary of Defense for Health Affairs establishes policy regarding
health care for all DOD beneficiaries and also plans and budgets for
health care operations and maintenance. At the same time, each
military service has its own medical department that operates medical
facilities (referred to as military treatment facilities) and recruits
and funds military medical personnel. Currently, the military
treatment facilities include 59 military hospitals and 650 medical and
dental clinics. DOD provides about half of MHS services through these
military facilities, supplementing this by contracting for health
services with civilian contract providers. Active-duty members are
required to obtain care at military treatment facilities if such care
is available; in contrast, retirees and dependents may obtain care at
either military facilities or through civilian contract providers.
History of DOD's Electronic Health Record System:
To facilitate the delivery of medical services, in 1988, DOD initiated
the acquisition of an electronic health record system to support all
of its hospitals and clinics. This system, the Composite Health Care
System (CHCS), was intended to be the primary medical information
system deployed worldwide to support the department's hospitals and
clinics. DOD envisioned that it would provide automated support for
patient administrative functions (such as registrations, admission,
and disposition); ordering and retrieving results of laboratory and
radiology procedures; ordering and recording prescriptions; and
patient appointment scheduling.
CHCS was deployed in 1993; however, it was supported by numerous stand-
alone medical information systems, such as the department's Ambulatory
Data System, Preventive Health Care Application, and Nutrition
Management Information System,[Footnote 1] and was not designed to
facilitate the exchange of information from one system or military
treatment facility to the next. Specifically, CHCS was facility-
centric, in which each facility stored only its own medical
information for patients using different data standards. Therefore, if
a medical provider wanted to obtain complete information about a
patient, a query would have to be made to each of the CHCS locations--
a time-and resource-intensive activity. Additionally, when a patient
moved to another region, the electronic records did not transfer
across the CHCS locations because of the different data standards at
each location. The lack of an integrated system perpetuated the
reliance on paper-based records, leading DOD to pursue a comprehensive
electronic health care record.
To this end, in 1997, the department initiated the CHCS II program to
address the need for a comprehensive, lifelong, computer-based health
care record for every service member and their beneficiaries. The
vision for CHCS II was to provide access to a patient's health care
information with a single query by providers in military treatment
facilities. Specifically, with this system, DOD planned to provide
worldwide access to outpatient, inpatient, dental, and vision records,
and to make them available 24 hours a day, 7 days a week. This new
system was to be accomplished with the use of a centralized repository
of all health care information derived using common data standards.
The system was to build on capabilities of existing systems, subsuming
their functionality over time, while adding new functionality to meet
mission needs.
CHCS II's architecture was to be an open system, client-server design
of three levels: the user (client) workstation at various DOD
locations, the DOD computers' (servers') operating system and storage
hardware and software, and a clinical data repository at a remote
computing center where the information would be stored.[Footnote 2]
The department had planned to connect all workstations at an
installation's hospital or clinic to the servers through the
installation's local or wide area network. It had planned to divide
the system acquisition into seven software releases to be delivered
incrementally by June 2006 at an estimated cost of $4.3 billion (in
1998 dollars).
The department's original plan had called for deploying a prototype
system in October 1998 and beginning deployment of the initial version
in about April 1999. However, the department did not meet its schedule
to deliver initial CHCS II system capabilities and associated mission
benefits by April 1999; it reported that the initial deployment was
delayed by 6 months because of a failure to meet initial performance
requirements and changes in system requirements.
In July 2000, the department redefined its plans for the system to
include adopting a new technical architecture, establishing a means
for controlling changes to requirements, and committing to the
incremental release of system capabilities. It also delayed the
decision date for deploying the initial system capabilities (for
outpatient documentation) to January 2001--21 months later than its
original commitment for the system.
However, the department did not meet this commitment, and subsequently
established a new plan that called for incrementally deploying
functionality to achieve the system's full operational capability.
Delivery of the system was to commence in July 2003 and was to be
completed by September 2007, yielding four blocks of capabilities that
would incrementally populate the system's electronic health record at
a revised estimated life-cycle cost of $3.8 billion through 2017.
* Block 1 was to make outpatient information available worldwide on a
continuous basis through the electronic health record system (as
opposed to CHCS legacy functionality which only made records available
at a single location), provide encounter documentation, aid in order
entry/results retrieval, assist in encounter coding support, provide
alerts and reminders (such as drug interaction alerts and special duty
status), facilitate role-based security, and establish a health data
dictionary and a master patient index.
* Block 2 was to provide automated clinical practice guidelines,
optometric documentation, and dental documentation.
* Block 3 was to replace CHCS ancillary functionality for results
retrieval and order entry for outpatient encounters such as laboratory
and automatic pathology, pharmacy, and radiology.
* Block 4 was to provide for inpatient order entry and management,
including inpatient clinical and critical care documentation.
When delivered, the system was to allow users to create and store
computer-based patient records using workstation-and computer-based
software packages. Each facility's workstations and servers were to be
connected via each installation's local or wide area networks.
Further, each installation was to be connected through a wide area
network to a defense computing center where the patient records would
be stored in a database known as the clinical data repository. DOD
intended that medical providers would ultimately be able to access a
patient's computer-based record from any military treatment facility,
no matter where the patient was being or had been treated.
According to program documentation, the department began worldwide
deployment of Block 1 in January 2004. It completed the deployment of
this block in December 2006. However, program officials stated that
users experienced numerous performance problems with the capabilities
that were delivered, which impacted its usability, speed, and
availability. Specifically, the department reported experiencing the
following problems with the delivery of Block 1:
* Usability. The system did not support varied clinical workflow to
meet the needs of various types of practitioners, had missing or
incomplete clinical capabilities (e.g., consult and referrals
management, ancillaries, specialty workflow support), did not support
fully unified or user-customizable patient data, and did not have a
user-friendly interface.
* Speed. The system did not have the speed or performance to
efficiently support the clinicians' workflow in certain environments
and was affected by problems such as coding and infrastructure which
impacted its speed.
* Availability. The system was not reliable on a 24-hour-a-day, 7-day-
a-week basis; it had no backup for disaster recovery; and the data
repository experienced system shutdowns and functional interruptions.
As a result of the system problems associated with Block 1, DOD set a
new date for system completion--September 2011--and increased the
projected life-cycle cost of the system to approximately $5 billion,
which it attributed primarily to the need for increased operations and
maintenance for Block 1.
The department also took a number of other steps with regard to the
initiative. Specifically, in May 2005, it terminated plans for
deploying the Block 4 inpatient functionality with the intent of
moving this functionality into Block 3.[Footnote 3] However, due to
continuing performance problems with the functionality that had been
delivered, and because the Block 3 deployment had exceeded the
department's 5-year limit for achieving initial operational capability
by January 2008, DOD terminated Block 3 (laboratory, radiology, and
pharmacy) as well. This action left only one of the four planned
blocks--Block 2--for implementation. Although the department reduced
the scope of the initiative to only two blocks, the estimated life-
cycle costs were revised back to the original $3.8 billion (through
2021). However, the department encountered performance problems with
the Block 2 dental module as well and, in December 2009, MHS senior
leadership implemented a strategic pause in its further deployment.
Beyond these actions, the department took other steps over the course
of the initiative. Specifically, in November 2005, the Assistant
Secretary of Defense for Health Affairs announced a change in the name
of the system from CHCS II to AHLTA, but did not give a specific
reason for doing so. Further, as part of its attempt to improve the
system, DOD awarded several contracts between fiscal year 2006 and
fiscal year 2009 for a total of approximately $40 million to address
performance problems and implement software enhancements. The
contractors began deployment of these software enhancements (which DOD
referred to as AHLTA 3.3) in December 2008.
DOD's Acquisition Process for Its Electronic Health Record:
To acquire its electronic health record system, DOD used several
contractors and types of contracts.[Footnote 4] These included fixed-
price, time-and-materials, and cost-plus-fixed-fee contracts, each of
which involved a different level of cost or performance risk for the
government.[Footnote 5] The prime developer and lead integrator for
CHCS II, Integic (acquired by Northrop Grumman in 2005), was awarded a
time-and-materials contract for about $65.4 million in 1997 and was
tasked to perform systems engineering, requirements analysis,
architecture evaluation, software design and development, engineering
and development testing, test and evaluation, maintenance, site
installation and implementation, and training. Contracts for system
development and integration continued through fiscal year 2009.
DOD also used noncompetitive contracts[Footnote 6] for the development
of the system. According to the program office, 11 noncompetitive
contracts and task or delivery orders, totaling approximately $44.6
million,[Footnote 7] were awarded for the system from fiscal year 2004
through fiscal year 2012. Program officials stated that the
noncompetitive contracts were awarded on the basis that (1) DOD's need
for the supplies or service was so urgent that providing each awardee
under a multiple award contact a fair opportunity would have resulted
in unacceptable delays; (2) only one awardee was capable of providing
the supplies or services required at the level of quality required
because the supplies or services ordered were unique or highly
specialized; or (3) an order was a logical follow-on to an order
already issued under the contract.[Footnote 8]
According to AHLTA program documentation, the system acquisition was
guided by the defense acquisition system, which is documented in the
department's DOD 5000.02 Instructions. The defense acquisition system
consists of five key program life-cycle phases and three related
milestone decision points that major acquisitions must meet in order
to proceed to the next phase of the acquisition.[Footnote 9] At each
milestone point, the program is reviewed by a milestone decision
authority to determine whether it can move to the next life-cycle
phase.
The five phases of the defense acquisition are as follows:
1. Materiel solution analysis: The purpose of this phase is to assess,
through an analysis of alternatives, potential solutions to satisfy an
approved capability need.
2. Technology development: The purpose of this phase is to determine
and mature the appropriate set of technologies to be integrated into
the investment solution by iteratively assessing the viability of the
various technologies while simultaneously refining user requirements.
To enter this phase, a program must have an approved analysis of
alternatives and pass milestone A. To exit this phase, the acquisition
must demonstrate affordable technology.
3. Engineering and manufacturing development: The purpose of this
phase is to develop a system or an increment of capability, and
demonstrate integrated system design through developer testing to show
that the system can function in its target environment. To enter this
phase, a program must have approved requirements and pass milestone B.
To exit this phase, the acquisition must meet performance requirements
in the intended environment.
4. Production and deployment: The purpose of this phase is to achieve
an operational capability that satisfies the mission needs, as
verified through independent operational test and evaluation, and to
implement the system at all applicable locations. To enter this phase,
a program must have completed development testing and pass milestone
C. To exit this phase, the system must be deployed and ready to
operate for all users.
5. Operations and support: The purpose of this phase is to
operationally sustain the system in the most cost-effective manner
over its life cycle. DOD criteria do not require that the milestone
decision authority conduct milestone reviews during the period after a
system has been deployed and stabilized.
For the purpose of conducting milestone reviews, AHLTA was assigned
the highest level of oversight for DOD information system
acquisitions.[Footnote 10] As such, oversight was provided within the
Office of the Secretary of Defense.
Management Structure for AHLTA:
Various DOD units were involved in acquiring and deploying AHLTA. As
the principal advisor to the Assistant Secretary of Defense for Health
Affairs and to the DOD medical leaders on all matters related to
information management and information technology, the MHS chief
information officer (CIO) has primary responsibility for overseeing
the acquisition, development, testing, and deployment of AHLTA to the
military treatment facilities. Key offices within the Office of the
MHS CIO perform critical information management and information
technology functions to support AHLTA, including the Joint Medical
Information Systems Office, which is responsible for the testing,
implementation, training, fielding of system components, operations,
maintenance, and ultimate disposal of system components.
Also within MHS, the Composite Health Care System (CHCS) II Program
Office was established in January 1997 to provide direct management of
the project; it had operational responsibility for the acquisition and
deployment of the electronic health record, as well as the migration
of the numerous standalone clinical information systems. In fiscal
year 2000, the CHCS II program office was renamed the Clinical
Information Technology Program Office (CITPO). In 2008, with the
merger of CITPO and the MHS Theater Medical Information Program
Office--Joint, the office is now called the Defense Health Information
Management System (DHIMS).
To provide oversight in accordance with DOD's defense acquisition
system, the Assistant Secretary of Defense for Networks and
Information Integration, within the Office of the Secretary of
Defense, was designated the milestone decision authority responsible
for deciding at each acquisition cycle milestone whether the project
could proceed to the next milestone. The project also received
oversight from several other bodies, including the Human Resources
Management Investment Review Board, headed by the MHS CIO, and the
Overarching Integrated Project Team, which evaluated project
performance in accordance with DOD 5000 and approved acquisition
program baselines and acquisition decision memorandums.
Table 1 summarizes the assignment of responsibilities for AHLTA among
the various DOD units.
Table 1: Organizations Responsible for Managing and Providing
Oversight of AHLTA:
Management organizations:
Organization: Office of the Assistant Secretary for Health Affairs;
Description: Responsible for the department's military health system
program. Establishes policy regarding health care operations and
maintenance. Several units within this office, including MHS, are
involved in acquiring and deploying AHLTA.
Organization: MHS CIO;
Description: Oversees the MHS information management and technology
program.
Organization: Joint Requirements Oversight Council;
Description: Approves mission need and operational requirements for
automated information systems with joint (i.e., multiservice) interest.
Organization: Joint Medical Information Systems Office--Deputy CIO;
Description: Supports health care operations through design,
development, test, evaluation, and deployment of medical information
systems. The Program Executive Office is responsible for each of the
program management offices that oversee this activity.
Organization: Defense Health Information Management System Program
Office;
Description: Manages the acquisition, development, deployment, and
maintenance of AHLTA and other related systems. The program office
reports to the Joint Medical Information Systems Deputy CIO. Within
the program office, the project officer is responsible for ensuring
successful planning, technical development, and acquisition of
specific information applications and elements of AHLTA. The office
was established in June 2008 with the merger of CITPO--the original
CHCS II program office--and the Theater Medical Information Program
(the office responsible for acquiring the theater portion of the
electronic health record).
Oversight organizations:
Organization: Office of the Assistant Secretary of Defense, Networks
and Information Integration;
Description: Acts as the milestone decision authority that authorizes
AHLTA's readiness to move into each phase of the acquisition life
cycle, based on successful completion of the criteria for the
preceding phase. Conducts milestone reviews and prepares decision
memorandums.
Organization: Human Resources Management Investment Review Board;
Description: This board is responsible for annual certification to
ensure AHLTA meets specified requirements and should be approved for
funding.
Organization: AHLTA Overarching Integrated Product Team;
Description: Reviews program planning in support of the milestone
decision authority, including oversight, review, and evaluation of
project execution performance relative to DOD guidance.
Source: GAO analysis of DOD data.
[End of table]
Previous Reviews of DOD's Electronic Health Record Initiatives
Highlighted Management Deficiencies and Risks:
DOD's Inspector General and we have previously reported on the
department's actions toward acquiring its new health care information
system and have noted the need for improvement in key management
areas, such as project management, contract management, and risk
management.
In reporting on the department's efforts in January l999,[Footnote 11]
the Inspector General noted that the project management system for the
acquisition (called CHCS II at the time of the report) was not
complete. While finding that DOD had taken positive actions to manage
the acquisition, the report noted that the department had not
established a project management control system to evaluate and
measure the program's performance. In addition, the report stated that
the program's funding visibility was limited because DOD was combining
funding for sustaining the system with modernization funding for CHCS
and other clinical business area automated systems. The Inspector
General made recommendations related to designing and implementing a
project management control system, the reporting of funding for the
system, and providing milestone exit criteria that demonstrated the
level of performance, accomplishments, and progression.
Further, in May 2006,[Footnote 12] the Inspector General conducted an
evaluation of the project's program requirements, the related
acquisition strategy, and system testing to determine whether the
system was being implemented to meet cost, schedule, and performance
requirements. While the report found that the program management
office was using risk mitigation techniques, such as risk management,
lessons learned, and performance monitoring, the program remained at
high risk because of the complexities of integrating commercial, off-
the-shelf software into the existing program. In particular, the
report noted that the program office had not identified any mitigation
strategies to reduce and control program risk related to integration
of commercial, off-the-shelf software for the third block of
functionality. As a result, the Inspector General concluded that the
program was vulnerable to continued increases in cost, extended
schedules for implementation, and unrealized goals in performance from
underestimating the difficulties of integrating commercial, off-the-
shelf products. Subsequently, the program office developed mitigation
strategies, but the Inspector General reported that they were
inadequate and did not follow risk management guidance, including
identifying significant activities and milestones. Accordingly, the
Inspector General recommended that the program office develop more
robust mitigation strategies in accordance with the program office's
risk management plan.
We have also reported on DOD's management of the system acquisition,
noting the need for improvements. For example, in 2002, we reported
that, because the department had not estimated the cost of delivering
the initial system capabilities, it had lacked a cost commitment
against which to measure progress.[Footnote 13] In addition, we noted
that program benefits were in question since measurements had not yet
begun and that costs were about two-and-a-half times the l998
estimate. Further, DOD had initially identified a single economic
justification for the entire project, which had been used as the basis
for its system releases, and had not treated the releases as separate
investment decisions. Finally, DOD had not followed performance-based
contracting practices, resulting in the risk that the system would
take longer to acquire and cost more than necessary.
Accordingly, we recommended that DOD expand its use of best practices
in managing the system by (1) modifying the project's investment
strategy to justify investment in each system release before beginning
development and measuring return on investment and (2) employing
performance-based contracting practices where possible on all future
delivery orders. The department agreed with these recommendations and
took actions to update and validate its life-cycle cost estimate in
September 2002. This was used by the department to approve the
deployment of the system release. Also, the department employed
performance-based contracting practices, such as using performance
standards, quality assurance plans, and contractor incentives on CHCS
II delivery orders.
AHLTA Has Limited Capabilities and Continues to Experience Performance
Problems:
Despite having obligated approximately $2 billion over the 13-year
life of its initiatives to acquire and operate an electronic health
record system, as of September 2010, DOD continued to experience
performance problems with the one block of AHLTA functionality (Block
1) that it had fully deployed and with a second block of functionality
(Block 2) that it had partially deployed. Further, after having
terminated its plans for deploying the two other blocks of
functionality (Block 3 and Block 4) that were intended to be part of
the system, the department has identified April 2011 as the date by
which it now expects to achieve full operational capability of the
scaled-backed AHLTA system. Program officials told us they are taking
steps to stabilize the existing system capabilities through 2015, as
the department proceeds with plans to pursue yet another new
electronic health record system.
In deploying Block 1, the department reported that it achieved all of
the planned outpatient capabilities for direct patient care, including
encounter documentation, order entry and results retrieval, encounter
coding support, consult tracking, and alerts and reminders. According
to the department, it deployed the AHLTA outpatient documentation
capability worldwide, providing 77,000 clinicians with the ability to
document over 148,000 outpatient encounters daily. The department
stated that medical providers can access the patient's computer-based
record from any military treatment facility. Also, DOD currently
shares a significant amount of patient information with the Department
of Veterans Affairs, including outpatient pharmacy data, laboratory
results, and radiology results on shared and separated service members.
In addition, with the deployment of Block 2, including enhancements to
Block 1, dental capabilities were provided to 73 of 375 dental
treatment facilities, allowing graphical dental charting, order and
entry results retrieval, and automated dental readiness
classification. In this regard, the capabilities were deployed to 46
Air Force dental medical facilities, 25 Navy facilities, and 2 Army
facilities. Further, program officials stated that in October 2009,
because of technical and functionality upgrades made over time to the
legacy Spectacle Request Transmission System, funding was ceased for
optometric capabilities for Block 2. The department stated that it
plans to achieve full operational capabilities by April 2011. Table 2
shows the capabilities planned and delivered for Blocks 1 and 2.
Table 2: Capabilities Planned and Delivered for Blocks 1 and 2:
Block 1 (outpatient care):
Capability: Encounter documentation;
Status: Met.
Capability: Order entry and results retrieval;
Status: Met.
Capability: Encounter coding support;
Status: Met.
Capability: Consult tracking;
Status: Met.
Capability: Alerts and reminders;
Status: Met.
Capability: Health data dictionary;
Status: Met.
Capability: Master patient index;
Status: Met.
Capability: Role-based security;
Status: Met.
Block 2:
Capability: Dental charting and documentation;
Status: In progress.
Capability: Optometric documentation and order entry;
Status: Not Met.
Source: GAO analysis of DOD data.
[End of table]
Nonetheless, program officials, as well as users of the system,
acknowledged that problems with the system's performance have
persisted. During a demonstration of the system's operation in April
2010, medical providers discussed problems with AHLTA, including
limitations in its availability and usability. For example, the
providers participating in the demonstration stated that it is time-
consuming to document encounters using AHLTA because of the time
required to enter information and navigate through the application
screens. Thus, they sometimes must document portions of an outpatient
encounter after the patient leaves. In their experience, using the
system at the time of the encounter would take attention away from the
patient for unacceptable periods of time. Also, they stated that when
system downtime occurs, providers can neither access patient data nor
electronically document care; in these instances, medical notes are
recorded manually and later entered in the system after it returns to
operation--an inefficient process.
As noted in the earlier discussion, since fiscal year 2006 the
department has been taking steps to address performance problems and
enhance existing system capabilities.[Footnote 14] DOD is proceeding
with what it refers to as a "stabilization effort" to continue making
improvements to the system and provide ongoing capabilities until a
new system is acquired. According to DOD officials, the estimated cost
of this effort for fiscal year 2010 through fiscal year 2015 is $826.3
million. The stabilization effort is expected to improve the speed,
availability, and usability of the system; moreover, according to
officials in the Office of the Deputy Secretary of Defense, the
stabilization effort is expected to allow the department to meet its
near-term needs and implement additional enhancements to support its
future system.
DOD Has Initiated Planning Activities for the EHR Way Ahead:
Because AHLTA has consistently experienced performance problems and
has not delivered the full operational capabilities intended, DOD has
initiated plans to develop a new electronic health record system. This
new initiative is called the Electronic Health Record (EHR) Way Ahead.
As with AHLTA, department officials stated that the new electronic
health record system is expected to be a comprehensive, real-time
health record for active and retired service members, their families,
and other eligible beneficiaries. They added that the new system is
being planned to address the capability gaps and performance problems
of previous iterations, and to improve existing information sharing
between DOD and the Department of Veterans Affairs and expand
information sharing to include private sector providers.
Thus far, the department has taken several steps to launch its
acquisition of the new system. Specifically, in February 2010 it
established the EHR Way Ahead Planning Office to identify options for
the future electronic health record system. The planning office
currently resides within the MHS Joint Medical Information Systems
Program Executive Office under the Office of the CIO.
In May 2010, the department approved plans to assess solutions for the
new electronic health record system. In this regard, the planning
office began conducting an analysis of alternatives to provide
guidance on selecting a technical solution. According to planning
officials, efforts to develop the analysis of alternatives are being
supervised by the Office of the Assistant Secretary of Defense for
Health Affairs, and this analysis is expected to define and evaluate
reasonable alternatives for meeting the capability requirements. The
analysis is currently scheduled to be completed by December 2010.
To facilitate the analysis of alternatives, planning officials stated
that they had identified system capabilities needed to meet the
department's medical mission. They added that a list of the "top 10"
priority capabilities for a new system had been developed based on the
gaps identified in prior iterations of their electronic health
systems. (These priorities are summarized in table 3.)
Table 3: Top 10 Priorities for EHR Way Ahead:
Priority: 1;
Capability needed to meet DOD's medical mission: Comprehensive medical
and dental documentation, including encounter data, medications,
physical examinations, occupational health (including industrial
hygiene), environmental exposure information and ancillary service
data (both inpatient and outpatient), documentation of care plan
objectives, alternatives, patient education, health care services
provided, patient disposition instructions (including deaths), and
disposition of remains.
Priority: 2;
Capability needed to meet DOD's medical mission: Global capture and
exchange of all health data for beneficiaries--direct care, network,
managed care, Veterans Affairs, active duty components, reserve
components, etc.
Priority: 3;
Capability needed to meet DOD's medical mission: Inpatient and
outpatient order entry and management (laboratory, pharmacy,
radiology, consults, health care plans, nutrition management,
prescription spectacle orders).
Priority: 4;
Capability needed to meet DOD's medical mission: Laboratory diagnostic
services (includes results, retrieval and reporting);
pharmacy services (includes dispensing, operations, reporting, and
pharmacy data transaction service); radiology diagnostic services
(includes imagery capture, results, retrieval, and reporting).
Priority: 5;
Capability needed to meet DOD's medical mission: En-route care
documentation on any transport platform.
Priority: 6;
Capability needed to meet DOD's medical mission: Results retrieval
(ancillary services and consults).
Priority: 7;
Capability needed to meet DOD's medical mission: Data collection and
decision support in austere environments starting at the point of
injury and continuing through all levels of care.
Priority: 8;
Capability needed to meet DOD's medical mission: Consult and referral
management (includes referrals to the civilian health care sector).
Priority: 9;
Capability needed to meet DOD's medical mission: Assessments of
medical deployability of individual service members.
Priority: 10;
Capability needed to meet DOD's medical mission: Patient
administration (includes who the patient is, what he/she is entitled
to, where he/she is located, etc.).
Source: GAO analysis of DOD data.
[End of table]
According to planning documents, following completion of the analysis,
DOD expects to select a technical solution and to develop and release
a delivery schedule.
DOD's fiscal year 2011 budget request includes $302 million for the
EHR Way Ahead initiative.[Footnote 15] For fiscal year 2012, the
department intends to submit an updated budget request and the
schedule for delivery of the EHR Way Ahead based on the results of the
analysis of alternatives.
AHLTA Performance Was Hindered by Weaknesses in Key Acquisition
Management and Planning Processes:
The success of a large information technology project such as AHLTA is
dependent on an agency possessing capabilities to effectively plan and
manage acquisitions, design the associated systems, define and manage
system requirements, and use effective measures to gauge user
satisfaction. In the case of AHLTA, weaknesses in these key management
areas contributed to DOD delivering a system that provided fewer
capabilities than originally expected, experienced persistent
performance problems, and ultimately, did not fully meet the needs of
its intended users. Alleviating these areas of weakness will be
essential to the success of further initiatives, including the AHLTA
stabilization effort and the EHR Way Ahead, that the department
undertakes in pursuit of its electronic health record system
capabilities.
Project Plan Was Incomplete and Not Maintained:
Program management principles and best practices emphasize the
importance of having a project management plan in place that, among
other things, establishes a complete description that ties together
all program activities and evolves over time to continuously reflect
the current status and desired end point of the project.[Footnote 16]
An effective plan is comprised of a description of the program's
scope, cost, lines of responsibility and authority, management
processes, and schedule. Such a plan incorporates all the critical
areas of system development and is to be used as a means of
determining what needs to be done, by whom, and when.
Other guidance, such as our Information Technology Investment
Management framework,[Footnote 17] states that effective program
oversight of IT projects and systems, including those in operation and
maintenance, involves maintaining approved project management plans
that include expected cost and schedule milestones and measurable
benefit and risk expectations.
However, officials did not follow best practices in developing a
project management plan to guide the department's electronic health
record system. Although the department established a project
management plan, it did not include several standard components such
as the project's scope, a requirements management plan, cost estimates
and baseline, a schedule, and a staffing management plan. In addition,
although DOD identified the plan as a keystone document for guiding
the project, the plan was last revised in 2005 and was not updated
during subsequent development work and the operations and maintenance
phase to reflect significant changes to the program. These changes
included termination and postponement of planned capabilities, and
revisions to the acquisition processes used to guide the AHLTA
program. As a result, a plan was not in place to effectively guide the
program throughout these changes. Moreover, there is no such plan to
guide current activities associated with the stabilization effort,
which, as discussed previously, involves attempts to address system
performance problems and enhance functionality.
According to program officials, the project management plan was last
revised in 2005 before their focus shifted to addressing the system
performance problems that occurred as a result of completing Block 1
deployment in December 2006. Nevertheless, significant changes
occurred to the program's scope, cost, and schedule after Block 1
deployment, and the agency lacked a current and complete plan to guide
activities and measure program progress. Going forward, developing and
maintaining a comprehensive project plan will be an essential tool for
overseeing the AHLTA stabilization effort, which is to provide crucial
improvements to the system and act as a bridge over the next 5 years
to the deployment of the EHR Way Ahead system. Further, having a
comprehensive and current project plan for the EHR Way Ahead program
will help to guide the project and provide oversight of the project's
progress. Without a project management plan that reflects the status
and goals of the project, DOD increases the risk that stakeholders
will not have the insight into program status that is needed to
exercise effective oversight of both the AHLTA stabilization effort
and the EHR Way Ahead acquisition.
DOD Lacked a Systems Engineering Plan to Guide the Electronic Health
Record System's Design:
According to industry best practices,[Footnote 18] systems engineering
governs the total technical and managerial effort required to
transform a set of user requirements and expectations into specific
capabilities and, ultimately, into a system design that will meet
users' needs. Systems engineering practices include developing
solutions for achieving system performance requirements such as system
availability, and ensuring compatibility when integrating multiple
systems and their components. Further, DOD guidance states that a
tailored and detailed systems engineering plan is a critical tool for
guiding systems engineering practices throughout the life of an
acquisition program. Having such a plan is particularly important for
a system characterized by significant technical complexities.
DOD's electronic health record system design reflected numerous
technical complexities, such as the need to capture, manage, and share
health information across a worldwide network that must be available
24 hours a day, 7 days a week, and that is to serve a transient
patient population. In addition, the system design involved a network
that had to be integrated with a central patient database and multiple
nonstandard hardware and software platforms, such as commercial, off-
the-shelf products at over 800 military treatment facilities.
Nonetheless, although the program office recognized these types of
system complexities as being part of the electronic health record
system design, the office never established a tailored systems
engineering plan to guide the acquisition, or to facilitate the
resolution of the many performance problems that have plagued the
system since its initial deployment.
In this regard, a particularly troublesome area for the department has
been in deploying enhancements to the system. For example, following
Block 1 deployment in 2006, the department implemented local cache
servers in an attempt to improve the system's operational
availability. According to the department, the specific purpose of the
local cache servers had been to mitigate the need to access patient
medical information in the central data repository during system
outages. However, after the servers were deployed, DOD realized that
the placement of the servers within the system architecture did not
resolve the problem and created a single point of failure. Rather than
yield operational improvements, department officials acknowledged that
these actions resulted in additional challenges, including the need
for a costly local cache server redesign, which was begun in fiscal
year 2009. Program documentation noted that the local cache server
effort was probably one of the most difficult engineering challenges
that the program office had faced so far. Further, as various issues
were faced, it became increasingly clear that detailed planning in the
earlier stages was not what it could have been. In April 2010,
clinicians demonstrating the system at the Bethesda Naval Medical
Center stated that the servers continued to be a major contributing
factor to system availability issues.[Footnote 19]
The lack of a systems engineering plan to guide the program office
through this type of complexity is particularly notable in light of
the DOD Inspector General's report of 2006, which stated that
inadequate planning for technical complexities significantly impacts
the cost, schedule, and performance of a program. The report further
stated that the AHLTA program office had underestimated the technical
complexity of integrating products with the electronic health record
system and, as a result, remained at high risk for continued cost
increases, schedule overruns, and unrealized performance goals.
In discussing this matter, agency officials stated that a tailored
systems engineering plan had not been developed to guide the design of
AHLTA because such a plan was not required when the system was
originally planned. Specifically, the officials stated that, it was
not until February 2004 that DOD issued a policy requiring that a
systems engineering plan be in place for acquisition programs'
milestone reviews; but all milestone reviews for AHLTA had been
completed prior to this time.
However, current DOD guidance emphasizes the need for a tailored
systems engineering plan to guide all systems engineering practices,
including those that occur after the completion of milestone reviews.
Without a tailored systems engineering plan to guide the program's
efforts to address long-standing system performance problems as part
of the AHLTA stabilization efforts, the department may continue to be
challenged in achieving the desired results. Further, in planning for
the acquisition of the new EHR Way Ahead system, it will be essential
that the department establish early in the process and have in place a
detailed and tailored plan to avoid encountering technical challenges
similar to those of the AHLTA program, and thus again failing to meet
users' needs.
Weaknesses in DOD's Requirements Processes Impacted AHLTA's Usability:
According to recognized guidance,[Footnote 20] using disciplined
processes for developing and managing requirements can help reduce the
risks of developing a system that does not meet user and operational
needs. Requirements should serve as the basis for establishing
agreement between users and developers and a shared understanding of
the system to be developed. Effective requirements development
practices include, among other things, involving users in identifying
requirements throughout the project's life cycle to ensure system
requirements are complete and accurately reflect their needs.
Effective requirements management practices include maintaining
bidirectional traceability of requirements to ensure that system-level
requirements can be traced both backward to high-level operational
requirements, and forward to low-level system design specifications.
For the AHLTA acquisition, program documentation revealed that users
were not adequately involved throughout the requirements development
process. According to the documentation, users did not seek
involvement in the requirements development process and system
developers did not seek user input when making changes to
requirements. As a result, requirements were neither complete nor
sufficiently detailed to guide system development, and did not
adequately provide a shared understanding between the users and
developers of how the system was to be developed. Program
documentation noted that requirements often were not adequately
specified and did not adequately reflect user needs. In particular,
the program documentation revealed that, while users were involved in
developing an initial set of requirements used to make system
acquisition decisions, they were largely not involved in identifying
new requirements and making changes to existing ones while the system
was being developed and deployed.
In certain instances, because users were involved only at the
beginning and end of the requirements development process, they were
only able to determine that capabilities would not meet their needs
after those capabilities had already been deployed. For example, when
the dental application was in the process of being deployed to Army,
Navy, and Air Force sites, the MHS senior leadership voted to halt
further training and implementation because users reported that the
capabilities were not complete and did not address their needs.
Consequently, alternate dental solutions will be explored as part of
the analysis of alternatives for the EHR Way Ahead, resulting in
additional costs and delays in deploying dental capabilities that will
meet users' requirements.
Since the initial deployment, the department has taken steps to
increase user involvement in defining requirements. For example, to
better involve users in the requirements process and identify issues
with system usability, the program office held conferences in 2006 at
which users identified over 200 new requirements for inclusion in the
system. Program officials stated that the requirements identified
during the conference were used to develop the AHLTA 3.3 software
release. However, our evaluation of the requirements traceability
matrix used to develop the AHLTA 3.3 release showed that bidirectional
traceability had not been fully established; thus, the requirements
were not always linked to high-level operational requirements or to
more detailed design specifications. Without adequate traceability,
the department cannot ensure that all agreed-upon requirements will be
developed, fully tested, and work as intended.
In addition, the department has plans for making improvements in the
requirements management process in its MHS Information Management/
Information Technology Strategic Plan 2010-2015 and includes a goal to
improve the requirements management process to enable greater
participation of system users. According to the plan, this will
improve the value, quality, timeliness, and stakeholder ownership of
the resulting system. However, because the department is in the early
stages of implementing improvements for greater user participation, it
is too early to determine their effectiveness.
As the department proceeds with the AHLTA stabilization effort and the
new EHR Way Ahead system, ensuring that user needs are met will be
essential to effective and cost-efficient delivery of system
capabilities. Until the department ensures that a requirements
development process with adequate user involvement is in place, it
will continue to lack a vital tool for ensuring the efficient and
effective delivery of electronic health record system capabilities
that will meet the needs of its users.
Efforts to Improve User Satisfaction Were Not Guided by Effective
Planning:
DOD has stated that the success of AHLTA can be gauged by improvements
in user satisfaction and user acceptance, among other things. In this
regard, effectively managing program improvement activities to improve
user satisfaction requires planning and executing such activities in a
disciplined fashion. The Software Engineering Institute's IDEALSM
[Footnote 21] model is a recognized approach for managing efforts to
make system improvements. According to this model, user satisfaction
improvement efforts should include a written plan that serves as the
foundation and basis for guiding improvement activities, including
obtaining management commitment to and funding for the activities,
establishing a baseline of commitments and expectations against which
to measure progress, prioritizing and executing activities and
initiatives, determining success, and identifying and applying lessons
learned. Through such a structured and disciplined approach,
improvement resources can be invested in a manner that produces
optimal results.
However, DOD has not demonstrated that user satisfaction improvement
efforts are being guided by a documented plan that defines prioritized
improvement projects and associated resource requirements, schedules,
and measurable goals and outcomes. Instead, efforts that the office
undertook to improve user satisfaction were ad hoc and did not meet
with success. Specifically, the program office stopped measuring AHLTA
user satisfaction levels in July 2007 after overall user satisfaction
had declined to its lowest point in more than 2 years. Between 2005
and 2007 the program office collected user satisfaction feedback
through online user surveys, and used the data to identify areas for
system improvements and to measure progress toward improving
satisfaction. The results of the surveys showed not only that users
rated their overall satisfaction level with the system between below
average and average, but that user satisfaction levels had declined to
a low point with the results of the final survey report of July 2007.
Thus, as shown in figure 1, the program office was not able to improve
user satisfaction during this time period.
Figure 1: Overall AHLTA User Satisfaction Ratings between April 2005
and July 2007:
[Refer to PDF for image: vertical bar graph]
Date: April 2005;
Score: 2.61 (below average).
Date: October 2005;
Score: 3 (average).
Date: February 2006;
Score: 3.17 (average).
Date: June 2006;
Score: 3.05 (average).
Date: October 2006;
Score: 2.72 (below average).
Date: January 2007;
Score: 2.94 (below average).
Date: July 2007;
Score: 2.36 (below average).
Source: GAO analysis of DOD data.
[End of figure]
According to program officials, they have implemented a major effort
toward improving user satisfaction with the AHLTA 3.3 software
release. The improvements associated with this software release began
as early as 2006 and include features such as improved medical coding
support and increased speed of the order entry connection, as well as
other changes to improve users' satisfaction with the system's
performance and capabilities. Yet, program officials did not provide
evidence of a plan to guide these efforts or a schedule for
implementing these improvements, and it is unclear how specific
capabilities of the software release will be used to address specific
user concerns. The lack of a documented plan to guide user
satisfaction improvement activities is of particular significance
because users have continued to express their dissatisfaction with the
system. Program officials stated that additional online user
satisfaction surveys were not conducted after 2007 because users had
grown weary of the surveys and efforts to address user feedback from
the existing survey results are ongoing. The next online survey is
expected to be conducted after full deployment of AHLTA 3.3, but a
schedule has not yet been set.
Given the history of system performance problems and the extent to
which users have not been able to effectively and efficiently use
AHLTA, it is critical that the department identify and implement
system improvements in a disciplined and structured fashion. Without a
documented improvement plan, efforts to improve user satisfaction,
including those associated with the ongoing AHLTA stabilization
effort, may be reduced to trial and error, and the office cannot
adequately ensure that it is effectively investing program resources
on improvement efforts that will result in a system that satisfies
users. Further, since increasing user satisfaction is a key goal for
the EHR Way Ahead, it is critical that a disciplined approach is
established and maintained throughout the program's life cycle.
MHS Lacks Assurance of a Disciplined Acquisition Management Process to
Guide Its Electronic Health Record Initiative:
The use of disciplined processes to guide the effort of acquiring and
implementing a major system has been shown to increase the likelihood
of achieving intended results and reduce the risks associated with an
acquisition to acceptable levels. Although there is no standard set of
practices that will ever guarantee success, several organizations,
such as Carnegie Mellon University's Software Engineering Institute
and the Institute of Electrical and Electronics Engineers (IEEE),
[Footnote 22] as well as individual experts, have identified and
developed the types of policies, procedures, and practices that have
been demonstrated to reduce development time and enhance
effectiveness. The key to having a disciplined system development
effort is to have disciplined processes in multiple areas, including
project planning, requirements management, systems engineering, system
testing, and risk management. Because change in a program is constant,
effective processes should be implemented in each of these throughout
the project life cycle. Effectively implementing the disciplined
processes necessary to reduce project risks to acceptable levels is
difficult because a project must effectively implement several best
practices, and inadequate implementation of any one may significantly
reduce or even eliminate the positive benefits of the others.
Recognizing weaknesses in its acquisition of systems such as AHLTA,
MHS has been taking steps to institutionalize more disciplined
management processes across all of its programs. In March 2008 the MHS
CIO identified an approach for improving its management processes that
included aligning MHS processes with best practices outlined in the
Software Engineering Institute's Capability Maturity Model Integration
(CMMI) for Acquisition. In support of the approach, certain program
offices, including DHIMS (the program office responsible for the AHLTA
acquisition), were selected for an internal evaluation to identify
areas for improvement in the existing MHS processes. The assessment,
which was conducted in May 2008, identified weaknesses in processes
such as project management, requirements development, and project
monitoring and control, among others. It also identified weaknesses in
MHS's oversight of the implementation of these processes within
program offices. Specifically, the assessment identified weaknesses in
the area of Process and Product Quality Assurance, which is supposed
to provide staff and management with objective insight into processes
associated with work products. The assessment found little evidence
that process evaluations were performed across the organization,
quality assurance audits were conducted, and noncompliance issues were
tracked and reported.
In response to the assessment, officials stated that they established
a plan for addressing the identified weaknesses. Specifically, their
goal was to achieve CMMI's "maturity level 2" for processes such as
project planning and acquisition requirements development. Level 2
processes are "managed" processes, or processes that are planned and
executed in accordance with policy; employ skilled people who have
adequate resources to produce controlled outputs; involve relevant
stakeholders; are monitored, controlled, and reviewed; and are
evaluated for adherence to their process description. The department
planned to conduct a formal external assessment of the maturity of its
processes by December 2008.
Program officials stated that they provided guidance and assistance
for program offices to adopt practices associated with CMMI maturity
level 2 processes. However, they have yet to perform the planned
external assessment of their processes, and there is therefore little
assurance that improvements have been carried out. As the department
proceeds with the AHLTA stabilization effort, it is critical that it
have disciplined processes in place to avoid past problems with not
delivering system improvements as planned. Further, as the department
is allocating resources to and planning for the EHR Way Ahead
acquisition, it is critical that it have disciplined management
processes in place to avoid repeating the mistakes of the past. Until
the department ensures that these disciplined and managed processes
are in place, it risks delivering another system with limited
functionality and performance problems and that does not meet the
needs of its users.
Conclusions:
After over a decade of effort, DOD has not accomplished what it set
out to achieve in acquiring a comprehensive electronic health record
system. While it has delivered a number of outpatient capabilities,
weaknesses in key management areas hindered its ability to deliver the
full complement of intended capabilities and to ensure that the
capabilities it has delivered meet required performance parameters.
The program office did not maintain a comprehensive and current
project management plan, a critical document that provides
stakeholders insight into the project's plans and status. Also,
despite the department's need to deliver a complex, worldwide system,
it did not develop a systems engineering plan to help address the
technical aspects of the project, and it continues to experience
problems with system availability, speed, and usability. Further, the
system requirements were too general and did not adequately reflect
user needs. Although the department has collected user feedback, it
did not establish a comprehensive plan for improving user satisfaction
with the system. Recognizing weaknesses in acquisition management
areas, the MHS CIO issued guidance for improving its management
processes, but it has not performed the planned external assessment
that it needs to certify that these improvements have been made or
established a date for doing so.
As DOD continues to invest significant resources in a stabilization
effort to address shortcomings of AHLTA and plan for the acquisition
of a new electronic health record system, it is imperative that the
department take immediate steps to improve its management of the
initiative. Until it does so, it risks a continuation of the problems
it has already experienced, which could again prevent DOD from
delivering a comprehensive health record system for serving its
service members and their families.
Recommendations for Executive Action:
To help guide and ensure the successful completion of the AHLTA
stabilization effort, we recommend that the Secretary of Defense,
through the Assistant Secretary of Defense for Health Affairs, direct
the MHS CIO to take the following six actions:
* Develop and maintain a comprehensive project plan that includes key
elements, such as the project's scope, cost, schedule, and risks and
update the plan to provide key information for stakeholders on the
project's plans and status.
* Develop a systems engineering plan in accordance with DOD guidance
to address the technical complexities of delivering a worldwide
electronic health record system.
* Ensure that its requirements development process involves system
users throughout the development process, to obtain an understanding
of what will satisfy their needs.
* Ensure the establishment of bidirectional traceability for all
system requirements.
* Develop and document a plan for improving user satisfaction that
prioritizes improvement projects; identifies needed resources;
includes schedules for improvement efforts, including future user
feedback surveys; and links efforts to measurable outcomes and
specific user needs.
* Establish acquisition management processes in accordance with
industry best practices, including identifying milestones and a
completion date for the external evaluation that MHS's processes are
at maturity level 2 of the Capability Maturity Model Integration for
Acquisition.
Further, to help ensure that the EHR Way Ahead does not have
shortfalls similar to those experienced with AHLTA, we recommend that
the above six management practices be implemented as part of the
planning for this important initiative.
Agency Comments and Our Evaluation:
The Deputy Assistant Secretary of Defense (Force Health Protection and
Readiness), performing the duties of the Assistant Secretary of
Defense (Health Affairs), provided written comments on a draft of this
report. In its comments, the department agreed with our six
recommendations and described actions planned to address them.
Specifically, to help guide and ensure the successful completion of
the AHLTA stabilization effort, DOD stated that it will develop and
maintain a comprehensive project plan in accordance with our
recommendation and DOD acquisition program guidelines. It also stated
that it plans to develop a systems engineering plan to address the
technical complexities of the project in accordance with current DOD
requirements. Further, to obtain an understanding of system users'
needs, the department stated that it plans to engage users and manage
the requirements development process in accordance with our
recommendation. The department stated that it will ensure that
bidirectional traceability is performed for all system requirements.
Regarding its intent to develop and document a plan for improving user
satisfaction, including identifying needed resources and a schedule
for improvement, the department stated that it will augment its
current user feedback plan to include these and other key elements,
such as measurable outcomes. Further, in response to the need to
establish acquisition management processes in accordance with industry
best practices, at maturity level 2, the department said it plans to
establish a milestone for completing the external review in accordance
with Capability Maturity Model guidelines.
Finally, the department stated that it will ensure that the six
recommendations are implemented as part of future EHR Way Ahead
initiative. To the extent that the department follows through on
implementing the recommendations, it should be better positioned to
deliver a comprehensive electronic health care record for serving its
service members and others entitled to military health care.
DOD also provided technical comments on our draft report. In these
comments, DOD said it took exception to several inaccurate,
misleading, and subjective statements provided in the report. The
department said that GAO's statements conflicted with the extensive
volume of programmatic documentation, written responses, and
consistent interview feedback provided during the course of the audit.
In particular, the department believed that the report did not
sufficiently reflect AHLTA's operational capabilities and its benefit
to DOD's worldwide health care operations. While we agree that the
department provided substantial documentation, we believe that our
analysis of the information received supports our findings. Where
appropriate, however, we have made revisions to statements in the
report to update our discussions of AHLTA's operational capabilities
and the program's management.
The department's written comments are reproduced in appendix II. The
department also provided technical comments, which we have
incorporated in the report as appropriate:
We are sending copies of the report to appropriate congressional
committees, the Secretary of Defense, and other interested parties. In
addition, the report will be available at no charge on the GAO Web
site at [hyperlink, http://www.gao.gov].
If you or your staff have questions about this report, please contact
me at (202) 512-6304 or melvinv@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. Key contributors to this report are
listed in appendix III.
Sincerely yours,
Signed by:
Valerie C. Melvin:
Director, Information Management and Human Capital Issues:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
Our objectives were to (1) determine the Department of Defense's (DOD)
status in implementing the Armed Forces Health Longitudinal Technology
Application (AHLTA) system, (2) determine the department's plans for
acquiring a new electronic health record system, and (3) evaluate the
department's acquisition management for its electronic health record
system.
To determine the department's status in implementing the AHLTA system,
we reviewed project status reports, acquisition decision memorandums,
quarterly defense acquisition executive summaries, monthly in-progress
review reports, monthly contractor performance reports, and
overarching integrated project team meeting minutes. We supplemented
these reviews with interviews of DOD officials in the Defense Health
Information Management System (DHIMS) Program Office, including the
DHIMS Program Manager, Deputy Program Manager, and Director of
Products Branch officials with whom we discussed the project's cost
and schedule, as well as the planning, development, and deployment of
the original and current release of AHLTA. We also attended two
demonstrations of AHLTA: at the program office located in Falls
Church, Virginia, and at the National Naval Medical Center in
Bethesda, Maryland. We observed demonstrations of AHLTA system
functionality and held discussions with system users. We also observed
a daily technical review meeting with technical staff from the Army,
Navy, and Air Force in which the discussion largely focused on the
reporting of issues that caused the system to be unavailable to users
at various locations for up to 24 hours. The discussion also included
identification of known root causes of the availability problems
(e.g., incorrectly configured firewalls, tripped network circuits, and
problems with virtual private networks) and planned actions to address
the issues.
To determine the department's plans for acquiring a new system, we
reviewed Electronic Health Record (EHR) Way Ahead planning documents.
Specifically, we reviewed the acquisition decision memorandum issued
by the milestone decision authority, the Joint Requirements Oversight
Council-approved Initial Capabilities Document to identify EHR needs,
and the Capabilities-Based Assessment. We also reviewed the analysis
of alternatives procedures for guidance on determining a technology
solution for the new EHR. We also reviewed department briefings issued
between 2008 and 2010, as well as a prepared statement to Congress
from 2009 on preliminary plans for the EHR Way Ahead. These documents
provided a high-level overview of the need and the goals for the new
system, as well as plans for the system's enterprise architecture and
expected capabilities. We supplemented our review by interviewing
officials from the EHR Way Ahead planning office, including the
department's Acting Chief Information Officer, the DHIMS Program
Manager, and the DHIMS Deputy Program Manager.
To evaluate the department's acquisition management for its electronic
health record system initiative, we evaluated key practices used by
the agency against best practices. In this regard, we examined
practices related to project management planning, systems engineering
planning, system requirements development and management, and user
satisfaction improvement planning and compared the agency's work with
agency policy, guidance, and recognized best practices. Specifically:
* To assess DOD's project planning for AHLTA, we compared the
program's project management plan against relevant guidance, including
the Military Health System's project management process area
description and our Information Technology Investment Management
framework for assessing and improving process maturity.
* We assessed the agency's approach to systems engineering by
comparing program documentation such as acquisition strategies and the
AHLTA project management plan to systems engineering guidance from the
Defense Acquisition University on systems engineering. We also
reviewed relevant agency policies, such as DOD Instruction 5000.02
which discusses the use of systems engineering across the acquisition
life cycle and memorandums from the Office of the Under Secretary of
Defense on a 2004 revision to the policy regarding use of a systems
engineering plan, to determine whether the AHLTA program was guided by
appropriate systems engineering planning documents such as a systems
engineering plan.
* Regarding requirements development, we reviewed program procedures
describing the processes for developing requirements and reviewed
relevant external evaluations of the effectiveness of those processes
against recognized guidance. Specifically, we reviewed an external
evaluation of the requirements development processes including the
2002 Carnegie Mellon External Assessment of the AHLTA program office
and the process area description or requirements management. We also
reviewed the 2008 internal assessment of requirements management; a
2009 concept of operations document for a more integrated,
departmentwide requirements development process; and the 2010 Joint
Requirements Oversight Council-approved Initial Capabilities Document,
which identifies past challenges with the department's requirements
processes. In addition, we analyzed the requirements traceability
matrix for the most recent version of AHLTA to determine the extent to
which bidirectional traceability had been performed. We also reviewed
program documentation relative to requirements development and user
community participation. In addition, we interviewed process
improvement officials, including the cognizant official from the
Office of the Chief Information Officer (CIO) about internal
acquisition process evaluations and their results and the status of
plans for improving acquisition management processes. We then compared
the department's current approach to requirements development and
management with best practices identified in the Software Engineering
Institute's Capability Maturity Model Integration for Acquisition.
* To assess the department's approach to improving user satisfaction,
we reviewed and analyzed program documentation pertaining to the
collection, analysis, and utilization of AHLTA user satisfaction
feedback such as seven survey reports and a postimplementation review
that were produced between 2005 and 2007 and compared the agency's
approach to best practices such as the Office of Management and
Budget's Capital Programming Guide and Standards and Guidelines for
Statistical Surveys. We also reviewed lessons learned reports from
2006 through 2008 and a user conference briefing from 2006 that
identified areas of user dissatisfaction. In addition, we reviewed
program office documents that identified improvement initiatives such
as the AHLTA 3.3 software release and the deployment of local cache
servers, which were intended to improve user satisfaction. We
supplemented our review by interviewing program officials, including
the DHIMS Program Manager and Deputy Program manager, to determine the
extent to which user satisfaction improvement efforts and initiatives
have been guided by documented plans. We then compared the
department's approach to improving user satisfaction with the Software
Engineering Institute's IDEALSM[Footnote 23] model, which is a
recognized approach for managing process improvement efforts such as
managing improvements to user satisfaction.
* To assess DOD's plans to improve acquisition management processes,
we reviewed documentation and interviewed officials from the DHIMS
Program Office and the Office of Process Improvement on their plans to
improve the processes based on the Software Engineering Institute's
Capability Maturity Model Integration for Acquisition. We also
reviewed DOD's 2008 internal assessment related to acquisition
management processes, action plans, and tasks planned for process
improvement.
We supplemented our analysis with interviews with officials in the
DHIMS Office, including, the Program Manager, Deputy Program Manager,
Director of Products Branch and Engineering and Resources offices. We
also obtained written responses from the responsible program manager
or subject matter expert for areas of our review. These responses were
approved by the MHS CIO or the Program Executive Officer, Joint
Medical Information Systems/Deputy MHS CIO.
We did not conduct an independent validation of the life-cycle costs
and obligations provided to us by DOD.
We conducted this performance audit at the DHIMS Program Office in
Falls Church, Virginia, and the National Naval Medical Center, in
Bethesda, Maryland, from September 2009 through October 2010 in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our objectives.
[End of section]
Appendix II: Comments from the Department of Defense:
Office Of The Assistant Secretary Of Defense:
Health Affairs:
TRICARE Management Activity:
Skyline Five, Suite 810, 5111 Leesburg Pike:
Falls Church, Virginia 22041-3206:
September 27, 2010:
Ms. Valerie C. Melvin:
Director, Information Management and Human Capital Issues:
U.S. Government Accountability Office:
441 G Street, N.W.
Washington, DC 20548:
Dear Ms. Melvin:
This is the Department of Defense (DoD) response to the
recommendations in the Government Accountability Office (GAO) Draft
Report GAO-11-50, "Information Technology ” Opportunities Exist to
Improve Management of DoD's Electronic Health Record Initiative,"
October 2010 (Engagement Code 310944).
DoD acknowledges receipt of the draft report and will address each of
the recommendations and ensure appropriate measures are carried out
effectively. DoD takes exception to several inaccurate, misleading,
and subjective statements provided in the draft report. GAO's
statements conflict with the extensive volume of programmatic
documentation, written responses, and consistent interview feedback
provided during the course of this audit. Enclosed are suggested
technical comments and corrections to GAO's draft report.
Thank you for the opportunity to review and comment on the draft
report. The points of contact for additional information are Ms. Lois
Kellett, Lois.Kellett@tma.osd.mil, or (703) 681-8836; and Mr. Gunther
Zimmerman, Gunther.Zimmerman@tma.osd.mil, or (703) 681-4360.
Sincerely,
Signed by:
George Peach Taylor, Jr., M.D.
Deputy Assistant Secretary of Defense (Force Health Protection and
Readiness):
Performing the Duties of the Assistant Secretary of Defense (Health
Affairs):
Attachments: As stated:
[End of letter]
GAO Draft Report-Dated October 2010:
GAO-11-50 (Engagement Code 310944):
"Opportunities Exist to Improve Management of Doll's Electronic Health
Record Initiative"
Department of Defense Comments to GAO Recommendations:
Recommendation: Develop and maintain a comprehensive project plan that
includes key elements, such as the project's scope, cost, schedule and
risks and update the plan to provide key information for stakeholders
on the project's plans and status.
DoD Response: Concur. DoD will develop and maintain a comprehensive
project plan in accordance with this recommendation and DoD
acquisition program guidelines.
Recommendation: Develop a systems engineering plan in accordance with
DoD guidance to address the technical complexities of delivering a
worldwide electronic health record system.
DoD Response: Concur. Since implementing a requirement to develop a
Systems Engineering Plan (SEP) in February 2004, DoD will continue to
develop and maintain a SEP in accordance with this recommendation.
Recommendation: Ensure that the requirements development process
involves system users throughout the development process, to obtain an
understanding of what will satisfy their needs.
DoD Response: Concur. DoD will continue to engage system users and
manage the requirements development process in accordance with this
recommendation.
Recommendation: Ensure the establishment of bidirectional traceability
for all system requirements.
DoD Response: Concur. DoD will ensure the bidirectional traceability
for requirements in accordance with this recommendation.
Recommendation: Develop and document a plan for improving user
satisfaction that prioritizes improvement projects; identifies needed
resources; includes schedules for improvement efforts, including
future user feedback survey; and links efforts to measureable outcomes
and specific user needs.
DoD Response: Concur. DoD will augment its current user feedback plan
for improved user satisfaction in accordance with this recommendation.
Recommendation: Establish acquisition management processes in
accordance with industry best practices, including indentifying
milestones and a completion date for the external evaluation that
MHS's processes are at maturity Level 2 of the Capability Management
Maturity Model Integrated for Acquisition.
DoD Response: Concur. DoD continues to follow DoD acquisition program
guidelines and will establish a milestone for an external review in
accordance with the Capability Management Maturity Model guidelines.
Recommendation: Further, to help ensure that the EHR Way Ahead does
not have shortfalls similar to those experienced with AHLTA, we
recommend that the above management practices be implemented as part
of the planning for this important initiative.
DoD Response: Concur. DoD will ensure that these recommendations are
implemented as part of the EIIR Way Ahead initiative.
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Valerie C. Melvin, (202) 512-6304 or melvinv@gao.gov:
Staff Acknowledgments:
In addition to the contact named above, Cynthia J. Scott (Assistant
Director); Harold Brumm, Jr.; Neil Doherty; Ronalynn Espedido; Rebecca
Eyler; Nancy Glover; Joel Grossman; Linda Kochersberger; Lee
McCracken; Madhav Panwar; Donald Sebers; Sylvia Shanks; Adam Vodraska;
Daniel Wexler; and Robert Williams, Jr. made key contributions to this
report.
[End of section]
Footnotes:
[1] Each of these systems provided certain patient-related
information. For example, the Ambulatory Data System captured certain
outpatient information relating to diagnosis and treatment; the
Preventive Health Care Application contained information on preventive
health services; and the Nutrition Management Information System
supported therapeutic nutrition therapy and medical food management.
[2] Open systems conform to industry standards so that commercial
products can easily be used and support costs can be minimized. A
client is usually a desktop computing device or program that is
"served" by one or more networked computing devices.
[3] Military hospitals currently use Essentris, a commercial-off-the-
shelf product, to document inpatient encounters that were originally
planned for Block 4. As of March 2010, inpatient functionality was
deployed at 29 sites, representing 62 percent of the Military Health
System's inpatient beds.
[4] We have identified DOD contracting in our high-risk list since
1992, and DOD business systems modernization as high risk since 1995;
however, we did not explicitly identify DOD's health care information
technology procurement processes as high-risk areas. See GAO, High-
Risk Series: An Update, [hyperlink,
http://www.gao.gov/products/GAO-09-271] (Washington, D.C.: Jan. 22,
2009).
[5] A fixed-price contract provides for a firm price or, in
appropriate cases, a ceiling or adjustable price. A time-and-materials
contract provides for acquiring supplies or services on the basis of
direct labor hours at specified fixed hourly rates that include wages,
overhead, general and administrative expenses, and profit and actual
cost of materials. A cost-reimbursement contract provides for payment
of allowable incurred costs, to the extent prescribed in the contract.
[6] The Federal Acquisition Regulation allows for contracts awarded
without full and open competition under certain circumstances and
requires written justification that addresses these circumstances.
[7] The noncompetitive contracts' costs are about 2 percent of
obligations of approximately $2 billion.
[8] Federal Acquisition Regulation, Part 16.505 (b) (2) i-iii.
[9] The defense acquisition system is a framework-based approach that
is intended to translate mission needs and requirements into stable,
affordable, and well-managed acquisition programs.
[10] AHLTA is assigned acquisition category IAM, which is the highest
information system acquisition category for IT Systems and is assigned
to acquisitions with at least $126 million in fiscal year 2000
constant dollars in development and deployment costs or at least $378
million in fiscal year 2000 constant dollars for all system costs.
[11] DOD Office of the Inspector General, Acquisition Management of
the Composite Health Care II Automated Information System, report
number 99-068 (January 21, 1999).
[12] DOD Office of the Inspector General, Information Technology
Management: Acquisition of the Armed Forces Health Longitudinal
Technology Application, report number D-2006-089 (May 18, 2006).
[13] GAO, Information Technology: Greater Use of Best Practices Can
Reduce Risks in Acquiring Defense Health Care System, [hyperlink,
http://www.gao.gov/products/GAO-02-345] (Washington, D.C.: Sept. 26,
2002).
[14] This effort included the AHLTA 3.3 software release discussed
above.
[15] In addition, DOD plans to spend $40 million on a related effort
to test the exchange of electronic health records with the Department
of Veterans Affairs and private health care providers and to work
toward a goal announced by President Obama on April 9, 2009, that the
departments would cooperate to create a joint virtual lifetime
electronic health record for service members and veterans.
[16] See Institute of Electrical and Electronics Engineers (IEEE),
IEEE/EIA Guide for Information Technology, IEEE/EIA 12207.1-1997
(April 1998) and Carnegie Mellon Software Engineering Institute,
Capability Maturity Model Integration for Acquisition, Version 1.2
(Pittsburgh, Pa, November 2007).
[17] GAO, Information Technology Investment Management: A Framework
for Assessing and Improving Process Maturity, [hyperlink,
http://www.gao.gov/products/GAO-04-394G] (Washington, D.C.: March
2004).
[18] Carnegie Mellon Software Engineering Institute, Capability
Maturity Model Integration for Acquisition, Version 1.2.
[19] According to DOD, the desirable target for AHLTA system
availability is 100 percent, meaning that the system is available to
users whenever it is needed, and the performance threshold is 99
percent, meaning that if availability falls below 99 percent,
performance is considered to be unacceptable. Further, a system
performance report for the time period October 2008 to February 2010
did not show any months with availability at the desired level of 100
percent, and only 1 month when it was available at the acceptable
level between 99 and 100 percent, and then only at the Army and Navy
facilities. The Air Force experienced the lowest levels of
availability, with 7 months that were between 93 and 97 percent
availability. The report showed that system downtime included some
system maintenance, but the primary cause of downtime was
implementation of improvements to address performance problems.
[20] See Federal Acquisition Regulation 39.102 and Carnegie Mellon,
Software Engineering Institute, Capability Maturity Model-Integration
for Development, Version 1.2 (Pittsburgh, Pa., August 2006) and
Software Acquisition Capability Maturity Model, Version 1.03, CMU/SEI-
2002-TR-010 (Pittsburgh, Pa., March 2002).
[21] The Software Engineering Institute is a federally funded research
and development center established at Carnegie Mellon University to
address software engineering practices. IDEALSM is a service mark of
Carnegie Mellon University and stands for initiating, diagnosing,
establishing, acting, and leveraging. For more information on this
model, see IDEALSM: A User's Guide for Software Process Improvement
(CMU/SEI-96-HB-001).
[22] The IEEE is a nonprofit, technical professional association that
develops standards for a broad range of global industries, including
the IT and information assurance industries and is a leading source
for defining best practices.
[23] The Software Engineering Institute is a federally funded research
and development center established at Carnegie Mellon University to
address software engineering practices. IDEAL is a service mark of
Carnegie Mellon University and stands for initiating, diagnosing,
establishing, acting, and leveraging. For more information on this
model, see IDEALSM: A User's Guide for Software Process Improvement
(CMU/SEI-96-HB-001).
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
