Information Technology
Centers for Medicare & Medicaid Services Needs to Establish Critical Investment Management Capabilities Gao ID: GAO-06-12 October 28, 2005To carry out its mission of ensuring health care security for beneficiaries, the Centers for Medicare & Medicaid Services (CMS) relies heavily on information technology (IT) systems. In fiscal year 2005, CMS's total IT appropriations was about $2.55 billion, of which about $760 million, or 30 percent, was to support internal investments, and $1.79 billion was to fund the Medicaid Management Information Systems (MMIS) that states use to support their Medicaid programs. (GAO is using the term "internal" to refer to all of CMS' IT investments excluding state MMISs.) In light of the size and significance of these investments, GAO's objectives were to (1) evaluate CMS's capabilities for managing its internal investments, (2) determine any plans the agency might have for improving these capabilities, and (3) examine CMS's process for approving and monitoring state MMISs.
Judged against GAO's framework for IT investment management, which measures the maturity of an organization's investment management process, CMS's capabilities for effectively managing its internal investments are limited. Specifically, the agency has established a little over half of the foundational practices it needs to manage individual investments and has executed 2 of the 27 key practices needed to manage investments as a portfolio. Until CMS fully establishes foundational and portfolio-level practices, executives will lack the assurance that they are managing the agency's collection of investments in a manner that minimizes risks and maximizes returns. CMS has initiated steps to improve its investment management process; however, these steps do not fully address the weaknesses GAO identifies in this report, nor are they coordinated with other needed improvement efforts into a plan that (1) is based on an assessment of strengths and weaknesses; (2) specifies measurable goals, objectives, and milestones; (3) specifies needed resources; (4) assigns clear responsibility and accountability for accomplishing tasks; and (5) is approved by senior-level management. Without such a plan and procedures for implementing it, CMS will be challenged in sustaining the commitment it needs to fully establish its investment management process. The process for approving requests for federal funding of MMIS activities (including development, operations, and maintenance activities) is characterized by standard procedures, guidance, and reported information to CMS's Center for Medicaid and State Operations. In contrast, the process for monitoring MMIS activities lacks standard procedures, guidance, and reporting requirements. Without these elements for monitoring MMIS activities, CMS may not be able to easily determine whether the state MMISs in which CMS invests close to $1.7 billion annually are facilitating the delivery of Medicaid benefits in the most effective and beneficial manner.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-06-12, Information Technology: Centers for Medicare & Medicaid Services Needs to Establish Critical Investment Management Capabilities
This is the accessible text file for GAO report number GAO-06-12
entitled 'Information Technology: Centers for Medicare & Medicaid
Services Needs to Establish Critical Investment Management
Capabilities' which was released on November 28, 2005.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Chairman, Committee on Finance, U.S. Senate:
October 2005:
Information Technology:
Centers for Medicare & Medicaid Services Needs to Establish Critical
Investment Management Capabilities:
GAO-06-12:
GAO Highlights:
Highlights of GAO-06-12, a report to the Chairman, Committee on
Finance, U.S. Senate:
Why GAO Did This Study:
To carry out its mission of ensuring health care security for
beneficiaries, the Centers for Medicare & Medicaid Services (CMS)
relies heavily on information technology (IT) systems. In fiscal year
2005, CMS‘s total IT appropriations was about $2.55 billion, of which
about $760 million, or 30 percent, was to support internal investments,
and $1.79 billion was to fund the Medicaid Management Information
Systems (MMIS) that states use to support their Medicaid programs. (GAO
is using the term ’internal“ to refer to all of CMS‘s IT investments
excluding state MMISs.) In light of the size and significance of these
investments, GAO‘s objectives were to (1) evaluate CMS‘s capabilities
for managing its internal investments, (2) determine any plans the
agency might have for improving these capabilities, and (3) examine
CMS‘s process for approving and monitoring state MMISs.
What GAO Found:
Judged against GAO‘s framework for IT investment management, which
measures the maturity of an organization‘s investment management
process, CMS‘s capabilities for effectively managing its internal
investments are limited. Specifically, the agency has established a
little over half of the foundational practices it needs to manage
individual investments (see figure below) and has executed 2 of the 27
key practices needed to manage investments as a portfolio. Until CMS
fully establishes foundational and portfolio-level practices,
executives will lack the assurance that they are managing the agency‘s
collection of investments in a manner that minimizes risks and
maximizes returns.
CMS has initiated steps to improve its investment management process;
however, these steps do not fully address the weaknesses GAO identifies
in this report, nor are they coordinated with other needed improvement
efforts into a plan that (1) is based on an assessment of strengths and
weaknesses; (2) specifies measurable goals, objectives, and milestones;
(3) specifies needed resources; (4) assigns clear responsibility and
accountability for accomplishing tasks; and (5) is approved by senior-
level management. Without such a plan and procedures for implementing
it, CMS will be challenged in sustaining the commitment it needs to
fully establish its investment management process.
The process for approving requests for federal funding of MMIS
activities (including development, operations, and maintenance
activities) is characterized by standard procedures, guidance, and
reported information to CMS‘s Center for Medicaid and State Operations.
In contrast, the process for monitoring MMIS activities lacks standard
procedures, guidance, and reporting requirements. Without these
elements for monitoring MMIS activities, CMS may not be able to easily
determine whether the state MMISs in which CMS invests close to $1.7
billion annually are facilitating the delivery of Medicaid benefits in
the most effective and beneficial manner.
Foundational Practices Implemented by CMS:
[See PDF for image]
[End of figure]
What GAO Recommends:
GAO recommends that the Secretary of Health and Human Services direct
CMS‘s Administrator to develop and implement a plan to (1) address the
IT investment management weaknesses identified in this report and (2)
take actions to better monitor MMISs. In response to a draft of this
report, CMS described actions under way and plans to address GAO‘s
recommendations.
www.gao.gov/cgi-bin/getrpt?GAO-06-12.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact David A. Powner, (202)
512-9286, pownerd@gao.gov, or Leslie G. Aronovitz, (312) 220-7600,
aronovitzl@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
CMS's Capabilities to Manage Its Internal Investments Are Limited:
CMS Does Not Have a Comprehensive Plan to Coordinate and Guide Its
Improvement Efforts:
Process for Monitoring MMISs Could Benefit from Standard Procedures,
Guidance, and Reporting Requirements:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendixes:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Comments from the Centers for Medicare & Medicaid
Services:
Appendix III: GAO Contacts and Staff Acknowledgments:
Tables:
Table 1: Stage 2 Critical Processes--Building the Investment
Foundation:
Table 2: Instituting the Investment Board:
Table 3: Meeting Business Needs:
Table 4: Selecting an Investment:
Table 5: Providing Investment Oversight:
Table 6: Capturing Investment Information:
Table 7: Stage 3 Critical Processes--Developing a Complete Investment
Portfolio:
Table 8: Frequency of Oversight Mechanisms Used by the 5 Regional
Offices Interviewed:
Figures:
Figure 1: Distribution of CMS's Information Technology Budget, Fiscal
Year 2005:
Figure 2: CMS Selection Process for Internal IT Investments:
Figure 3: The Five ITIM Stages of Maturity with Critical Processes:
Figure 4: Summary of Results for Stage 2 Critical Processes and Key
Practices for Internal IT Investments:
Abbreviations:
APD: advance planning document:
CIO: Chief Information Officer:
CMS: Centers for Medicare & Medicaid Services:
ESC: Executive Steering Committee:
HHS: Department of Health and Human Services:
IT: information technology:
ITIM: Information Technology Investment Management framework:
ITIRB: Information Technology Investment Review Board:
MMA: Medicare Prescription Drug, Improvement, and Modernization Act of
2003:
MMIS: Medicaid Management Information Systems:
Letter:
October 28, 2005:
The Honorable Charles E. Grassley:
Chairman, Committee on Finance:
United States Senate:
Dear Mr. Chairman:
The Centers for Medicare & Medicaid Services (CMS), formerly called the
Health Care Financing Administration, within the Department of Health
and Human Services (HHS), is responsible for overseeing the Medicare
and Medicaid programs. In 1990, we designated the Medicare program as
high-risk, in part, because of its sheer size and complexity.
Similarly, in 2003, we placed the Medicaid program on our high-risk
list, noting the growing concerns about the quality of fiscal
oversight. In our latest high-risk series, issued in January
2005,[Footnote 1] we continued to designate both these programs as high
risk. While the Medicare program is financed and administered by the
federal government, the Medicaid program is jointly financed by the
federal government and the states and is administered directly by the
states.[Footnote 2]
To carry out its responsibilities, CMS depends on hundreds of
information technology (IT) systems to maintain information on Medicare
beneficiaries, providers, and medical services provided as well as to
carry out its oversight of the states' Medicaid programs for low-income
Americans. For example, IT systems support the Medicare program, which
enrolls about 41 million elderly and disabled beneficiaries and, in
fiscal year 2004, had estimated outlays of $297 billion in health care
benefits. The agency also provides funding assistance (through grants)
to the states to develop and operate automated systems, known as
Medicaid Management Information Systems (MMIS), to support their
Medicaid programs.[Footnote 3] While the responsibility for managing
CMS's internal[Footnote 4]IT investments falls to its Information
Technology Investment Review Board, the responsibility for approving
requests for federal funding of state MMIS activities and for
monitoring these activities[Footnote 5] falls to CMS's Center for
Medicaid and State Operations and the 10 regional offices. For fiscal
year 2005, CMS's total IT appropriations was about $2.55 billion, of
which about $1.79 billion, or 70 percent, was to be used to support
Medicaid state IT investments.
This report is one of two we prepared in response to your request that
we review HHS's and CMS's IT management processes.[Footnote 6] It
focuses on CMS's processes for making IT investment management
decisions and evaluates how well these processes compare with the
accepted practices presented in our IT Investment Management
framework.[Footnote 7] This framework provides a method for evaluating
and assessing how well an agency is selecting and managing its IT
resources. As we agreed with your office, our objectives were to (1)
evaluate CMS's capabilities for managing its internal IT investments,
(2) determine any plans the agency might have for improving these
capabilities, and (3) examine CMS's processes for approving and
monitoring the state MMISs it funds. To address these objectives, we
analyzed documents and interviewed agency officials to (1) validate and
update CMS's self-assessment of key practices in the framework, (2)
evaluate the agency's plans for improving its capabilities, and (3)
examine CMS's processes for approving and monitoring the state MMISs.
We performed our work from January 2005 through September 2005 in
accordance with generally accepted government auditing standards.
Appendix I contains further details on our objectives, scope, and
methodology.
Results in Brief:
Judged against our framework for information technology investment
management, which measures the maturity of an organization's investment
management process, CMS's capabilities for effectively managing its
internal investments are limited. Specifically, CMS has established a
little over half of the foundational practices needed to manage its
internal investments individually and 2 of the 27 key practices
required to manage its investments as a portfolio--that is, an
integrated, agencywide collection of investments that are assessed and
managed collectively on the basis of common criteria. For example, CMS
has established most of the practices for capturing investment
information and many of the practices associated with instituting an
investment board. However, weaknesses remain in several areas.
Specifically:
* the agency's investment management guide does not reflect current
processes;
* procedures for selecting and reselecting investments are not fully
documented;
* procedures for involving the board in efforts to systematically
review the progress of IT projects and systems in meeting cost,
schedule, risk, and benefit expectations have not been defined; and:
* critical processes for defining portfolio criteria, creating the
portfolio, evaluating the portfolio, and conducting the
postimplementation reviews--necessary for portfolio management--have
not been implemented.
According to CMS officials, the agency's investment management
capabilities are limited because investment management has only
recently become an area of management focus. Until CMS implements all
of the key practices it needs to build the investment foundation and
manage its investments as a portfolio, executives cannot be assured
that they are selecting and managing the mix of investments that best
meets the agency's needs and priorities, or that its investment
decisions will result in the most effective support and minimized risk
for the multibillion-dollar Medicare and Medicaid programs.
CMS has initiated steps to improve its investment management process;
however, these steps do not fully address the weaknesses we identify in
this report, nor are they coordinated with other needed improvement
efforts into a plan that (1) is based on an assessment of strengths and
weaknesses; (2) specifies measurable goals, objectives, and milestones;
(3) specifies needed resources; (4) assigns clear responsibility and
accountability for accomplishing tasks; and (5) is approved by senior-
level management. Without such a plan and procedures for implementing
it, CMS will be challenged in sustaining the commitment it needs to
fully establish its investment management process.
In approving funding for MMISs that CMS jointly funds with the states,
regional office staff use standard procedures, rely on established
guidance, and are required to report on their approval activities to
CMS's Center for Medicaid and State Operations. In contrast, in
monitoring MMIS activities, regional office staff lack standard
procedures, guidance, and reporting requirements. Without these
elements for monitoring MMIS activities, CMS may not be able to easily
determine whether the state MMISs, in which CMS invests close to $1.7
billion annually, are facilitating the delivery of Medicaid benefits in
the most effective and beneficial manner.
To strengthen CMS's capability to manage its internal IT investments,
we are recommending that the Secretary for Health and Human Services
direct CMS's Administrator to develop and implement a plan aimed at
addressing the weaknesses identified in this report. We also are making
recommendations to improve CMS's process for monitoring the state MMISs
that it funds.
In commenting on a draft of this report, CMS provided information on
actions it is taking or plans to take to address our recommendations.
The agency, however, contended that many of the improvements to its IT
investment management process were not fully reflected in the report.
This is not accurate. The report sections in which we discuss the
implementation of specific key practices associated with critical
processes from our IT investment management framework each describe
CMS's efforts and accomplishments to improve its IT investment
management processes. In its written comments, CMS also took exception
with our recommendation for up-to-date, documented processes to ensure
consistency, and noted that the emphasis should be on strengthening
these processes first, and updating the documentation later. As we note
in the report, documenting processes does not preclude future revisions
or improvements to them, and provides a basis for consistent
implementation across the agency.
Background:
CMS has become the largest purchaser of health care in the United
States, serving nearly 83 million Medicare and Medicaid
beneficiaries.[Footnote 8] The agency administers the Medicare program,
enacted in 1965, which provides health insurance to people who are aged
65 years and over and to some people with disabilities who are under
aged 65 years. The agency also works with the states to administer the
Medicaid program, enacted in 1965 as a jointly funded program in which
the federal government matches state spending according to a formula to
provide medical and health-related services to low-income Americans.
In fiscal year 2005, CMS will reportedly spend about $519 billion: 63
percent for Medicare, 35 percent for Medicaid and Medicaid
administration, and the remaining 2 percent for the State Children's
Health Insurance Program and other administrative costs. CMS estimates
that its total budget in fiscal year 2006 will be $622 billion.
The agency carries out its responsibilities from its national
headquarters located in Baltimore, Maryland, and its 10 regional
offices located throughout the nation. It is organized around three
centers (to support its key functions): the Center for Medicare
Management, the Center for Beneficiary Choices, and the Center for
Medicaid and State Operations.[Footnote 9] Numerous other offices
throughout the agency support these centers.
CMS's Use of Information Technology:
IT systems play a vital role in helping CMS to fulfill its
responsibilities in carrying out the Medicare and Medicaid programs.
These systems help to maintain Medicare information on the millions of
beneficiaries, providers, and medical services provided. For example,
CMS's Medicare Fee-for-Service claims processing systems process more
than 1 billion claims annually and make benefit payments for the 41
million elderly and disabled beneficiaries. In fiscal year 2004, the
Medicare program had estimated outlays of $297 billion in health care
benefits.
Similarly, IT systems are relied on to manage the Medicaid program. In
fiscal year 2003, this program provided benefits totaling about $261
billion to nearly 54 million people. Of this amount, the federal share
was about $153 billion.
To assist the states in developing and operating MMISs used to process
Medicaid claims and administer the program, CMS provides funding
assistance through grants. In fiscal year 2005, about $1.79 billion, or
70 percent, of CMS's nearly $2.55 billion total appropriations for IT
went to support Medicaid state investments. The remaining approximately
$0.76 billion, or 30 percent, was used for CMS's internal investments.
Figure 1 shows the breakdown of this funding between CMS's internal IT
investments and Medicaid state IT investments.
Figure 1: Distribution of CMS's Information Technology Budget, Fiscal
Year 2005:
[See PDF for image]
[End of figure]
Weaknesses Previously Identified in CMS's IT Investment Management
Processes:
In September 2001,[Footnote 10] we reported that CMS's processes for
managing its IT investments omitted key review, approval, and
evaluation steps. We recognized that the agency was making efforts to
strengthen its IT planning and had developed guidance for an improved
management process, but stated that it would need to make considerable
progress in implementing these changes to ensure that its ongoing
modernization efforts stayed on track. To improve its investment
management processes, we made several recommendations to the CMS
Administrator, including establishing sufficient and written criteria
to ensure a consistent process for funding IT projects agencywide, and
establishing a systematic process for evaluating completed IT projects
that included cost, milestone, and performance data.
CMS's Approach to Investment Management:
Several groups and individuals play a role in CMS's process to manage
its internal IT investments, including an investment board for
establishing the IT investment governance principles. However, a
different process is used to oversee the Medicaid IT systems that the
agency jointly funds with the states. This process is carried out by
CMS's Center for Medicaid and State Operations and 10 regional offices.
Both of these processes, along with the roles and responsibilities of
the groups and individuals involved, are described below.
Process for Managing Internal Investments:
The groups and individuals who play a role in CMS's internal IT
investment management process include the Information Technology
Investment Review Board, Executive Steering Committees, Enterprise
Architecture Group, and Component Leads.
* Information Technology Investment Review Board (ITIRB). This board
was established in January of 2004 to provide a corporate perspective
in evaluating IT investments against CMS's business priorities. Its
members consist of senior leadership from CMS centers, offices, and
regional offices, and it is chaired by the agency's Chief Information
Officer (CIO). Initially, the primary ITIRB responsibility was
overseeing investments associated with the Medicare Prescription Drug,
Improvement, and Modernization Act of 2003 (MMA) and with CMS's
revitalization initiative.[Footnote 11] These investments made up about
one-third of CMS's fiscal year 2005 Operating Plan for internal
systems. In the spring of 2005, the role of the board was expanded to
include all internal IT investments. To assist the ITIRB in its
activities, CMS staff from the Office of Information Services and the
Office of Financial Management provide administrative support.
According to its charter, the board is responsible for:
* establishing the criteria for the selection, control, and evaluation
of CMS's portfolio of IT projects;
* developing the agency's IT operation plan and responding to the
President's budget request;
* reviewing the performance of IT investments using the criteria and
checkpoints in meeting cost, schedule, risk, and benefit expectations
and taking corrective actions when expectations are not being met;
* ensuring that IT investments in operation are periodically evaluated
to determine whether they should be retained, modified, replaced, or
terminated; and:
* comparing the results of implemented investments with the
expectations that were set for them and developing a set of lessons
learned for future process improvement.
* Executive Steering Committees (ESC). The ESCs were established to
support the ITIRB in carrying out its responsibilities. Each ESC is
responsible for managing IT projects (or investments) that are grouped
together into a portfolio for each of CMS's business components. This
responsibility includes maintaining the appropriate mix of IT
investments in its portfolio, managing the investments in its
portfolio, and providing funding recommendations to the ITIRB for these
investments. The membership of each ESC depends on the IT investments
contained in the portfolio, but, at a minimum, every CMS component that
sponsors a project is to have a representative on the ESC.
* Enterprise Architecture Group. This group, formally known as the IT
Architecture Planning Staff, supports the IT investment management
process, by, among other things, reviewing business case analyses for
new investments and major enhancements to ensure that they are
consistent with the enterprise architecture, by making recommendations
based upon that review that are aimed at the optimal leveraging of
assets.
* Component Leads. These individuals provide support in the IT
investment management process by serving as liaisons between the Office
of Information Services and individual project managers. Component
Leads are to assist project managers in understanding CMS's investment
management process and other operational policies and processes. They
can also provide project managers with key contacts for various IT
services that project owners may require during implementation of a
project.
In the spring of 2005, CMS implemented a new budget formulation process
and used it to select its IT investments. This process begins with an
information request from the CIO asking that each component submit
information on all of its investments, both new and ongoing. This
information is to include (1) a score sheet for each investment that
shows how it compared with prescribed criteria, such as alignment with
business drivers and IT strategic goals, and (2) a prioritized list of
all investments for the component. For new investments, the components
also are to submit an IT Fact Sheet (an investment proposal) that the
ITIRB support staff; the Enterprise Architecture Group; and,
ultimately, the board review to determine if the need for the new
investment is justified. If the need is found to be justified, project
managers receive funding to develop a Business Case Analysis (smaller
projects may not require such a document), which goes through the same
review process as the IT Fact Sheet.
The ITIRB support staff review all information submitted in response to
the information request and prepare it for the ESCs' review. The ESCs
reevaluate the investments against the criteria, making adjustments to
the scoring if necessary, and make funding recommendations to the
board. The ITIRB makes strategic and funding recommendations regarding
CMS's IT capital investment portfolio to CMS's Chief Operating Officer
who, in turn, provides recommendations to the CMS's Office of Financial
Management for integration into the agency's overall budget. Figure 2
illustrates CMS's process for selecting its internal IT investments.
Figure 2: CMS Selection Process for Internal IT Investments:
[See PDF for image]
[End of figure]
To date, the ITIRB's role in controlling (overseeing) IT investments
has been primarily limited to those associated with the MMA and
revitalization initiatives. According to CMS officials, efforts to
define procedures for the board to control all internal investments, in
accordance with the responsibilities described in its charter, are
currently under way.
Process for Approving and Monitoring State Medicaid IT Investments:
The ITIRB plays no role in approving and monitoring state Medicaid IT
investments. Instead, the process for approving states' requests for
matching funds for MMIS activities--including the design, development,
and installation of new MMISs, and the operations, maintenance, and
enhancement of existing MMISs--is the shared responsibility of CMS's
Center for Medicaid and State Operations (hereafter referred to as the
central office) and its 10 regional offices. According to
regulations,[Footnote 12] the State Medicaid Manual,[Footnote 13] and
officials we interviewed at CMS's central office and 5 regional
offices, CMS's process for approving states' requests generally
consists of the activities discussed below:
* To request federal funds for state MMIS activities, states must
prepare an advance planning document (APD), which identifies, among
other things, the purpose, scope, benefits, and preliminary cost
estimates for the activities they want to undertake. States submit this
document to the regional office, which reviews the APD for completeness
and technical content. Regional office staff generally ensure that
requests support the Medicaid program, are in compliance with federal
requirements, and represent cost-effective solutions. Also, the
regional office may have suggestions for the states to improve their
APDs. Some of the officials we interviewed told us that they work with
the states to complete the APDs to expedite the review and approval
process.
* Once regional office staff determine that an APD adequately justifies
the request for funding and the request is approved by that regional
office's Associate Regional Administrator for Medicaid, the CMS central
office and HHS are notified of the approval through a process referred
to as the Office of the Secretary Notice process.[Footnote 14] Once the
central office concurs, the regional office can send an approval letter
to the state.
* The states typically hire contractors to perform the MMIS activities.
With the approval of an APD, a state is given the clearance to develop
the request-for-proposals for soliciting contractor proposals. While
the APD is a high-level justification for funding, the request-for-
proposals is to contain the more detailed requirements of the MMIS
activities. Before it is issued, the request-for-proposals must be
approved by the CMS regional office through a process similar to that
used for the APD.
* The states review the proposals received and evaluate them in order
to make the final selection. While regional office staff do not
formally approve a state's evaluation process, they do review the
process to ensure that it allows for open and free competition, to the
maximum extent practicable.
* The states draft a contract for the MMIS activities. Prior to its
award, the contract is reviewed by regional office staff and approved
by the Associate Regional Administrator for Medicaid. The state then
makes an award to the contractor whose bid or offer is responsive to
the solicitation and most advantageous to the state--considering price,
quality, and other factors.[Footnote 15]
* When the contracted MMIS activities start, regional office staff
begin monitoring the status of these activities through a variety of
mechanisms, including reviews of status reports; on-site visits; and
meetings with external groups, such as industry associations, provider
groups, and vendors.
* Once MMISs are built and become operational, CMS establishes a team
consisting of headquarters and regional office staff with expertise in
relevant areas to do on-site reviews, referred to as certification
reviews. During these reviews, which are to be conducted about 6 months
after a system has been in operation, the team makes sure that the
system satisfies the terms of the state's APD, meets minimal federal
requirements, and complies with current regulations and policy. CMS has
written guidance for conducting these reviews, which it is in the
process of revising.[Footnote 16]
* Regional office staff are to continue monitoring MMIS activities
through the previously mentioned mechanisms.
Information Technology Investment Management Maturity Framework:
The Information Technology Investment Management (ITIM) framework is a
maturity model comprising five progressive stages of maturity that an
agency can achieve in its investment management capabilities.[Footnote
17] The ITIM framework was developed on the basis of our research into
the IT investment management practices of leading private-and public-
sector organizations. It identifies critical processes for making
successful IT investments, organized into the five increasingly mature
stages. These maturity stages are cumulative; that is, in order to
attain a higher stage of maturity, the agency must have
institutionalized all of the requirements for all of the lower stages
in addition to the higher stage.
The ITIM framework can be used to assess the maturity of an agency's
investment management processes and as a tool for organizational
improvement. The overriding purpose of the framework is to encourage
investment processes that increase business value and mission
performance, reduce risk, and increase accountability and transparency
in the decision process. We have used the framework in several of our
evaluations,[Footnote 18] and a number of agencies have adopted it.
These agencies have used ITIM for purposes ranging from self-assessment
to the redesign of their IT investment management processes.
The ITIM framework's five maturity stages represent steps toward
achieving stable and mature processes for managing IT investments. Each
stage builds on the lower stages; the successful attainment of each
stage leads to improvement in the organization's ability to manage its
investments. With the exception of the first stage, each maturity stage
is composed of "critical processes" that must be implemented and
institutionalized in order for the organization to achieve that stage.
These critical processes are further broken down into key practices
that describe the types of activities that an organization should be
performing to successfully implement each critical process. An
organization may be performing key practices from more than one
maturity stage at the same time. This is not unusual, but efforts to
improve investment management capabilities should focus on becoming
compliant with lower-stage practices before addressing higher-stage
practices.
Stage 2 of the ITIM framework encompasses building a sound investment
management process by establishing basic capabilities for selecting new
IT projects. It also involves developing the capability to control
projects so that they finish predictably within established cost and
schedule expectations and the capability to identify potential
exposures to risk and put in place strategies to mitigate that risk.
The basic selection processes established in Stage 2 lays the
foundation for more mature selection capabilities in Stage 3.
Stage 3 requires that an organization continually assess both proposed
and ongoing projects as parts of a complete investment portfolio--an
integrated and competing set of investment options. It focuses on
establishing a consistent, well-defined perspective on the IT
investment portfolio and maintaining mature, integrated selection (and
reselection), control, and evaluation processes that can be evaluated
during postimplementation reviews. This portfolio perspective allows
decision makers to consider the interaction among investments and the
contributions to organizational mission goals and strategies that could
be made by alternative portfolio selections, rather than focusing
exclusively on the balance between the costs and benefits of individual
investments. Organizations implementing Stages 2 and 3 have in place
the selection, control, and evaluation processes that are required by
the Clinger-Cohen Act of 1996.[Footnote 19]
Stages 4 and 5 require the use of evaluation techniques to continuously
improve both the investment portfolio and the investment processes in
order to better achieve strategic outcomes. At Stage 4 maturity, an
organization has the capacity to conduct IT succession activities and,
therefore, can plan and implement the deselection of obsolete, high-
risk, or low-value IT investments. An organization with Stage 5
maturity conducts proactive monitoring for breakthrough information
technologies that will enable it to change and improve its business
performance. Stages 4 and 5 define key attributes that are associated
with the most capable organizations.
Figure 3 shows the five ITIM stages of maturity and the critical
processes associated with each stage.
Figure 3: The Five ITIM Stages of Maturity with Critical Processes:
[See PDF for image]
[End of figure]
As defined by the model, each critical process consists of "key
practices" that must be executed to implement the critical process.
CMS's Capabilities to Manage Its Internal Investments Are Limited:
In order to have the capabilities to effectively manage IT investments,
an agency, at a minimum, should (1) build an investment foundation by
putting basic, project-level control and selection practices in place
(Stage 2 capabilities) and (2) manage its projects as a portfolio of
investments, treating them as an integrated package of competing
investment options and pursuing those that best meet the strategic
goals, objectives, and mission of the agency (Stage 3 capabilities).
CMS has executed 20 of the 38 key practices that are required to build
a foundation for IT investment management. In addition, because CMS has
focused primarily on establishing the Stage 2 practices, it has
executed only 2 of the 27 Stage 3 key practices. Until CMS implements
all of the key practices associated with building the investment
foundation and managing its investments as a portfolio, the agency will
not have much assurance that it has selected the mix of investments
that best supports its strategic goals, or that it will be able to
manage the investments to successful completion.
CMS Has Established about Half of the Foundational Practices for
Investment Management:
At the ITIM Stage 2 level of maturity, an organization has attained
repeatable, successful IT project-level investment control processes
and basic selection processes. Through these processes, the
organization can identify expectation gaps early and take the
appropriate steps to address them. According to the ITIM framework,
critical processes at Stage 2 include (1) defining IT investment
board[Footnote 20] operations, (2) identifying the business needs for
each IT investment, (3) developing a basic process for selecting new IT
proposals and reselecting ongoing investments, (4) developing project-
level investment control processes, and (5) collecting information
about existing investments to inform investment management decisions.
Table 1 describes the purpose of each of these Stage 2 critical
processes.
Table 1: Stage 2 Critical Processes--Building the Investment
Foundation:
Critical process: Instituting the investment board;
Purpose: To define and establish an appropriate information technology
(IT) investment management structure and the processes for selecting,
controlling, and evaluating IT investments.
Critical process: Meeting business needs;
Purpose: To ensure that IT projects and systems support the
organization's business needs and meet users' needs.
Critical process: Selecting an investment;
Purpose: To ensure that a well-defined and disciplined process is used
to select new IT proposals and reselect ongoing investments.
Critical process: Providing investment oversight;
Purpose: To review the progress of IT projects and systems, using
predefined criteria and checkpoints, in meeting cost, schedule, risk,
and benefit expectations and to take corrective action when these
expectations are not being met.
Critical process: Capturing investment information;
Purpose: To make available to decision makers information to evaluate
the impacts and opportunities created by proposed (or continuing) IT
investments.
Source: GAO.
[End of table]
Because IT investment management has only recently become an area of
management attention, CMS has put in place 20 of the 38 Stage 2 key
practices required for basic project-level selection and control. The
agency has satisfied the majority of the key practices associated with
establishing an IT investment review board, capturing investment
information, and meeting business needs. CMS also has recently
established a process for selecting investments, but it has not yet
established a process for the IT investment review board to provide
investment oversight. Figure 4 summarizes the status of CMS's critical
processes for Stage 2, showing how many key practices CMS has executed
in managing its internal IT investments.
Figure 4: Summary of Results for Stage 2 Critical Processes and Key
Practices for Internal IT Investments:
[See PDF for image]
[End of figure]
CMS Has an Investment Review Board, but its Investment Management
Process Guide Is Not Current:
The creation of decision-making bodies or boards is central to the IT
investment management process. At the Stage 2 level of maturity,
organizations define one or more boards, provide resources to support
their operations, and appoint members who have expertise in both
operational and technical aspects of the proposed investments. The
boards operate according to a written IT investment process guide that
is tailored to the organization's unique characteristics, thus ensuring
that consistent and effective management practices are implemented
across the organization. Once board members are selected, the
organization ensures that they are knowledgeable about policies and
procedures for managing investments. Organizations at the Stage 2 level
of maturity also take steps to ensure that executives and line managers
support and carry out the decisions of the investment board. According
to the ITIM framework, an IT investment management process guide should
(1) be a key authoritative document that the organization uses to
initiate and manage IT investment processes and (2) provide a
comprehensive foundation for policies and procedures developed for all
other related processes. (The complete list of key practices is
provided in table 2.)
CMS has executed 5 of the 8 key practices for this critical process.
For example, in January 2004, the agency established the ITIRB to
manage internal investments and provide business-driven leadership to
its operations and development. While the ITIRB was initially only
responsible for overseeing MMA and revitalization initiatives, its
responsibilities were expanded this past spring to include management
and oversight responsibilities for all internal investments. ITIRB
members are senior-level officials from both business and IT areas who
understand board policies and procedures.
The ITIRB is adequately resourced to maintain its operations. For
example, the Program Management and Support Group within the Office of
Information Services assists the board in such ways as coordinating and
integrating the investment management process. This group serves as the
principal contact and entry point for all new and proposed IT projects.
In addition, nine Executive Steering Committees were recently
established to support the work of the ITIRB by managing a subset of
investments grouped together according to business function. Their
responsibilities include, among other things, scoring and ranking IT
investments, and recommending investments to the ITIRB for funding.
Notwithstanding these strengths, CMS does not have an IT investment
process guide that reflects the agency's current investment management
practices. For example, the agency uses ESCs to work with the board on
specific areas of IT investments, but its process guide does not
identify this critical group. Moreover, the process guide does not
mention the agency's move to classify its IT investments in line with
the department's classification scheme. (The new classification scheme
consists of three levels in which projects are rated as major,
supporting, or tactical.) Instead, the process guide outlines a four-
level classification scheme that identifies investments as A, B, C, or
D, depending on the nature and sensitivity of the project. According to
CMS officials, the guide has not yet been updated because the agency
has made a priority of fully defining its processes before documenting
them. Documenting the process, however, does not preclude it from
future revisions or improvements, but does provide a basis for
consistent implementation across the agency. Until CMS's documented IT
investment process guidance is updated, executives are at risk of
inconsistently performing key investment decision-making activities and
inaccurately communicating management practices. Such updated guidance
would also provide a process that could lead to greater accountability
about future IT investment outcomes, which would be helpful to new
members joining the board.
Another key weakness is that CMS's ITIRB has not operated in accordance
with its assigned roles and responsibilities. For example, the ITIRB
has not yet been involved in systematically controlling investments nor
has it actively maintained the documented investment management
process. Until the ITIRB fully carries out its assigned roles and
responsibilities, executives will not have assurance that the whole IT
investment management process is functioning smoothly and effectively
as intended.
Table 2 shows the rating for each key practice required to implement
the critical process for instituting the investment board at the Stage
2 level of maturity. Each of the "executed" ratings shown below
represents instances where, on the basis of the evidence provided by
CMS officials, we concluded that the specific key practices were
executed by the agency.
Table 2: Instituting the Investment Board:
Type of practice: Organizational commitments;
Key practice: 1. An enterprisewide IT investment board composed of
senior executives from IT and business units is responsible for
defining and implementing the organization's IT investment governance
process;
Rating: Executed;
Summary of evidence: CMS has an enterprisewide IT investment board that
is responsible for defining and implementing the organization's IT
investment management process. The board consists of the agency's
senior leadership from CMS centers, offices, and regional offices and
is chaired by the Chief Information Officer.
Key practice: Type of practice: Organizational commitments;
2. The organization has a documented IT investment process directing
each investment board's operations;
Rating: Not executed;
Summary of evidence: Although CMS has a documented IT investment
management process guide, this guide has not been updated to reflect
current investment management processes. For example, CMS has
established several working groups supporting the investment management
process--for example, the Executive Steering Committees--which are not
identified in the IT investment management process guide. According to
officials, CMS plans to update its process guide in the near future.
Type of practice: Prerequisites;
Key practice: 1. Adequate resources, including people, funding, and
tools, are provided for supporting the operations of each IT investment
board;
Rating: Executed;
Summary of evidence: According to CMS officials, adequate resources are
provided to support board operations. For example, to support the work
of the ITIRB, the agency has established Executive Steering Committees.
In addition, the Planning, Management, and Support Group serves as the
principal contact and entry point for all new and proposed IT projects.
CMS also has an ITIRB support group to support the operations of the
ITIRB.
Type of practice: Prerequisites;
Key practice: 2. The board members understand the organization's IT
investment management policies and procedures and the tools and
techniques used in the board's decision-making process;
Rating: Executed;
Summary of evidence: ITIRB members are senior-level managers who
understand CMS's investment management policies and procedures as they
currently stand. Board members have also undertaken activities that
would contribute to their understanding of board policies and
procedures, including attending a 2-day retreat and monthly meetings.
Type of practice: Prerequisites;
Key practice: 3. Each board's span of authority and responsibility is
defined to minimize overlaps or gaps among the boards;
Rating: Executed;
Summary of evidence: CMS's enterprisewide investment board is
responsible for defining and implementing the investment management
process.
Type of practice: Activities;
Key practice: 1. The enterprisewide investment board has oversight
responsibilities for the development and maintenance of the
organization's documented IT investment process;
Rating: Not executed; Summary of evidence: Although the ITIRB has
responsibility for developing and maintaining the documented investment
management process, it has not been actively maintaining this process.
Type of practice: Activities;
Key practice: 2. Each investment board operates in accordance with its
assigned authority and responsibility;
Rating: Not executed;
Summary of evidence: CMS's enterprisewide investment board is not yet
fully carrying out the scope of its responsibilities. To date, board
members have selected investments for inclusion in the fiscal year 2007
budget, but the board has not yet been involved in systematically
controlling investments. In addition, the board has not been actively
maintaining the organization's documented IT investment management
process.
Type of practice: Activities;
Key practice: 3. The organization has established management controls
for ensuring that investment boards' decisions are carried out;
Rating: Executed;
Summary of evidence: The ITIRB develops the agency's operating plan,
and, according to officials, only investments listed in the operating
plan are funded.
Source: GAO.
[End of table]
CMS Has a Process for Ensuring That Projects Align with Business Needs
and Meet Users' Needs:
Defining business needs for each IT project helps to ensure that
projects and systems support the organization's business needs and meet
users' needs. This critical process ensures that a link exists between
the organization's business objectives and its IT management strategy.
According to the ITIM, effectively meeting business needs requires,
among other things, (1) documenting business needs with stated goals
and objectives, (2) identifying specific users and other beneficiaries
of IT projects and systems, (3) providing adequate resources to ensure
that projects and systems support the organization's business needs and
meet users' needs, and (4) periodically evaluating the alignment of IT
projects and systems with the organization's strategic goals and
objectives. (The complete list of key practices is provided in table
3.)
CMS has in place 5 of the 7 key practices for meeting business needs.
The agency's IT Investment Management Process Guide and Business Case
Analysis Development Guide require business needs for both proposed and
ongoing IT projects and systems to be identified in an IT fact sheet
and, in some instances, a business case analysis document. The agency
also has detailed procedures for developing these documents that call
for identifying users. We verified that the four projects we reviewed
identified specific users and also documented how the projects linked
back to CMS business needs.[Footnote 21] Resources for ensuring that IT
projects and systems support the organization's business needs and meet
users' needs include Component Leads, the Enterprise Architecture
Group, and detailed procedures and associated templates for developing
the IT fact sheet and business case analysis document.
Although CMS has performed most of the key practices associated with
meeting business needs, a few weaknesses remain. Specifically,
officials told us they rely on the HHS strategic plan to guide their
efforts because CMS's strategic plan documenting the agency's business
mission, goals, and objectives is outdated.[Footnote 22] However, the
primary tool used to justify funding for investments does not tie into
the HHS plan but provides high-level business drivers[Footnote 23] for
aligning these investments with business needs. While, according to
agency officials, these business drivers reflect a common understanding
of the agency's goals and objectives, they are not descriptive enough
to drive IT investments. Until CMS develops a current strategic plan or
other detailed statement of business mission with supporting goals and
objectives, the agency is at risk of not being able to thoroughly
communicate critical information on its goals and objectives or to
provide clear and transparent direction for its IT investment
management process.
Finally, CMS's budget formulation process serves as a mechanism to
reevaluate the alignment of projects and systems with the
organization's goals and objectives. However, the ITIRB selected
investments for the first time this past spring and, therefore, has not
yet had to reevaluate projects' and systems' alignment with
organizational goals and objectives. When CMS executes all key
practices associated with this critical process, it will have greater
assurance that its projects effectively meet the agency's business
needs.
Table 3 shows the rating for each key practice required to implement
the critical process for meeting business needs at the Stage 2 level of
maturity and summarizes the evidence that supports these ratings.
Table 3: Meeting Business Needs:
Type of practice: Organizational commitments;
Key practice: 1. The organization has documented policies and
procedures for identifying IT projects or systems that support the
organization's ongoing and future business needs;
Rating: Executed;
Summary of evidence: The IT Investment Management Process Guide and the
Business Case Analysis Development Guide both document procedures for
ensuring that IT projects and systems support the organization's
business needs.
Type of practice: Prerequisites;
Key practice: 1. The organization has a documented business mission
with stated goals and objectives;
Rating: Not executed;
Summary of evidence: CMS does not have an updated strategic plan or
other detailed statement of business mission with supporting goals and
objectives. Instead, the agency uses a list of business drivers to
align IT projects and systems with business needs. Although these
business drivers may reflect a common understanding of the agency's
business drivers, they are not descriptive enough to drive IT
investments.
Type of practice: Prerequisites;
Key practice: 2. Adequate resources, including people, funding, and
tools, are provided for ensuring that IT projects and systems support
the organization's business needs and meet users' needs;
Rating: Executed;
Summary of evidence: According to CMS officials, adequate resources
have been provided for ensuring that IT investment systems meet
business and users' needs. These resources include the Component Leads,
the Enterprise Architecture Group, and detailed procedures and
associated templates for developing the IT fact sheet and business case
analysis.
Type of practice: Activities;
Key practice: 1. The organization defines and documents business needs
for both proposed and ongoing IT projects and systems;
Rating: Executed;
Summary of evidence: CMS requires that all projects have an IT fact
sheet and, in some instances, a business case analysis. These two
documents identify the business needs for both proposed and ongoing IT
projects and systems. We verified that business needs were documented
for the four projects we reviewed.
Type of practice: Activities;
Key practice: 2. The organization identifies specific users and other
beneficiaries of IT projects and systems;
Rating: Executed;
Summary of evidence: CMS requires that users be identified in the
business case analysis and an IT fact sheet. We verified that specific
users were documented for the four projects we reviewed.
Type of practice: Activities;
Key practice: 3. Users participate in project management throughout an
IT project's or system's life cycle;
Rating: Executed;
Summary of evidence: CMS has procedures specifying the involvement of
users in project management throughout a project's life cycle. We
verified that for the four projects we reviewed, users participated in
project management throughout the projects' life cycles.
Type of practice: Activities;
Key practice: 4. The investment board periodically evaluates the
alignment of its IT projects and systems with the organization's
strategic goals and objectives and takes corrective actions when
misalignment occurs;
Rating: Not executed;
Summary of evidence: CMS's budget formulation process serves as a
mechanism to reevaluate the alignment of projects and systems with the
organization's goals and objectives. The ITIRB, however, selected
investments for the first time this past spring and, therefore, has not
yet had to reevaluate projects' and systems' alignment with
organizational goals and objectives.
Source: GAO.
[End of table]
CMS Has Processes to Select Investments, but Selection Criteria Do Not
Consider Critical Factors:
Selecting new IT proposals and reselecting ongoing investments require
a well-defined and disciplined process to provide the agency's
investment board, business units, and developers with a common
understanding of the process and the cost, benefit, schedule, and risk
criteria that will be used both to select new projects and to reselect
ongoing projects for continued funding. According to the ITIM, this
critical process requires, among other things, (1) making funding
decisions for new proposals according to an established process; (2)
providing adequate resources for investment selection activities; (3)
using a defined selection process to select new investments and
reselect ongoing investments; (4) establishing criteria for analyzing,
prioritizing, and selecting new IT investments and for reselecting
ongoing investments; and (5) creating a process for ensuring that the
criteria change as organizational objectives change. (The complete list
of key practices is provided in table 4.)
CMS has executed 4 of the 10 key practices associated with selecting an
investment. Specifically, CMS used a process it defined in February
2005--its budget formulation process--to select new investments and
reselect existing investments using a set of limited criteria. We
confirmed that the four projects we reviewed were reselected using this
new process. In addition, by using the budget formulation process to
select investments, executives had assurance that funding decisions
were aligned with selection decisions. Officials indicated that
adequate resources were provided for identifying and selecting
investments.
However, weaknesses remain in the selection area. Although CMS has a
number of documents that address investment selection and reselection,
these documents are not linked to provide a clear understanding of the
selection and reselection process. In addition, they do not define (1)
the roles and responsibilities for each participating unit involved in
the project selection process and (2) the decision-making procedures.
CMS officials told us they chose to first implement the selection
process and then go back to document it. Another key weakness in the
selection area is that, although selection and reselection criteria
have been defined, they do not include cost, benefit, schedule, and
risk factors. Officials indicated that because the Executive Steering
Committees and the ITIRB had a short amount of time to perform
selection activities this year, they defined a limited set of criteria
to evaluate projects. Further, CMS does not have a mechanism to ensure
that its selection criteria continue to reflect organizational
objectives.
Until CMS implements all of the key practices associated with selecting
investments, executives will not be adequately assured that they are
consistently and objectively selecting projects that meet the needs and
priorities of the agency in a cost-effective and risk-insured manner.
Table 4 shows the rating for each key practice required to implement
the critical process for selecting an investment at the Stage 2 level
of maturity and summarizes the evidence that supports these ratings.
Table 4: Selecting an Investment:
Type of practice: Organizational commitments;
Key practice: 1. The organization has documented policies and
procedures for selecting new IT proposals;
Rating: Not executed;
Summary of evidence: Although CMS has a number of documents that
address investment selection, they are not linked to provide a clear
understanding of the selection process. In addition, these documents do
not define the roles and responsibilities for each participating unit
involved in the project selection process, nor do they define the
decision-making procedures.
Type of practice: Organizational commitments;
Key practice: 2. The organization has documented policies and
procedures for reselecting ongoing IT investments;
Rating: Type of practice: Not executed;
Summary of evidence: Type of practice: Although CMS has a number of
documents that address investment reselection, they are not linked to
provide a clear understanding of the reselection process. In addition,
they do not define the roles and responsibilities for each
participating unit involved in the project reselection process, nor do
they define the decision-making procedures.
Type of practice: Organizational commitments;
Key practice: 3. The organization has documented policies and
procedures for integrating funding with the process of selecting an
investment;
Rating: Not executed;
Summary of evidence: Although the process used to formulate the budget
for the fiscal year 2006/2007 budget cycle integrated funding with
selection, there are no policies and procedures documenting this
integration.
Type of practice: Prerequisites;
Key practice: 1. Adequate resources, including people, funding, and
tools, are provided for identifying and selecting IT projects and
systems;
Rating: Executed;
Summary of evidence: According to the CMS Director of Investment
Tracking and Assessment, there were adequate resources to support
selection activities this year. For example, the Office of Financial
Management provided some staff resources, as did the Office of
Information Services.
Type of practice: Prerequisites;
Key practice: 2. Criteria for analyzing, prioritizing, and selecting
new IT investment opportunities have been established;
Rating: Type of practice: Not executed;
Summary of evidence: Type of practice: For the fiscal year 2006/2007
budget cycle, CMS's ITIRB developed and used criteria, including
alignment with IT strategic goals and primary business drivers, for the
selection process. However, these criteria did not include cost,
benefit, schedule, and risk factors.
Type of practice: Prerequisites;
Key practice: 3. Criteria for analyzing, prioritizing, and
reselecting[A] IT investment opportunities have been established;
Rating: Type of practice: Not executed;
Summary of evidence: Type of practice: For the fiscal year 2006/2007
budget cycle, CMS's ITIRB developed and used criteria, including
alignment with IT strategic goals and primary business drivers for the
reselection process. However, these criteria did not include cost,
benefit, schedule, and risk factors.
Type of practice: Prerequisites;
Key practice: 4. A mechanism exists to ensure that the criteria
continue to reflect organizational objectives;
Rating: Not executed;
Summary of evidence: CMS reported in its self- assessment that there
are no mechanisms to ensure that the selection criteria continue to
reflect organizational objectives.
Type of practice: Activities;
Key practice: 1. The organization uses its defined selection process,
including predefined selection criteria, to select new IT investments;
Rating: Executed;
Summary of evidence: This past spring, CMS used its defined selection
process, including a limited set of predefined selection criteria, to
select new IT investments.
Type of practice: Activities;
Key practice: 2. The organization uses its defined selection process,
including predefined selection criteria, to reselect[A] ongoing IT
investments;
Rating: Type of practice: Executed;
Summary of evidence: Type of practice: For the fiscal year 2006/2007
budget cycle, CMS began using a new budget formulation process,
including a limited set of predefined criteria, to reselect ongoing IT
investments. We verified that the four projects we reviewed were
reselected using this process.
Type of practice: Activities;
Key practice: 3. Executives' funding decisions are aligned with
selection decisions;
Rating: Executed;
Summary of evidence: Because CMS uses its budget formulation process to
select investments, executives' funding decisions are aligned with
selection decisions.
Source: GAO.
[A] According to the GAO Information Technology Investment Management
framework, "reselecting" is the periodic reconsideration of an
investment's continuing value to the organization and the decision to
continue funding. It is a recurring process that continues for as long
as a project is receiving funding.
[End of table]
CMS Has Not Defined Procedures for Management Oversight of IT Projects
and Systems:
An organization should provide effective oversight for its IT projects
throughout all phases of their life cycles. Its investment board should
maintain adequate oversight and observe each project's performance and
progress toward predefined cost and schedule expectations as well as
each project's anticipated benefits and risk exposure. The investment
board should also employ early warning systems that enable it to take
corrective action at the first sign of cost, schedule, or performance
slippages. This board has ultimate responsibility for the activities
within this critical process. According to the ITIM framework,
effective project oversight requires, among other things, (1) having
written policies and procedures for management oversight; (2)
developing and maintaining an approved management plan for each IT
project; (3) making up-to-date cost and schedule data for each project
available to the oversight boards; (4) having regular reviews by each
investment board of each project's performance against stated
expectations; and (5) ensuring that corrective actions for each
underperforming project are documented, agreed to, implemented, and
tracked until the desired outcome is achieved. (The complete list of
key practices is provided in table 5.)
CMS has only executed 1 of the 7 key practices associated with
effective project oversight. While CMS's IT Investment Management
Process Guide addresses management oversight of IT projects and
systems, it does not include specific procedures for the ITIRB's
oversight of IT projects and systems. In addition, while the board is
receiving performance data for some investments, including
revitalization investments, it is not yet performing oversight of
projects on a systematic basis. CMS officials indicated that the
ITIRB's involvement in overseeing investments to date has been limited
because the board was first focusing on selecting investments. However,
CMS recognizes the importance of the ITIRB's involvement in oversight
of IT investments, and, according to officials, the agency is currently
developing an approach to address this issue.
While CMS is in the process of developing a structured process for the
ITIRB to oversee investments, other entities are involved in the
oversight of projects. For example, performance information for one of
the projects we reviewed was not provided to CMS's ITIRB, but instead
was provided to senior-level management, such as the Chief Technology
Officer and directors from some CMS components. Until the ITIRB
systematically oversees CMS's investments, the oversight process will
not benefit from the corporate perspective that is gained by having an
enterprisewide board. As a result, executives may not be able to easily
determine the impact individual project decisions may have on other
projects and on the attainment of organizational goals and objectives.
Table 5 shows the rating for each key practice that is required to
implement the critical process for project oversight at the Stage 2
level of maturity and summarizes the evidence that supports these
ratings.
Table 5: Providing Investment Oversight:
Type of practice: Organizational commitment;
Key practice: 1. The organization has documented policies and
procedures for management oversight of IT projects and systems;
Rating: Not executed;
Summary of evidence: CMS's IT Investment Management Process Guide
addresses management oversight of IT projects and systems, but it does
not include specific procedures for the ITIRB's oversight of IT
projects and systems. According to CMS officials, these procedures are
currently being defined.
Type of practice: Prerequisites;
Key practice: 1. Adequate resources, including people, funding, and
tools, are provided for IT project oversight;
Rating: Not executed;
Summary of evidence: According to CMS officials, CMS does not have the
resources it needs to oversee IT projects and systems. For example,
they reported that additional skilled staff are needed to perform
oversight activities.
Type of practice: Prerequisites;
Key practice: 2. IT projects and systems, including those in steady
state (operations and maintenance), maintain approved project
management plans that include expected cost and schedule milestones and
measurable benefit and risk expectations;
Rating: Executed;
Summary of evidence: CMS IT projects and systems, including those in
operations and maintenance, maintain approved project management plans
that include cost, schedule, benefit, and risk expectations. We
verified that the case-study projects we reviewed maintained project
management plans that include these expectations.
Type of practice: Activities;
Key practice: 1. Data on actual performance, including cost, schedule,
benefit, and risk performance, are provided to the appropriate IT
investment board;
Rating: Not executed;
Summary of evidence: Although data on actual performance are being
provided to the ITIRB for some projects, there are no standard
procedures for involving the ITIRB in investment oversight. According
to CMS officials, these procedures are currently being determined.
Type of practice: Activities;
Key practice: 2. Using verified data, each investment board regularly
reviews the performance of IT projects and systems against stated
expectations;
Rating: Not executed;
Summary of evidence: The ITIRB has begun to review the performance of
some IT projects and systems against stated expectations. For example,
the ITIRB has recently begun to review the performance of the National
Plan and Provider Enumeration System. According to CMS officials,
procedures for the ITIRB to do this on a more systematic basis are
currently being determined.
Type of practice: Activities;
Key practice: 3. For each underperforming IT project or system,
appropriate actions are taken to correct or terminate the project or
system in accordance with defined criteria and the documented policies
and procedures for management oversight;
Rating: Type of practice: Not executed;
Summary of evidence: Type of practice: According to CMS officials,
procedures for involving the ITIRB in investment oversight, including
procedures for taking corrective actions, are currently being
determined.
Type of practice: Activities;
Key practice: 4. The investment board regularly tracks the
implementation of corrective actions for each underperforming project
until the actions are completed;
Rating: Not executed;
Summary of evidence: According to CMS officials, procedures for
involving the ITIRB in investment oversight, including procedures for
tracking the implementation of corrective actions for underperforming
projects, are currently being determined.
Source: GAO.
[End of table]
CMS Has a Collection of Information to Support Investment Management
Decisions:
To make good IT investment decisions, an organization must be able to
acquire pertinent information about each investment and store that
information in a retrievable format. During this critical process, an
organization identifies its IT assets and creates a comprehensive
repository of investment information. This repository provides
information to investment decision makers to help them evaluate the
impacts and opportunities that would be created by proposed or
continuing investments. It can provide insights and trends about major
IT cost and management drivers. The repository can take many forms and
does not have to be centrally located, but the collection method should
identify each IT investment and its associated components. This
critical process may be satisfied by the information contained in the
organization's current enterprise architecture, augmented by additional
information--such as financial information and information on risk and
benefits--that the investment board may require to ensure that informed
decisions are being made. According to the ITIM framework, effectively
managing this repository requires, among other things, (1) developing
written policies and procedures for identifying and collecting the
information, (2) assigning responsibility for ensuring that the
information being collected meets the needs of the investment
management process, (3) identifying IT projects and systems and
collecting relevant information to support decisions about them, and
(4) making the information easily accessible to decision makers and
others. (The complete list of key practices is provided in table 6.)
CMS has in place 5 of the 6 key practices associated with capturing
investment information. For example, CMS's IT Investment Management
Process Guide identifies specific information that is needed in the
investment management process, such as how each IT project relates to
the business needs of CMS. According to officials, adequate resources
are provided to support the collection of investment information, such
as the agency's IT Investment Tracking Database, and an individual
assigned responsibility for ensuring that the necessary information is
collected to meet the needs of the investment management process.
CMS is collecting specific information about IT investments to support
decisions about these investments, including projects' scores against
selection criteria and earned value management[Footnote 24]
information. We verified that this information was collected for the
four projects we reviewed. However, although the ITIRB has used
investment information to support selection decisions, it has not yet
used it to systematically oversee projects. According to CMS officials,
specific procedures for the ITIRB's oversight of IT projects and
systems are currently being defined.
Table 6 shows the rating for each key practice required to implement
the critical process for capturing investment information at the Stage
2 level of maturity and summarizes the evidence that supports these
ratings.
Table 6: Capturing Investment Information:
Type of practice: Organizational commitments;
Key practice: 1. The organization has documented policies and
procedures for identifying and collecting information about IT projects
and systems to support the investment management process;
Rating: Executed;
Summary of evidence: CMS's IT Investment Management Process Guide
defines procedures for identifying and collecting information in a
database to support the investment management process.
Type of practice: Organizational commitments;
Key practice: 2. An official is assigned responsibility for ensuring
that the information collected during project and systems
identification meets the needs of the investment management process;
Rating: Executed;
Summary of evidence: The director of CMS's Division of Investment
Analysis and Budget in the Planning, Management, and Support Group of
the CIO's office is responsible for ensuring that the information
collected about IT projects and systems meets the needs of the
investment management process.
Type of practice: Prerequisite;
Key practice: 1. Adequate resources, including people, funding, and
tools, are provided for identifying IT projects and systems and
collecting relevant investment information about them;
Rating: Executed;
Summary of evidence: According to CMS officials, there are adequate
resources in this area, including staff in the Planning, Management,
and Support Group of CMS's Office of Information Services and an IT
Investment Tracking Database.
Type of practice: Activities;
Key practice: 1. The organization's IT projects and systems are
identified, and specific information is collected to support decisions
about them;
Rating: Executed;
Summary of evidence: IT projects and systems are identified and
specific information is collected about them in an IT Investment
Tracking Database and Excel spreadsheets. We verified that information
about our four case-study projects was collected to support the
selection and control processes.
Type of practice: Activities;
Key practice: 2. The information that has been collected is easily
accessible and understandable to decision makers and others;
Rating: Type of practice: Executed;
Summary of evidence: Type of practice: CMS collects information about
IT projects and systems and makes it available to decision makers and
other stakeholders in various forms, such as in spreadsheets and
graphs. The director of CMS's Division of Investment Analysis and
Budget in the Planning, Management, and Support Group of the CIO's
office ensures that the ITIRB has all the relevant information for IT
investment decision making, and that it is in a format that the ITIRB
is able to easily use.
Type of practice: Activities;
Key practice: 3. The information repository is used by investment
decision makers and others to support investment management;
Rating: Not executed;
Summary of evidence: Although the board is using investment information
to support selection decisions, procedures have not yet been defined
for the board to use this information to support control decisions.
Source: GAO.
[End of table]
CMS Lacks the Key Capabilities Needed to Manage Its Investments as a
Portfolio:
During Stage 3, the investment board enhances the investment management
process by developing a complete investment portfolio. An investment
portfolio is an integrated, agencywide collection of investments that
are assessed and managed collectively on the basis of common criteria.
Managing investments within the context of such a portfolio is a
conscious, continuous, and proactive approach to expending limited
resources on an organization's competing initiatives in light of the
relative benefits expected from these investments. Taking an agencywide
perspective enables an organization to consider its investments
comprehensively, so that, collectively, the investments optimally
address the organization's missions, strategic goals, and objectives.
Managing investments with a portfolio approach also allows an
organization to determine priorities and make decisions about which
projects to fund, and continue to fund, on the basis of analyses of the
relative organizational value and risks of all projects, including
projects that are proposed, under development, and in operation. For an
organization to reap the full benefits of the portfolio process, it
should collect all of its investments into an enterprise-level
portfolio that is overseen by its senior investment board. Although
investments may initially be selected into subordinate portfolios--on
the basis of, for example, the lines of business or life-cycle stages-
--and managed by subordinate investment boards, they should ultimately
be aggregated into this enterprise-level portfolio.
According to our ITIM framework, critical processes performed by Stage
3 organizations include (1) defining the portfolio criteria, (2)
creating the portfolio, (3) evaluating the portfolio, and (4)
conducting postimplementation reviews.[Footnote 25] Table 7 shows the
purpose of each critical process in Stage 3.
Table 7: Stage 3 Critical Processes--Developing a Complete Investment
Portfolio:
Critical process: Defining the portfolio criteria;
Purpose: To ensure that the organization develops and maintains IT
portfolio selection criteria that support its mission, organizational
strategies, and business priorities.
Critical process: Creating the portfolio;
Purpose: To ensure that IT investments are analyzed according to the
organization's portfolio selection criteria, and that an optimal IT
investment portfolio with manageable risks and returns is selected and
funded.
Critical process: Evaluating the portfolio;
Purpose: To review the performance of the organization's investment
portfolio(s) at agreed- upon intervals, and to adjust the allocation of
resources among investments as necessary.
Critical process: Conducting postimplementation reviews;
Purpose: To compare the results of recently implemented investments
with the expectations that were set for them, and to develop a set of
lessons learned from these reviews.
Source: GAO.
[End of table]
CMS has executed very few key practices--2 of 27--associated with Stage
3 critical processes. Specifically, under the critical process for
defining the portfolio criteria, CMS provided evidence that it had
designated a working group to be responsible for developing and
modifying portfolio selection criteria. Under the critical process for
creating the portfolio, CMS provided evidence that it was capturing and
maintaining investment information for future reference. In its self-
assessment, the agency stated that it was not executing any other Stage
3 key practices. According to officials, CMS has not concentrated on
implementing Stage 3 key practices because the agency is first focusing
its resources on establishing the practices associated with Stage 2.
Until CMS fully implements the critical processes associated with
managing its investments as a complete portfolio, it will not have the
data or enterprisewide perspective it needs to make informed decisions
about its collection of investments.
CMS Does Not Have a Comprehensive Plan to Coordinate and Guide Its
Improvement Efforts:
CMS has initiated efforts to improve its investment management process.
While these efforts do not fully address any of the weaknesses we
identify in this report, they enhance the agency's ability to perform
key activities. Specifically:
* CMS has begun to implement a tool for capturing project information.
According to officials, the tool will bring together investment
information currently residing in various locations, including project
description information captured in its IT Investment Tracking
Database, information such as project scores collected to support
project selection activities, and earned value management data.
Although information to support investment decisions does not have to
be in one location, doing so will improve accessibility and facilitate
its use by decision makers.
* CMS recently established Executive Steering Committees to support the
ITIRB in carrying out its investment management responsibilities. These
groups played a key role in selecting investments for the fiscal year
2007 budget by reviewing investment information and making
recommendations for funding to the investment board. They are currently
determining procedures for overseeing investments. According to
officials, once procedures for the Executive Steering Committee
oversight have been determined, CMS plans to focus on defining
procedures for determining how and when to involve the investment board
in oversight--a key weakness identified in this report.
Although CMS has initiated these improvement efforts, it has not
coordinated them with the additional efforts needed to address the
weaknesses identified in this report in a comprehensive plan that (1)
specifies measurable goals, objectives, and milestones; (2) specifies
needed resources; (3) assigns clear responsibility and accountability
for accomplishing tasks; and (4) is approved by senior- level
management. We have previously reported that such a plan is
instrumental in helping agencies coordinate and guide improvement
efforts.
CMS officials recognize the value of having a comprehensive plan and
told us they have begun to develop one; however, a time frame for
completing the plan has not been established. Until CMS develops this
plan, the agency risks not being able to put in place an effective
management process that will provide appropriate executive-level
oversight for minimizing risks and maximizing returns.
Process for Monitoring MMISs Could Benefit from Standard Procedures,
Guidance, and Reporting Requirements:
As we previously noted, the responsibility for approving and monitoring
MMISs that CMS funds jointly with the states falls to CMS's central
office and its 10 regional offices, with the bulk of the activities
being performed by the regional offices. Although the process for
approving states' funding requests for MMIS activities is characterized
by (1) standard procedures performed consistently across the regional
offices, (2) guidance that staff can rely on in carrying out their
duties, and (3) requirements for reporting information to the central
office, the process for monitoring MMIS activities is not.
Standard Procedures, Guidance, and Reporting Requirements Exist for the
Approval Process:
The process for approving states' requests for federal funding of MMISs
is characterized by a defined set of activities that are performed
consistently across the regional offices. These activities include
regional office staff review and approval of the standard documentation
(i.e., the APDs, request-for-proposals, and contracts) that the states
prepare to justify their requests. Specifically, as we previously
described:
* States prepare an APD to request funding for MMISs. Regional office
staff review the document to ensure that states' requests support the
Medicaid program, are in compliance with federal requirements, and
represent cost-effective solutions. Once regional office staff
determine that the APD adequately justifies the request, they issue a
formal approval letter to the states (with concurrence from CMS's
central office).
* The request-for-proposals that the states prepare to solicit
contractor bids for MMIS activities, including development and
operations, is reviewed and approved by regional office staff through a
process similar to that used to approve the APDs.
* Regional office staff review the states' process for reviewing
contractors' proposed bids.
* Regional office staff review and approve the contract, after which
the state makes an award to the contractor whose bid or offer is
responsive to the solicitation and is most advantageous to the state--
considering price, quality, and other factors.
Regional office staff told us that they rely on the State Medicaid
Manual and the Code of Federal Regulation for guidance in performing
activities for approving states' requests for federal funding. Regional
staff are also required to inform the CMS central office of all
approval actions through the Office of the Secretary Notice process
previously mentioned.
Process for Monitoring State MMISs Lacks Standard Procedures, Guidance,
and Reporting Requirements:
In contrast to the approval process, the process for monitoring MMIS
activities lacks (1) standard procedures regional office staff must
perform to carry out their responsibilities, (2) guidance for staff to
rely on, and (3) requirements for staff to report on the results of
their monitoring efforts to the central office.
First, regional office staff use a variety of mechanisms to monitor
MMIS activities. These mechanisms include reviews of project status
reports; site visits; telephone calls; and meetings with external
groups, such as industry associations, provider groups, and vendors. In
addition, regional office staff determine if and when to use these
mechanisms. Table 8 shows the different mechanisms used by the regional
office staff we interviewed and the number of regional offices who used
them.
Table 8: Frequency of Oversight Mechanisms Used by the 5 Regional
Offices Interviewed:
Mechanism: Reviews of status reports;
Number of regional offices claiming use of mechanism: 5.
Mechanism: Site visits;
Number of regional offices claiming use of mechanism: 3.
Mechanism: Meetings with provider community;
Number of regional offices claiming use of mechanism: 2.
Mechanism: In-process reviews;
Number of regional offices claiming use of mechanism: 1.
Mechanism: Meeting with vendors;
Number of regional offices claiming use of mechanism: 1.
Mechanism: Participation in status meetings;
Number of regional offices claiming use of mechanism: 1.
Mechanism: Contact with state medical society;
Number of regional offices claiming use of mechanism: 1.
Mechanism: Assistance from National Account Representatives[A];
Number of regional offices claiming use of mechanism: 1.
Source: GAO.
[A] Philadelphia office regional staff told us their office staff
includes these representatives. They are responsible for staying
abreast of state Medicaid activities. In performing their work, they
communicate with the states' Medicaid director and perform at least two
visits a year to each state.
[End of table]
Second, CMS has no guidance for regional office staff to use in
monitoring MMIS activities. While CMS has a Regional Office Manual that
includes guidance for monitoring MMIS activities, this manual is not
used by regional office staff because, according to officials, it has
not been maintained throughout the years, and it no longer reflects
current processes.
Third, there are no requirements for regional office staff to report to
CMS's central office on their monitoring of states' federally funded
MMISs activities. Monthly teleconferences are conducted between the
central office and regional offices to discuss activities performed by
these offices, including activities to monitor state MMISs. According
to CMS officials, there is some communication outside of the scheduled
teleconferences to discuss any issues that might arise regarding the
status of state MMISs. In addition, according to officials, the
certification reviews performed about 6 months after the MMISs have
become operational provide opportunities to determine firsthand how
systems are performing. Despite these mechanisms, the central office
has no requirements for regional office staff to regularly report on
the results of their efforts to monitor MMIS activities.
According to CMS officials, the central office has traditionally placed
greater emphasis on the front-end approval of requests for federal
funding. The central office, however, now recognizes the need for and
value of adopting an approach for maintaining the visibility of MMISs
from beginning to end. To address this need, central office staff told
us that they plan to ask the regional offices to provide them with
quarterly reports on the status of MMIS activities in their states as
part of a broader effort that is currently under way to improve the
administration of the Medicaid program.[Footnote 26] Central office
staff stated this effort would also result in standard procedures and
guidance to support regional office staff's monitoring efforts. While
these activities would strengthen the monitoring process, during our
review central office staff did not yet have specific plans or time
frames for implementing them.
Until CMS defines standard procedures for monitoring MMIS activities,
guidance for staff to rely on, and reporting requirements, CMS's
central office may not be able to easily determine whether state MMISs
are facilitating the delivery of Medicaid benefits in the most
effective and beneficial manner.
Conclusions:
Because IT investment management has only recently become an area of
management focus, CMS capabilities to manage its internal investments
are limited. Specifically, the agency has established about half of the
practices for building the investment foundation, but few practices to
manage its investments as a portfolio. Although the foundational
practices have equipped CMS with the capabilities it needs to improve
its management of individual investments, the agency is hampered in its
ability to manage them as a portfolio because it has not implemented
the practices for doing so. Until CMS fully establishes the key
practices required to build the investment foundation and manage its
investments as a portfolio, it will not have the capabilities it needs
to ensure that investments supporting its multibillion-dollar Medicare
and Medicaid programs are being managed to minimize risks and maximize
returns.
Critical to CMS's success in going forward will be the development of
an implementation plan that (1) is based on an assessment of strengths
and weaknesses; (2) specifies measurable goals, objectives, and
milestones; (3) specifies needed resources; (4) assigns clear
responsibility and accountability for accomplishing tasks; and (5) is
approved by senior-level management. Although the agency has initiated
improvement efforts, it has not developed a comprehensive plan to guide
these and other efforts needed to improve its investment management
process. Without such a plan and procedures for implementing it, CMS
will be challenged in sustaining the commitment it needs to fully
establish its investment management process.
Finally, the process for approving states' funding requests for MMIS
activities is characterized by standard procedures that are performed
consistently across the regional offices, guidance, and requirements
for informing the central office of regional office staff activities.
The process for monitoring the development and operations of state
MMIS, on the other hand, has no standard procedures for regional office
staff, no guidance, and no requirement to report information to the
central office. Without these elements for monitoring MMIS activities,
CMS's central office may not be able to easily determine whether state
MMISs are facilitating the delivery of Medicaid benefits in the most
effective and beneficial manner.
Recommendations for Executive Action:
To strengthen CMS's capability to manage its internal IT investments
and address the related weaknesses addressed in this report, we
recommend that the Secretary of the Department of Health and Human
Services direct the CMS Administrator to develop and implement a plan
for improving CMS's IT investment management processes. The plan should
address the weaknesses described in this report. The plan should (1)
first focus on correcting the weaknesses in Stage 2 critical processes,
and next focus on the Stage 3 critical processes, and (2) at a minimum,
provide for accomplishing the following 12 actions:
In Stage 2:
* Update the agency's investment management guide to reflect current
investment management processes.
* Establish a process for the board to actively maintain the agency's
documented investment management process.
* Use an updated strategic plan or other detailed statement of business
mission with supporting goals and objectives to align investments with
business needs.
* Ensure that the board periodically evaluates the alignment of IT
projects and systems with strategic goals and objectives and take
corrective actions when misalignment occurs.
* Fully document procedures that address investment selection and
reselection and (1) provide a clear understanding of the selection and
reselection process, (2) define the roles and responsibilities for each
participating unit involved in the project reselection process, and (3)
define the decision-making procedures.
* Document procedures for integrating funding with investment
selection.
* Revise the ITIRB's selection and reselection criteria to include
cost, benefit, schedule, and risk factors, and establish a mechanism to
ensure these criteria continue to reflect organizational objectives.
* Define, document, and implement procedures for the ITIRB's oversight
of projects and systems.
* Implement processes to use investment information to fully support
investment management decisions.
In Stage 3:
* Implement the Stage 3 critical processes for defining portfolio
criteria, creating the portfolio, evaluating the portfolio, and
conducting postimplementation reviews, which are necessary for
portfolio management.
We also recommend that the Secretary for Health and Human Services
direct the CMS Administrator to ensure that the plan draw together
ongoing efforts and additional efforts that are needed to address the
weaknesses identified in this report. The plan should also (1) specify
measurable goals, objectives, and milestones; (2) specify needed
resources; (3) assign clear responsibility and accountability for
accomplishing tasks; and (4) be approved by senior-level management. In
implementing the plan, the Administrator should ensure that progress is
measured and reported periodically to the Secretary of Health and Human
Services.
To improve CMS's process for monitoring states' progress in developing
and maintaining Medicaid management information systems, we are
recommending that the Secretary of the Department of Health and Human
Services direct the CMS Administrator to take the following two
actions:
* Define standard procedures and supporting guidance for regional
offices to monitor MMIS activities.
* Require regional offices to regularly report on their MMIS monitoring
activities to CMS's central office.
Agency Comments and Our Evaluation:
The Administrator of CMS provided written comments on a draft of this
report (reprinted in app. II). In these comments, CMS identified
actions it is taking or plans to take to address our recommendations
and stated that effective management of IT investments is a critical
priority at the agency. CMS contended that many of the agency's
improvements to its IT investment management process were not fully
reflected in the report, and took exception with the need for up-to-
date, documented processes to ensure consistency.
Concerning our description of progress in implementing investment
management processes, CMS commented that the report indicates that the
agency has only established 2 out of 27 key practices needed to manage
investments as a portfolio. CMS stated that this is misleading since
the report also indicates that the agency has accomplished 20 of 38
foundational IT investment management practices. CMS also provided
examples of the practices it has implemented, such as establishing an
investment review board. In our report, we make a distinction between
foundational practices, which are the Stage 2 key practices for
establishing basic project-level selection and control capabilities,
and portfolio-level practices, which are the Stage 3 key practices for
managing investments as an integrated set of competing options. We also
note that both of these sets of key practices are needed to implement
the processes required by the Clinger-Cohen Act of 1996. On the basis
of this, we state that CMS does not have the full suite of capabilities
to manage its internal investments because it has only established a
little over half of the foundational practices and 2 of 27 portfolio-
level key practices, and we reiterate the need to fully establish both
sets of practices to increase assurance that executives are selecting
and managing the mix of investments that best meets the agency's needs
and priorities. In our report, the sections in which we discuss the
implementation of specific key practices associated with critical
processes from our IT investment management framework each describe
CMS's efforts and accomplishments to improve its IT investment
management processes. These include all of the examples of
accomplishments CMS provided in its comments.
In its comments, CMS took issue with our reporting that its IT
investment management guide did not reflect the current process, and
that its procedures for selecting and reselecting IT investments were
not fully documented. Although the agency fully agreed that an up-to-
date guide would constitute a piece of an effective process, it
commented that the emphasis should be on strengthening the process
first and updating documentation later. CMS made three points: (1) it
is not practical to publish an updated guide without having the
effective and repeatable underlying process in place and noted that it
is not provided the latitude to do this; (2) in the section of the
report discussing instituting the investment board, the noted
successful execution of key practices appears to be negated by the
statement that the investment management processes are not documented;
and (3) in the same section of the report, we are implying that an
updated guide would improve rather than explain the process. We
disagree with CMS that the process needs to be repeatable and
strengthened before it can be documented. Documented procedures could
actually serve to strengthen and improve the process by ensuring it is
performed consistently. Finally, we are not negating the successful
implementation of key practices to institute the investment board. We
are simply emphasizing the importance of having documentation to drive
the investment management process.
In its comments, CMS also noted actions it is taking to (1) develop a
plan to implement key practices in Stages 2 and 3; (2) revise existing
documentation to reflect processes in place that are not formally
documented; and (3) develop a plan that will be approved by senior
management that will incorporate goals, objectives, and milestones
required to further close the gaps between existing processes and our
ITIM framework. Regarding our recommendation to improve its process for
monitoring state MMIS activities and reporting to the central office,
CMS stated that it is developing standard procedures and supporting
guidance for the regional office(s) for monitoring these systems
activities and reporting to the central office. We agree with CMS that
these actions would address many of the weaknesses we identify in this
report.
CMS also provided some technical comments, which we have incorporated
into the report as appropriate.
As agreed with your offices, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days
from the date of this letter. At that time, we will send copies to
other interested congressional committees, the Secretary of Health and
Human Services, the CMS Administrator, the CMS Chief Information
Officer, and other interested parties. Copies will also be made
available at no charge on our Web site at [Hyperlink,
http://www.gao.gov]. If you have any questions on matters discussed in
this report, please contact David A. Powner at (202) 512- 9286, or at
[Hyperlink, pownerd@gao.gov], or Leslie G. Aronovitz at (312) 220-7600,
or at [Hyperlink, aronovitzl@gao.gov]. Contact points for our Offices
of Congressional Relations and Public Affairs may be found on the last
page of this report. GAO staff who made major contributions to this
report are listed in appendix III.
Sincerely yours,
Signed by:
David A. Powner:
Director, Information Technology Management Issues:
Signed by:
Leslie G. Aronovitz:
Director, Health Care:
[End of section]
Appendixes:
Appendix I: Objectives, Scope, and Methodology:
The objectives of our review were to (1) evaluate Centers for Medicare
& Medicaid Services (CMS) capabilities for managing its internal
information technology (IT) investments, (2) determine any plans the
agency might have for improving these capabilities, and (3) examine
CMS's process for approving and monitoring the state Medicaid
management systems it funds.
To address our first objective, we reviewed the results of the agency's
self-assessment of Stages 2 and 3 practices using our Information
Technology Investment Management framework (ITIM) and validated and
updated the results of the self-assessment through document reviews and
interviews with officials. We reviewed written policies, procedures,
and guidance and other documentation providing evidence of executed
practices, including CMS's IT Investment Management Process Guide,
CMS's Policy for IT Investment Management, and CMS's fiscal year
2006/2007 budget process. We also reviewed the CMS Information
Technology Investment Review Board (ITIRB) meeting minutes. We did not
assess CMS's progress in establishing the capabilities found in Stages
4 and 5 of the ITIM framework because CMS acknowledged that it had not
executed any of the key practices in these higher maturity stages. In
addition, we conducted interviews with officials from the Office of
Information Services who have responsibility for the development and
implementation of CMS's IT investment management process.
We compared the evidence collected from our document reviews and
interviews with the key practices in our ITIM framework. We rated the
key practices as "executed" on the basis of whether the agency
demonstrated (by providing evidence of performance) that it had met the
criteria of the key practice. A key practice was rated as "not
executed" when we found insufficient evidence of a practice being
executed or when we determined that there were significant weaknesses
in CMS's execution of the key practice. In addition, CMS was provided
with the opportunity to produce evidence for key practices rated as
"not executed."
As part of our analysis, we selected four CMS IT projects as case
studies to verify that the critical processes and key practices were
being applied. The projects were selected because they (1) supported
different functional areas, (2) were in various life-cycle phases, and
(3) required different levels of funding. The four projects are
described below:
* Healthcare Integrated General Ledger Accounting System--The project
is intended to standardize the collection, recording, and reporting of
Medicare financial information by contractors. It is to replace the
cumbersome ad hoc spreadsheets and "cuff" systems being used by
Medicare contractors to accumulate and report financial information to
CMS. The project's life-cycle cost is estimated at about $567 million.
* Medicare Claims Processing Redesign--This project is intended to
integrate and modernize the Common Working File system and Redesign and
the Medicare Shared Systems enterprise claims processing applications
and data systems. The modernization and unification of these systems is
to allow CMS to significantly enhance program capabilities, integrity,
performance, efficiencies, and maintainability; reduce program change
implementation time frames; improve accuracy, timeliness, and quality
of Medicare transaction processing; reduce system exposure to security
risks; and facilitate use of the Internet. The project's life-cycle
cost is estimated at nearly $494 million.
* Medicare Managed Care System--This project is intended to cover the
redesign of CMS's managed care family of systems, including the legacy
Group Health Plan system. It is to provide the platform for
implementing requirements under the MMA. The project is intended to
replace aging operations and to continue to support the agency's
managed care business needs until all functions are migrated to a new
system. Its life-cycle cost is estimated at about $111 million.
* National Plan and Provider Enumeration System--The project is
intended to implement a Health Insurance Portability and Accountability
Act requirement to issue a unique identifier to each covered health
care provider in the United States. It is expected to result in
administrative savings by simplifying a complicated, multifaceted
enumeration scheme, whereby a provider is issued different identifiers
for electronic transactions by each health plan with which it does
business, and sometimes multiple identifiers from a single plan. It
will impact several million providers and health plans in the nation.
The project's life-cycle cost is estimated at about $38 million.
For these projects, we reviewed project management documentation, such
as project plans, business cases, status reports, and documentation on
how these projects were selected by the ITIRB. We also interviewed the
project managers for these projects.
To address our second objective, we examined documentation on what
management actions had been taken and what initiatives had been planned
by the agency. This documentation included a requirements document for
a tool CMS is currently implementing that is to help the agency with IT
investment management, among other things. We also interviewed
officials from the Office of Information Services to determine what
efforts CMS had undertaken to improve IT investment management
processes.
To address our third objective, we reviewed documentation supporting
CMS's implementation of processes for (1) approving states' requests
for funding their Medicaid Management Information Systems (MMIS) and
(2) monitoring these MMISs, including related legislation, policy, and
implementing guidance. We also interviewed officials at CMS
headquarters and at the 5 CMS regional offices with the highest fiscal
year 2004 expenditures for administrative services, which includes
MMISs.
We conducted our work at CMS headquarters in Washington, D.C., and at 5
CMS regional offices located in New York, New York; Philadelphia,
Pennsylvania; Chicago, Illinois; San Francisco, California; and
Atlanta, Georgia, from January 2005 through September 2005 in
accordance with generally accepted government auditing standards.
[End of section]
Appendix II: Comments from the Centers for Medicare & Medicaid
Services:
Department Of Health & Human Services: Centers for Medicare & Medicaid
Services:
Administrator:
Washington, DC 20201:
October 14 2005:
Date:
TO: David A. Powner:
Director, Information Technology Management Issues:
Government Accountability Office:
FROM: Mark B. McClellan, MD., Ph.D.: Administrator:
SUBJECT: Government Accountability Office's (GAO) Draft Report:
INFORMATION TECHNOLOGY: Centers for Medicare & Medicaid Services Need
to Establish Critical Investment Management Capabilities (GAO-06-12):
Thank you for the opportunity to review and comment on the above GAO
draft report. The GAO evaluated the Centers for Medicare & Medicaid
Services' (CMS) capabilities for managing its information technology
investments, determined any plans CMS might have for improving
capabilities, and examined CMS' process for approving and monitoring
the State Medicaid management information systems.
Effective management of IT investments is a critical priority at CMS.
This includes many improvements to our IT investment process that were
not fully reflected in GAO's report. It is also reflected in strong
leadership at CMS. This year, Mr. Dean Mesterharm, former CIO of the
Social Security Administration, joined our senior management team with
the strategic objective of strengthening our IT investment and
management control. We have taken many steps to achieve this objective.
The report appears to indicate that CMS has just established 2 out of
27 key IT management practices. This is misleading, since according to
the report, CMS has already accomplished 20 of the 38 foundational IT
investment management practices. In addition, as noted in our response,
CMS has many further steps underway to advance IT investment and
management control.
We appreciate the time and effort that went into this GAO report.
Attached are the detailed comments to the GAO's recommendations and
concerns.
Centers for Medicare & Medicaid Services' (CMS) Comments to the
Government Accountability Office's (GAO) Draft Report: INFORMATION
TECHNOLOGY.-Centers for Medicare & Medicaid Services Need to Establish
Critical In vestment Management Capabilities (GAO-06-12):
GAO Concern:
The GAO asserts that because the process is not fully mature, CMS does
not have the capability to manage its internal investments. For example
on the cover page it states, "Until CMS fully establishes foundational
and portfolio-level practices, executives will lack the assurance that
they are managing the agency's collection of investments in a manner
that minimizes risks and maximizes returns." This thought is repeated
throughout the report.
CMS Response:
This is not an accurate description of CMS' IT investment management
program. Specifically, the report states that CMS has "executed 2 of
the 27 key practices needed to manage investments as a portfolio." This
is misleading since, as cited in the GAO report, CMS has established 20
of the 38 foundational IT investment management practices. These
include establishing an Information Technology Investment Review Board
(ITIRB) comprised of office directors from every business area in CMS.
This board is chaired by the CMS Chief Information Officer (CIO), who
previously served as the CIO of the Social Security Administration. For
Fiscal Years 2006 and 2007 IT investments, the ITIRB ranked and
prioritized CMS' IT investments against agency goals, and reviewed and
approved funding levels. As a result, CMS executives have increased
assurance that their IT investments are better aligned with agency
priorities.
GAO Concern:
GAO criticizes CMS in several places for not having an updated process
guide or other documentation and frequent calls for updating the guide
to reflect the current process. The lack of an updated guide leads to
three separate criticisms.
CMS Response:
First, on page 6, the first two bullets note that the Agency's
investment management guide does not reflect current processes and
procedures for selecting and reselecting investments that are not fully
documented. Although we fully understand the need for a process guide,
we do not feel it is practical to publish one without having the
effective and repeatable underlying process in place. It is
acknowledged in the section entitled, CMS's Approach to Investment
Management that our process is in flux, but we are not provided the
latitude to codify the new process before describing it. Based on
lessons learned, CMS likely will modify some activities carried out
last year in order to improve them.
Second, on page 23 the report states, "Notwithstanding these strengths,
CMS does not have an IT investment process guide which reflects the
agency's current investment management practices." This statement
implies that the successful execution of key practices cited earlier in
the section are negated because they are not yet documented.
Third, GAO implies that an updated guide would improve the process,
rather than the guide explaining the process. On pages 23-24 the report
states, "According to CMS officials, the guide has not yet been updated
because the agency has made a priority of fully defining its processes
before documenting them. Until CMS's documented IT investment process
guidance is updated, executives are at risk of performing key
investment decision-making activities inconsistently and communicating
management practices inaccurately. Such updated guidance would also
provide a process that could lead to greater accountability about
future IT investment outcomes that would be helpful to new members
joining the board."
We contend that a comprehensive and repeatable process, not a document,
ensures that executives are managing projects consistently. We fully
agree that an up-to-date guide would constitute a piece of an effective
process, but we feel that the emphasis should be on strengthening the
process first and updating documentation later.
GAO Recommendation:
Recommend that the Secretary of the Department of Health and Human
Services direct the CMS Administrator to develop and implement a plan
for improving CMS's IT investment management processes and to ensure
that the plan draw together ongoing efforts and additional efforts that
are needed to addresses the weaknesses identified in this report.
CMS Response:
CMS is already developing a plan to implement additional key practices
in Stages 2 and 3. CMS is to revising existing documentation to reflect
processes in place that are not formally documented. CMS will develop a
plan that will be approved by senior management that will incorporate
goals, objectives, and milestones required to further close the gaps
between existing processes and GAO's framework.
GAO Recommendation:
Recommend that the Secretary of the Department of Health and Human
Services direct the CMS Administrator to take the following actions:
* Define standard procedures and supporting guidance for regional
offices to monitor MMIS activities;
* Require regional offices to regularly report on their MMIS monitoring
activities to CMS's central office.
CMS Response:
The CMS is developing standard procedures and provide supporting
guidance to the regional office for monitoring the Medicaid management
information systems (MMIS) activities and reporting to CMS central
office.
The regional office manual was published in 1992 and has not been
updated. New guidelines for monitoring progress of State advance
planning document (APD) projects and periodic progress reporting to
central office is in process and scheduled for the 4th quarter of
fiscal year (FY) 2006. Plans are in place to incorporate new procedures
into the MMIS Certification Toolkit, currently under development. The
MMIS Certification project was undertaken in FY 2005 with contractor
assistance in order to transform the current certification process from
a single focal event occurring after the implementation of a new MMIS
to one that flows from initial planning, throughout design, development
and installation (DDI) of an MMIS. Included in the toolkit will be
checklists for communications regarding progress throughout the DDI
period. The State agency will provide progress reports to the regional
offices from approval of the original APD, the issuance of the request
for proposal, and the DDI activities until the systems are ready for
certification. In turn, progress reports will be submitted, from the
regions, to central office so that an annual aggregate report can be
produced summarizing activities across the regions. Periodic state
progress reports and regional office monitoring of approved ADP
activities and plans will be guided by the size, scope and complexity
of the activity and/or project. At a minimum state progress reports and
regional office monitoring will provide sufficient information to
support quarterly progress reports to central office.
[End of section]
Appendix III: GAO Contacts and Staff Acknowledgments:
GAO Contacts:
David A. Powner, (202) 512-9286, [Hyperlink, pownerd@gao.gov]
ownerd@gao.gov Leslie G. Aronovitz, (312) 220-7600, [Hyperlink,
aronovitzl@gao.gov]:
Staff Acknowledgments:
In addition to the persons named above, William G. Barrick, Shaunessye
Curry, Mary Beth McClanahan, Sabine R. Paul, and Amos Tevelow made key
contributions to this report.
(310480):
FOOTNOTES
[1] GAO, High-Risk Series: An Update, GAO-05-207 (Washington, D.C.:
January 2005).
[2] Medicaid consists of 56 distinct state-level programs, including 1
for each of the 50 states; the District of Columbia; Puerto Rico; and
the U.S. territories of American Samoa, Guam, the Northern Mariana
Islands, and the Virgin Islands. Hereafter, these 56 entities are
referred to as states. Within broad federal guidelines, each program
establishes its own eligibility standards; determines the type, amount,
duration, and scope of covered services; and sets payment rates.
[3] The Medicaid Management Information System is the primary claims
processing and information retrieval system, which states are required
to have.
[4] We are using the term "internal" to refer to all of CMS's IT
investments, excluding the state MMISs. Internal investments include
Medicare claims processing systems used by contractors.
[5] States request funding for the design, development, and
installation of a new MMIS or for the operations and maintenance of or
enhancement to an existing MMIS.
[6] Our second report, Information Technology: HHS Has Several
Investment Management Capabilities in Place, but Needs to Address Key
Weaknesses, GAO-06-11 (Washington, D.C.: Oct. 28, 2005), addresses
HHS's (1) capabilities for managing its IT investments and (2) plans
for improving those capabilities.
[7] GAO, Information Technology Investment Management: A Framework for
Assessing and Improving Process Maturity, GAO-04-394G (Washington,
D.C.: March 2004).
[8] Of these nearly 83 million beneficiaries, more than 6 million are
children covered by the State Children's Health Insurance Program.
[9] The Center for Medicare Management is responsible for the Medicare
Fee-for-Service program. The Center for Beneficiary Choices is
responsible for Medicare's managed care program and also focuses on
beneficiary educational efforts. The Center for Medicaid and State
Operations focuses on programs administered by the states, such as
Medicaid.
[10] GAO, Medicare: Information Systems Modernization Needs Stronger
Management and Support, GAO-01-824 (Washington, D.C.: Sept. 20, 2001).
[11] The Medicare Prescription Drug, Improvement, and Modernization Act
of 2003, Pub. L. No. 108-173, is to provide seniors and individuals
with disabilities with prescription drug benefits, more choices, and
better benefits under Medicare. CMS's revitalization initiative is the
agency's effort to address long-term IT issues.
[12] Medicaid regulations are in 42 C.F.R. Ch. IV. Regulations
pertaining to the advance planning document process are set forth at 45
C.F.R. Part 95.
[13] The State Medicaid Manual provides instructions, regulatory
citations, and information for carrying out the Medicaid program.
[14] The Office of the Secretary Notices are one-page summaries of the
reviews performed by CMS regional office staff of documentation (APDs,
request-for-proposals, and contracts) submitted by a state for MMIS
funding assistance. The summaries are submitted to CMS's Center for
Medicaid and State Operations, which must review and "clear" them
before the regional office can release the official approval letter to
the state.
[15] 45 C.F.R. 95.613(b) and 45 C.F.R. 74.43.
[16] CMS's certification guidance is defined in the agency's Medicaid
Management Information System Certification Review Protocol.
[17] GAO-04-394G.
[18] GAO, Information Technology: DLA Needs to Strengthen Its
Investment Management Capability, GAO-02-314 (Washington, D.C.: Mar.
15, 2002); United States Postal Service: Opportunities to Strengthen IT
Investment Management Capabilities, GAO-03-3 (Washington D.C.: Oct. 15,
2002); Information Technology: Departmental Leadership Crucial to
Success of Investment Reforms at Interior, GAO-03-1028 (Washington,
D.C.: Sept. 12, 2003); Bureau of Land Management: Plan Needed to
Sustain Progress in Establishing IT Investment Management Capabilities,
GAO-03-1025 (Washington, D.C.: Sept. 12, 2003); and Information
Technology: FAA Has Many Investment Management Capabilities in Place,
but More Oversight of Operational Systems Is Needed, GAO-04-822
(Washington, D.C.: Aug. 20, 2004).
[19] 40 U.S.C. § 11312(b)(1).
[20] An IT investment board is a decision-making body--made up of
senior program, financial, and information managers--that is
responsible for making decisions about IT projects and systems on the
basis of comparisons and trade-offs among competing projects, with an
emphasis on meeting mission goals.
[21] The four projects we reviewed--Healthcare Integrated General
Ledger Accounting System, Medicare Claims Processing Redesign, Medicare
Managed Care System, and National Plan and Provider Enumeration System-
-are described in appendix I.
[22] According to the Director of Investment Tracking and Assessment,
the strategic plan has not been updated because of turnover in upper-
level management.
[23] The tool lists the following four business drivers: (1)
beneficiary health and satisfaction, (2) efficiency and integrity of
operations, (3) health care delivery, and (4) health care quality.
[24] "Earned value management" is a project management tool that
integrates the investment scope of work with schedule and cost elements
for investment planning and control. This method compares the value of
work accomplished during a given period with the value of the work
expected in the period. Differences in expectations are measured in
both cost and schedule variances.
[25] The purpose of a postimplementation review is to evaluate an
investment after its development has been completed (i.e., after its
transition from the implementation phase to the in-service management
phase) in order to validate actual investment results. This review is
conducted to (1) examine differences between estimated and actual
investment costs and benefits and their possible ramifications for
unplanned funding needs in the future and (2) extract "lessons learned"
about the investment selection and control processes that can be used
as the basis for management improvements. Similarly, postimplementation
reviews should be conducted for investment projects that were
terminated before completion, to help to readily identify potential
management and process improvements.
[26] This effort, known as the "Medicaid Information Technology
Architecture initiative," involves the development of a framework of
enabling technologies and processes intended to improve the
administration of the Medicaid program. CMS expects to complete this
initiative within the next 2 years.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
