Information Technology
HHS Has Several Investment Management Capabilities in Place, but Needs to Address Key Weaknesses Gao ID: GAO-06-11 October 28, 2005The Department of Health and Human Services (HHS) is one of the largest federal agencies, the nation's largest health insurer, and the largest grant- making agency in the federal government. The department manages over 300 programs that serve to improve the health and well-being of the American public and is comprised of several component agencies covering a wide range of activities including conducting and sponsoring medical and social science research, guarding against the outbreak of infectious diseases, assuring the safety of food and drugs, and providing health care services and insurance. It also manages and funds a variety of information technology (IT) initiatives ranging from those facilitating the payment of claims for Medicare and Medicaid services to those supporting health surveillance and communications. In fiscal year 2006, the department plans to spend over $5 billion on information technology--the third largest IT expenditure in the federal budget. As we agreed with Congress, our objectives were to (1) assess the department's capabilities for managing its IT investments and (2)determine any plans the department might have for improving those capabilities. To address these objectives, we analyzed documents and interviewed agency officials to (1)validate and update HHS's self-assessments of key practices in the framework and (2)evaluate HHS's plans for improving its capabilities.
Because of the management attention that has been given to IT investment management, HHS has established over half of the foundational practices needed to manage its IT investments individually and about 30 percent of the key practices needed to effectively manage its portfolio of investments. For example, HHS has implemented many of the practices required to ensure that (1) projects support business needs and meet users' requirements, (2) a well-defined and disciplined process is used to select IT investments, (3) investment information is captured in a repository for decision makers, and (4) IT portfolio selection criteria are developed and maintained. However, critical weaknesses remain in several areas. Specifically, HHS lacks: (1) business representation on its senior IT investment review board of component agencies to carry out its full scope of responsibilities, (2) an established process for the IT investment board to regularly review a defined set of the component agencies' IT investments and maintain visibility of other investments, (3) criteria for assessing portfolio performance or regularly reviewing the performance of the organization's investment portfolio, and (4) processes for conducting post-implementation reviews (PIR) of its IT investments. The department also does not have a structured mechanism in place for ensuring that component agencies define and implement investment management processes that are aligned with those of the department. Until the department fully establishes all foundational and portfolio-level practices and establishes a mechanism to ensure that component agencies define and implement processes that are aligned with those of the department, executives cannot be assured that they are appropriately selecting, managing, and evaluating the mix of investments that will maximize returns to the organization, taking into account the appropriate level of risk. HHS has initiated steps to improve its investment management process; however, these steps do not fully address the weaknesses we identify in this report, nor are they coordinated along with other needed improvement efforts into a plan that (1) is based on an assessment of strengths and weaknesses; (2) specifies measurable goals, objectives, and milestones; (3) specifies needed resources; (4) assigns clear responsibility and accountability for accomplishing tasks; and (5) is approved by senior management. Without such a plan and procedures for implementing it, the department risks being unable to effectively establish mature investment management capabilities. As a result, executives may not be able to make informed and prudent investment decisions in managing the department's annual multibillion-dollar IT budget.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-06-11, Information Technology: HHS Has Several Investment Management Capabilities in Place, but Needs to Address Key Weaknesses
This is the accessible text file for GAO report number GAO-06-11
entitled 'Information Technology: HHS Has Several Investment Management
Capabilities in Place, but Needs to Address Key Weaknesses' which was
released on November 28, 2005.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Chairman, Committee on Finance, U.S. Senate:
October 2005:
Information Technology:
HHS Has Several Investment Management Capabilities in Place, but Needs
to Address Key Weaknesses:
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-11]:
GAO Highlights:
Highlights of GAO-06-11, a report to the Chairman, Committee on
Finance, U.S. Senate:
Why GAO Did This Study:
The Department of Health and Human Services (HHS) is one of the largest
federal agencies, the nation‘s largest health insurer, and the largest
grant-making agency in the federal government. The department manages
over 300 programs that serve to improve the health and well-being of
the American public. To support these programs, the department funds
numerous information technology (IT); in fiscal year 2006, it plans to
spend over $5 billion on IT. GAO was asked to evaluate HHS‘s processes
for making IT investment management decisions. Specifically, the
objectives of this review were to (1) assess the department‘s
capabilities for managing its IT investments and (2) determine what
plans, if any, the department might have for improving those
capabilities.
What GAO Found:
Judged against the criteria of GAO‘s framework for information
technology investment management (ITIM), which measures the maturity of
an organization‘s investment management processes, HHS has established
63 percent of the foundational practices that it needs to manage its IT
investments individually; and 30 percent to manage its investments as a
portfolio (see table below). Specifically, HHS has implemented
processes to ensure that projects support business needs and meet
users‘ requirements, established a process for selecting investments,
and has created portfolio selection criteria. However, weaknesses
remain in several areas. The department‘s senior investment board does
not regularly review component agencies‘ IT investments, leaving close
to 90 percent of its discretionary investments without an appropriate
level of executive oversight. In addition, HHS does not evaluate the
performance of its portfolio on a continuing basis or conduct
postimplementation reviews. Finally, HHS currently has no structured
mechanism in place to ensure that the component agencies are defining
and implementing investment processes that are aligned with those of
the department. Until HHS establishes the practices it needs to
effectively manage its IT investments, executives cannot be assured
that they are appropriately selecting, managing, and evaluating the mix
of investments that will maximize returns to the organization, taking
into account the appropriate level of risk.
HHS has initiated efforts to improve its investment management
processes, but has not coordinated these and additional efforts that
would be needed to address the weaknesses we identify in a
comprehensive plan that defines and prioritizes improvements to the
investment process. Such a plan is instrumental in helping HHS to
coordinate and guide its improvement efforts and sustain its commitment
to the efforts already under way. Without such a plan and procedures
for implementing it, the department risks being unable to effectively
establish mature investment management capabilities. As a result,
executives may not be able to make informed and prudent investment
decisions in managing HHS‘s multibillion-dollar IT budget.
HHS‘s Current IT Investment Management Capabilities:
Stage 2: Building the investment foundation
Percentage of key practices executed:
Stage 2: Building the investment foundation
Instituting the investment board;
Percentage of key practices executed: 63
Stage 2: Building the investment foundation
Meeting business needs;
Percentage of key practices executed: 100.
Stage 2: Building the investment foundation
Selecting an investment;
Percentage of key practices executed: 70.
Stage 2: Building the investment foundation
Providing investment oversight;
Percentage of key practices executed: 0.
Stage 2: Building the investment foundation
Capturing investment information;
Percentage of key practices executed: 83.
Stage 2: Building the investment foundation
Overall Percentage of key practices executed: 63.
Stage 3: Developing a complete investment portfolio
Conducting postimplementation reviews;
Percentage of key practices executed: 0.
Stage 3: Developing a complete investment portfolio
Evaluating the portfolio;
Percentage of key practices executed: 0.
Stage 3: Developing a complete investment portfolio
Creating the portfolio;
Percentage of key practices executed: 43.
Stage 3: Developing a complete investment portfolio
Defining the portfolio criteria;
Percentage of key practices executed: 71.
Stage 3: Developing a complete investment portfolio
Overall Percentage of key practices executed: 30.
Source: GAO.
[End of Table]
What GAO Recommends:
To strengthen HHS‘s investment management capability, GAO recommends
that HHS develop and implement a plan to address the weaknesses
identified in this report. In written comments on a draft of this
report, HHS generally agreed with our findings and recommendations and
stated that it will leverage the report in its continuing efforts to
improve its investment management processes.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-11]
To view the full product, including the scope and methodology, click on
the link above. For more information, contact David Powner, 202-512-
9286, pownerd@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
HHS Has Established Many Key Practices for Managing Its Investments,
but Has Provided Limited Guidance and Oversight to Component Agencies
Processes:
HHS Does Not Have a Plan to Coordinate and Guide Improvement Efforts:
Conclusions:
Recommendations for Executive Action:
Agency Comments:
Appendixes:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Comments from the Department of Health and Human Services:
Appendix III: GAO Contact and Staff Acknowledgments:
Tables Tables:
Table 1: Estimated IT Budget of HHS Component Agencies for Fiscal Year
2006:
Table 2: Stage 2 Critical Processes--Building the Investment
Foundation:
Table 3: Summary of Results for Stage 2 Critical Processes and Key
Practices:
Table 4: Instituting the Investment Board:
Table 5: Meeting Business Needs:
Table 6: Selecting an Investment:
Table 7: Providing Investment Oversight:
Table 8: Capturing Investment Information:
Table 9: Stage 3 Critical Processes--Developing a Complete Investment
Portfolio:
Table 10: Summary of Results for Stage 3 Critical Processes and Key
Practices:
Table 11: Defining the Portfolio Criteria:
Table 12: Creating the Portfolio:
Table 13: Evaluating the Portfolio:
Table 14: Conducting Postimplementation Reviews:
Figures:
Figure 1: Simplified HHS Organizational Chart:
Figure 2: HHS Discretionary IT Investments for Fiscal Year 2006:
Figure 3: Detailed Breakdown of HHS's Investment Management Process:
Figure 4: The Five ITIM Stages of Maturity with Critical Processes:
Abbreviations:
CPIC: Capital Planning and Investment Control:
CIO: Chief Information Officer:
HHS: Department of Health and Human Services:
IT: information technology:
ITIM: information technology investment management framework:
ITIRB: Information Technology Investment Review Board:
PMT: Portfolio Management Tool:
PIR: postimplementation reviews:
Letter:
October 28, 2005:
The Honorable Charles E. Grassley:
Chairman, Committee on Finance:
United States Senate:
Dear Mr. Chairman:
The Department of Health and Human Services (HHS) is one of the largest
federal agencies, the nation's largest health insurer, and the largest
grant-making agency in the federal government. The department manages
over 300 programs that serve to improve the health and well-being of
the American public and is comprised of several component agencies
covering a wide range of activities including conducting and sponsoring
medical and social science research, guarding against the outbreak of
infectious diseases, assuring the safety of food and drugs, and
providing health care services and insurance. It also manages and funds
a variety of information technology (IT) initiatives ranging from those
facilitating the payment of claims for Medicare and Medicaid services
to those supporting health surveillance and communications. In fiscal
year 2006, the department plans to spend over $5 billion on information
technology--the third largest IT expenditure in the federal
budget.[Footnote 1]
This report is one of two we prepared in response to your request that
we evaluate HHS's information technology investment management
capabilities.[Footnote 2] It focuses on HHS's processes for making IT
investment management decisions and evaluates how well these processes
compare with the accepted practices presented in our IT investment
management (ITIM) framework.[Footnote 3] This framework provides a
method for evaluating and assessing how well an agency is selecting and
managing its IT resources. As we agreed with your office, our
objectives were to (1) assess the department's capabilities for
managing its IT investments and (2) determine any plans the department
might have for improving those capabilities. To address these
objectives, we analyzed documents and interviewed agency officials to
(1) validate and update HHS's self-assessments of key practices in the
framework and (2) evaluate HHS's plans for improving its capabilities.
We performed our work from January through September 2005, in
accordance with generally accepted government auditing standards.
Appendix I contains details about our objectives, scope, and
methodology.
Results in Brief:
Because of the management attention that has been given to IT
investment management, HHS has established over half of the
foundational practices needed to manage its IT investments individually
and about 30 percent of the key practices needed to effectively manage
its portfolio of investments. For example, HHS has implemented many of
the practices required to ensure that (1) projects support business
needs and meet users' requirements, (2) a well-defined and disciplined
process is used to select IT investments, (3) investment information is
captured in a repository for decision makers, and (4) IT portfolio
selection criteria are developed and maintained. However, critical
weaknesses remain in several areas. Specifically, HHS lacks:
* business representation on its senior IT investment review board of
component agencies to carry out its full scope of responsibilities,
* an established process for the IT investment board to regularly
review a defined set of the component agencies' IT investments and
maintain visibility of other investments,
* criteria for assessing portfolio performance or regularly reviewing
the performance of the organization's investment portfolio, and:
* processes for conducting postimplementation reviews (PIR) of its IT
investments.
The department also does not have a structured mechanism in place for
ensuring that component agencies define and implement investment
management processes that are aligned with those of the department.
Until the department fully establishes all foundational and portfolio-
level practices and establishes a mechanism to ensure that component
agencies define and implement processes that are aligned with those of
the department, executives cannot be assured that they are
appropriately selecting, managing, and evaluating the mix of
investments that will maximize returns to the organization, taking into
account the appropriate level of risk.
HHS has initiated steps to improve its investment management process;
however, these steps do not fully address the weaknesses we identify in
this report, nor are they coordinated along with other needed
improvement efforts into a plan that (1) is based on an assessment of
strengths and weaknesses; (2) specifies measurable goals, objectives,
and milestones; (3) specifies needed resources; (4) assigns clear
responsibility and accountability for accomplishing tasks; and (5) is
approved by senior management. Without such a plan and procedures for
implementing it, the department risks being unable to effectively
establish mature investment management capabilities. As a result,
executives may not be able to make informed and prudent investment
decisions in managing the department's annual multibillion-dollar IT
budget.
To further strengthen HHS's investment management capability, we are
recommending that the department develop and implement a plan aimed at
addressing the weaknesses that we identify in this report.
In commenting on a draft of this report, HHS generally agreed with our
findings and recommendations and stated that it will leverage the
report in its efforts to improve its investment management processes.
However, it expressed differing perspectives on the inclusion of
component agency business representation on the investment review and
the performance of postimplementation reviews. Specifically, the
department commented that it used a hierarchy of investment reviews
combined with investment review board members representing mission
support areas such as Finance, Acquisition, and Human Resources, to
provide a structure for making the business decisions regarding the
department's investments. Nevertheless, we reiterate the importance of
having business representation from component agencies to make these
decisions. In addition, the department stated that it was performing
postimplementation reviews in an informal manner through closeout
reviews of investments that have recently been implemented and annual
reviews of systems in operations and maintenance. However, neither of
these reviews currently identify lessons learned or capture benefits
realized, key elements of postimplementation reviews.
Background:
HHS's Mission, Organizational Structure, and Use of IT:
HHS is the primary organization within the federal government that is
devoted to protecting the health of Americans. It provides essential
human services, such as ensuring food and drug safety and assisting
needy families. HHS administers more grant dollars than all other
federal agencies combined, providing over $200 billion of the more than
$350 billion in federal funds that were awarded to states and other
entities in fiscal year 2002, the most recent year for which these data
are available. For fiscal year 2005, HHS had a budget of $581 billion
and a workforce of over 67,000 employees.
To accomplish its mission, HHS is comprised of 12 component
agencies[Footnote 4] and several staff offices that cover a wide range
of activities--including conducting and sponsoring medical and social
science research, guarding against the outbreak of infectious diseases,
assuring the safety of food and drugs, and providing health care
services and insurance. The Office of the Secretary consists of several
staff divisions and offices, including the Office of the Assistant
Secretary for Budget, Technology, and Finance. The HHS Office of the
Chief Information Officer (CIO) is located within this staff office
(see fig. 1).
Figure 1: Simplified HHS Organizational Chart:
[See PDF for image]
[End of figure]
Information technology investments play a critical role in helping HHS
carry out its diverse mission. According to the President's most recent
budget, HHS expects to spend about $5 billion in IT in fiscal year
2006, making the department's IT investment budget the third largest in
the federal government. As figure 2 illustrates, approximately $3
billion is designated as grants to states for investments for Medicaid
programs and other purposes, such as child support enforcement systems.
Approximately $2 billion is for discretionary investment spending, of
which 89 percent is used to fund IT investments for component agencies;
7 percent is invested in HHS enterprisewide initiatives;[Footnote 5]
and 4 percent is used to fund other initiatives, including Office of
the Inspector General IT investments.
Figure 2: HHS Discretionary IT Investments for Fiscal Year 2006 (in
millions):
[See PDF for image]
[End of figure]
Table 1 provides additional information about the component agencies
and their estimated IT budget for fiscal year 2006.
Table 1: Estimated IT Budget of HHS Component Agencies for Fiscal Year
2006:
Component agency: Centers for Medicare & Medicaid Services;
Mission: To administer the Medicare program and work in partnership
with the states to administer Medicaid and the State Children's Health
Insurance Program. The agency also enforces health insurance
portability standards and is responsible for implementing a number of
statutory provisions that have been enacted in recent years, including
the Medicare Prescription Drug, Improvement, and Modernization Act of
2003;
Estimated budget for FY 2006 (in millions)[A]: $780.
Component agency: National Institutes of Health;
Mission: To extend healthy life and reduce the burdens of illness and
disability by pursuing fundamental knowledge about the nature and
behavior of living systems and the application of that knowledge;
Estimated budget for FY 2006 (in millions)[A]: $479.
Component agency: Centers for Disease Control and Prevention;
Mission: To promote health and quality of life by preventing and
controlling disease, injury, and disability;
Estimated budget for FY 2006 (in millions)[A]: $309[B].
Component agency: Food and Drug Administration;
Mission: To protect the public health by ensuring the safety, efficacy,
and security of human and veterinary drugs, biological products,
medical devices, the nation's food supply, cosmetics, and products that
emit radiation;
Estimated budget for FY 2006 (in millions)[A]: $194.
Component agency: Agency for Healthcare Research and Quality;
Mission: To improve the quality, safety, efficiency, and effectiveness
of health care for all Americans;
Estimated budget for FY 2006 (in millions)[A]: $65.
Component agency: Indian Health Service;
Mission: To raise the physical, mental, social, and spiritual health of
American Indians and Alaska Natives;
Estimated budget for FY 2006 (in millions)[A]: $57.
Component agency: Health Resources and Services Administration;
Mission: To provide national leadership, program resources, and
services needed to improve access to culturally competent, quality
health care;
Estimated budget for FY 2006 (in millions)[A]: $51.
Component agency: Program Support Center;
Mission: To provide a full range of program support services to all
components of HHS and other federal agencies, primarily in the areas of
Human Resources, Health Resources, Acquisition Services, Administrative
Services, and Financial Management;
Estimated budget for FY 2006 (in millions)[A]: $44.
Component agency: Substance Abuse and Mental Health Services
Administration;
Mission: To build resilience and facilitate recovery for people with or
at risk for substance abuse and mental illness;
Estimated budget for FY 2006 (in millions)[A]: $35.
Component agency: Administration for Children and Families;
Mission: To administer federal programs that promote the economic and
social well- being of families, children, individuals, and communities;
Estimated budget for FY 2006 (in millions)[A]: $34.
Component agency: Administration on Aging;
Mission: To promote the dignity and independence of older people, and
to help society prepare for an aging population by serving as an
advocate for older people, and by overseeing the development of a
comprehensive and coordinated system of care that is responsive to the
needs and preferences of older people and their family caregivers;
Estimated budget for FY 2006 (in millions)[A]: $2.
Component agency: Agency for Toxic Substances and Disease Registry;
Mission: To provide health information and take public health actions
in order to prevent harmful exposures and disease related to toxic
substances;
Estimated budget for FY 2006 (in millions)[A]: $0[B].
Component agency: Total;
Mission: [Empty];
Estimated budget for FY 2006 (in millions)[A]: $2.0 billion.
Source: GAO analysis based on Office of Management and Budget and HHS
data.
[A] Office of Management and Budget, Budget of the U.S. Government,
Fiscal Year 2006, Report on IT Spending for the Federal Government for
Fiscal Years 2004, 2005, and 2006. We did not verify these data.
[B] The Agency for Toxic Substances and Disease Registry investments
are included in the total for Centers for Disease Control and
Prevention.
[End of table]
HHS' investments reflect the diversity of the department's missions and
operating environments. For example, HHS currently has several
enterprisewide IT initiatives that enable stakeholders to advance the
causes of better health, safety, and well-being for American people.
These initiatives include:
* Unified Financial Management System, a new core financial system, to
help management monitor budgets, conduct operations, evaluate program
performance, and make financial and programmatic decisions. As a core
financial system, it will interface with an estimated 110 other HHS
information systems.[Footnote 6]
* The Office of the Assistant Secretary for Public Health Emergency
Preparedness maintains a command center where it can coordinate the
response to public health emergencies from one centralized location.
This center is equipped with satellite teleconferencing capability,
broadband Internet hookups, and analysis and tracking software.
In addition, HHS's component agencies have several projects and systems
that are critical to the effective implementation of HHS's mission,
including the following:
* The Food and Drug Administration's Automated Drug Information
Management System is to be developed as a fully electronic information
management system that will receive, evaluate, and disseminate
information about investigational and marketing submissions for human
drugs and therapeutic biologics.
* The National Institutes of Health's major IT initiative, the Clinical
Research Information System, is a comprehensive effort to modernize the
systems that support clinical care and the agency's collection of
research data for the intramural clinical research programs.
* The Centers for Disease Control and Prevention's major IT initiative,
Public Health Information Network, is a national initiative to
implement a multiorganizational business and technical architecture for
public health information systems.[Footnote 7]
Prior Reviews Identified Weaknesses in HHS's IT Investment Management
Process:
In January 2004, we reported[Footnote 8] on a broad view of the
government's implementation of investment management practices at 26
major departments and agencies, including HHS. We also reported--and
HHS acknowledged--that there were serious weaknesses in investment
management. Notably, the department had not yet established selection
criteria for project investments or a requirement that investments
support work processes that have been simplified or redesigned. In
addition, the department did not have decision-making rules to guide
oversight of IT investments, review projects at major milestones, or
systematically track corrective actions. Accordingly, we made several
recommendations, including that HHS revise its investment management
policy and require PIRs to address validating benefits and costs. In
response to our recommendations, the department has been modifying
several of its investment management policies, including its capital
planning and investment control guidance and its governance policies.
More recently, in June 2005, we reported[Footnote 9] that the HHS IT
Investment Review Board had conducted only budgetary reviews of the
Centers for Disease Control and Prevention's Public Health Information
Network and some of its initiatives, until this past February, when HHS
initiated steps for better monitoring of system development projects.
We concluded that until management implements a systematic method for
IT investment reviews, it will have difficulty minimizing risks while
maximizing returns on these critical public health investments.
HHS's Approach to Investment Management:
HHS has several groups and individuals involved in managing both the
enterprisewide and component agency IT investments.[Footnote 10] They
are involved from reviewing and approving a proposed IT project,
through the process of budgeting for it, monitoring it through
implementation, and evaluating it at its conclusion. The composition,
roles, and responsibilities of these individuals and groups are
described below:
Information Technology Investment Review Board (ITIRB)--Chaired by
HHS's CIO, this board is responsible for selecting, controlling, and
evaluating all departmental IT investments. Members include the Deputy
Assistant Secretary for Budget, Finance, Performance and Planning; the
Directors for Acquisition Management Policy and Human Resources; and
the component agency CIOs. The board is supported by an executive
secretary who is responsible for, among other things, managing the flow
of IT investment documentation, scheduling meetings, and assisting the
members in preparing for their meetings. Currently, this board reviews
all enterprisewide investments and delegates responsibilities for
component agency investments to each individual component agencies
investment review boards in accordance with departmental policies and
procedures.
CIO Council--Also chaired by the HHS CIO and comprised of component
agency CIOs, this board advises the HHS ITIRB on the technical
soundness of all IT investments that require departmental review and
provides recommendations regarding, among other things, technical
aspects of affordability, soundness of design, risk, and compliance
with architectural and security standards.
Critical Partners--Comprised of departmental officials from various
functional areas, including enterprise architecture, security and
privacy, acquisition management, finance, budget, human resources, and
e-government; this group is responsible for ensuring that most
investments[Footnote 11] comply with the HHS policy in each of the
functional areas and for advising the HHS ITIRB and individual IT
investment managers on issues in their areas of expertise. Each review
results in a determination whether the investment is approved,
conditionally approved, or not approved. A not approved result is
flagged for executive review.
Business Case Quality Review Team--Comprised of component agency
officials, this group evaluates the justifications for IT investments-
-both formal business cases and information documented in the
department's portfolio management tool's Select forms--against the
criteria used by the Office of Management and Budget's to evaluate
business cases[Footnote 12] agencies submit to the office as part of
the formulation of the federal budget[Footnote 13] and provides
recommendations for improving these justifications.
Capital Planning and Investment Control (CPIC) Reengineering/Portfolio
Management Tool (PMT) Implementation Team--Chaired by the Office of the
CIO officials with representatives from the Critical Partners and the
Business Case Quality Review Team, this group advises the board on
issues regarding investment management policies and procedures and the
implementation of the department's portfolio management tool.
Investment Managers--Responsible for managing investments in accordance
with approved cost, schedule, and performance baselines, and for
maintaining information on project status, control, performance, risk,
and corrective actions.
Process for Managing Investments:
The department has defined a three-phase process for managing
investments that involves selecting proposed projects and reselecting
ongoing projects (select phase), controlling ongoing projects through
development (control phase), and evaluating projects that have been
deployed (evaluate phase). The department retains direct management of
HHS enterprisewide IT investments and delegates considerable authority
for other investments to component agencies. Specifically, the
department selects ongoing and new component agency investments through
the process for selecting enterprisewide IT investments described
below. Controlling and evaluating component agency IT investments are
delegated to the component agencies, which are required by the
department to follow a process similar to the one described below.
Each phase of the process for enterprisewide investments is comprised
of multiple steps that set out requirements needed for the HHS ITIRB to
make the decision to move forward with the project.
The purpose of the select phase is to ensure that HHS chooses the
projects that best support its mission and applies resources to the
most important and valuable investments. The select phase is also
intended to help the department justify budget requests by
demonstrating sound business cases and project plans. To select
investments, HHS has established two separate components--investment
screening for new investment proposals and investment scoring and
screening for ongoing investments.
During the new investment screening, the investment manager is expected
to develop a project prospectus, which identifies a specific business
need and preliminary, high-level system requirements. A high-level
determination of resource and schedule requirements is also to be
conducted as part of the business need identification activities.
Approval of the project prospectus by the HHS ITIRB signifies that the
agency agrees that the need is critical enough to proceed to the next
step in which the business case is developed. During business case
development, the investment manager is required to develop the business
case, which establishes the lifecycle cost, schedule, benefits, and
performance baselines and includes an analysis for each investment to
identify alternatives that may satisfy the needs of the department. In
addition, the investment managers sign a document called the
accountability agreement form to accept responsibility for reporting on
the project status in achieving performance baselines throughout the
remaining phases of the investment management process.
After the project is initially approved by the HHS ITIRB, the business
cases and Select forms for most IT investments are updated annually as
part of the budget formulation process. (The Select forms are a
collection of forms with HHS's portfolio management tool that capture
investment data to justify funding and ensure adequate project planning
during the select phase.) The first step within the annual budget
formulation process requires that all component agencies use the Select
forms to report the project cost estimates that best represent the
level of funding required to meet program or business needs. At this
point, the Critical Partners and the Business Case Quality Review Team
score and rank the Select forms using the department's portfolio
management tool[Footnote 14] to create a single HHS portfolio as well
as component agency portfolios to provide recommendations to the
component agencies for making final adjustments to their portfolio
ranking.
Once the component agencies have made the appropriate changes, the
Office of the CIO develops prioritized IT portfolios for HHS as a whole
as well as each component agency to present to the HHS ITIRB. The
departmental board and CIO Council review and comment on the
prioritized portfolio and submit it to the Secretary's Budget Council
for input into their budget deliberations. The Secretary's Budget
Council then makes recommendations to the Secretary regarding HHS and
component agencies' budgets. Finally, the department submits its
approved Secretary's IT budget to the Office of Management and Budget
for inclusion in the President's Budget.
Once selected for inclusion in the department's IT portfolio, each
project is to be managed by an investment manager and reviewed by the
ITIRB on a quarterly basis throughout the end of development. The board
performs reviews of projects that deviate from predetermined budget,
schedule, or performance milestones established in the business case
and works with the investment managers to develop a correction action
plan. The ITIRB must also decide whether to continue to fund the
project; rebaseline the scope, schedule, or budget; or to terminate the
project.
Once a project has been fully implemented, the HHS ITIRB is to conduct
annual reviews of all HHS enterprisewide steady state investments--that
is, investments in operations and maintenance--to determine whether
they continue to meet the business needs. In addition, investments that
have recently completed implementation or a significant phase are to
undergo PIRs to evaluate actual development events against project
management plans and to identify lessons learned that can be applied to
current and future investments.
Figure 3 illustrates HHS's investment management process phases and
steps. The highlighted steps represent the activities that the
department conducts for both enterprisewide and component agency
investments.
Figure 3: Detailed Breakdown of HHS's Investment Management Process:
[See PDF for image]
[End of figure]
ITIM Maturity Framework:
The ITIM framework is a maturity model composed of five progressive
stages of maturity that an agency can achieve in its investment
management capabilities.[Footnote 15] It was developed on the basis of
our research into the IT investment management practices of leading
private-and public-sector organizations. In each of the five stages,
the framework identifies critical processes for making successful IT
investments. The maturity stages are cumulative; that is, in order to
attain a higher stage the agency must have institutionalized all of the
critical processes at the lower stages, in addition to the higher stage
critical processes.
The framework can be used to assess the maturity of an agency's
investment management processes and as a tool for organizational
improvement. The overriding purpose of the framework is to encourage
investment processes that increase business value and mission
performance, reduce risk, and increase accountability and transparency
in the decision process. We have used the framework in several of our
evaluations,[Footnote 16] and a number of agencies have adopted it.
These agencies have used ITIM for purposes ranging from self-assessment
to redesign of their IT investment management processes.
ITIM's five maturity stages represent steps toward achieving stable and
mature processes for managing IT investments. Each stage builds on the
lower stages; the successful attainment of each stage leads to
improvement in the organization's ability to manage its investments.
With the exception of the first stage, each maturity stage is composed
of "critical processes" that must be implemented and institutionalized
in order for the organization to achieve that stage. These critical
processes are further broken down into key practices that describe the
types of activities that an organization should be performing to
successfully implement each critical process. It is not unusual for an
organization to be performing key practices from more than one maturity
stage at the same time, but efforts to improve investment management
capabilities should focus on implementing all lower stage practices
before addressing higher stage practices.
In the ITIM framework, Stage 2 critical processes lay the foundation
for sound IT investment processes by helping the agency to attain
successful, predictable, and repeatable investment control processes at
the project level. Specifically, Stage 2 encompasses building a sound
investment management foundation by establishing basic capabilities for
selecting new IT projects. It also involves developing the capability
to control projects so that they finish predictably within established
cost and schedule expectations and the capability to identify potential
exposures to risk and put in place strategies to mitigate that risk.
The basic selection processes established in Stage 2 lays the
foundation for more mature selection capabilities in Stage 3, which
represents a major step forward in maturity, in which the agency moves
from project-centric processes to a portfolio approach, evaluating
potential investments by how well they support the agency's missions,
strategies, and goals.
Stage 3 requires that an organization continually assess both proposed
and ongoing projects as parts of a complete investment portfolio--an
integrated and competing set of investment options. It focuses on
establishing a consistent, well-defined perspective on the IT
investment portfolio and maintaining mature, integrated selection (and
reselection), control, and evaluation processes, which are to be
evaluated during PIRs. This portfolio perspective allows decision
makers to consider the interaction among investments and the
contributions to organizational mission goals and strategies that could
be made by alternative portfolio selections, rather than to focus
exclusively on the balance between the costs and benefits of individual
investments.
Stages 4 and 5 require the use of evaluation techniques to continuously
improve both the investment portfolio and the investment processes in
order to better achieve strategic outcomes. At Stage 4 maturity, an
organization has the capacity to conduct IT succession activities and,
therefore, can plan and implement the deselection of obsolete, high-
risk, or low-value IT investments. An organization with Stage 5
maturity conducts proactive monitoring for breakthrough information
technologies that will enable it to change and improve its business
performance. Organizations implementing Stages 2 and 3 have in place
the selection, control, and evaluation processes that are required by
the Clinger-Cohen Act of 1996.[Footnote 17] Stages 4 and 5 define key
attributes that are associated with the most capable organizations.
Figure 4 shows the five ITIM stages of maturity and the critical
processes associated with each stage.
Figure 4: The Five ITIM Stages of Maturity with Critical Processes:
[See PDF for image]
[End of figure]
As defined by the model, each critical process consists of "key
practices" that must be executed to implement the critical process.
HHS Has Established Many Key Practices for Managing Its Investments,
but Has Provided Limited Guidance and Oversight to Component Agencies
Processes:
In order to have the capabilities to effectively manage IT investments,
an agency, at a minimum, should, (1) build an investment foundation by
putting basic, project-level control and selection practices in place
(Stage 2 capabilities) and (2) manage its projects as a portfolio of
investments, treating them as an integrated package of competing
investment options and pursuing those that best meet the strategic
goals, objectives, and mission of the agency (Stage 3 capabilities).
These practices may be executed at various organizational levels of the
agency, including at the component level. However, overall
responsibility for their success remains at the department level.
Therefore, at a minimum, the department should effectively oversee
component agencies' IT investment management processes.
HHS has executed 24 of the 38 key practices that the ITIM framework
requires to build a foundation for IT investment management (Stage 2)
and 8 of the 27 key practices required to manage investments as a
portfolio (Stage 3). However, the department has only provided limited
oversight of component agencies' ITIM processes. Until HHS implements
and oversees a stable investment management process throughout the
department, it will lack essential management controls over all of its
IT investments, and it will be unable to ensure that it is
appropriately selecting, managing, and evaluating the mix of
investments that will maximize returns to the organization, taking into
account the appropriate level of risk.
HHS Has Established Over Half of the Foundational Practices Needed to
Manage Its Investments:
At the ITIM Stage 2 level of maturity, an organization has attained
repeatable, successful IT project-level investment control processes
and basic selection processes. Through these processes, the
organization can identify expectation gaps early and take the
appropriate steps to address them. According to the ITIM, critical
processes at Stage 2 include (1) defining IT investment board[Footnote
18] operations, (2) identifying the business needs for each IT
investment, (3) developing a basic process for selecting new IT
proposals and reselecting ongoing investments, (4) developing project-
level investment control processes, and (5) collecting information
about existing investments to inform investment management decisions.
Table 2 describes the purpose of each of these Stage 2 critical
processes.
Table 2: Stage 2 Critical Processes--Building the Investment
Foundation:
Critical process: Instituting the investment board;
Purpose: To define and establish an appropriate IT investment
management structure and the processes for selecting, controlling, and
evaluating IT investments.
Critical process: Meeting business needs;
Purpose: To ensure that IT projects and systems support the
organization's business needs and meet users' needs.
Critical process: Selecting an investment;
Purpose: To ensure that a well-defined and disciplined process is used
to select new IT proposals and reselect ongoing investments.
Critical process: Providing investment oversight;
Purpose: To review the progress of IT projects and systems, using
predefined criteria and checkpoints, in meeting cost, schedule, risk,
and benefit expectations and to take corrective action when these
expectations are not being met.
Critical process: Capturing investment information;
Purpose: To make available to decision makers information to evaluate
the impacts and opportunities created by proposed (or continuing) IT
investments.
Source: GAO.
[End of table]
In the federal government, the agency head and the CIO are responsible
for effectively managing information technology.[Footnote 19] The
agency head, through the department-level CIO, is responsible for
providing leadership and oversight for foundational critical processes
by ensuring that written policies and procedures are established,
repositories of information are created that support investment
decision making, resources are allocated, responsibilities are
assigned, and all the activities are properly carried out where they
may be most effectively executed. In a large and diverse organization
such as HHS, it is especially critical that the CIO create this
structure and framework to ensure that the organization is effectively
managing its investments at every level. This means that the CIO must
ensure that component agencies have investment management processes in
place that adequately support the department's investment management
process to make certain that funds are being expended on component
agency investments that will fulfill mission needs.
Because of the management attention that has been given to IT
investment management, the department has put in place over half of the
key practices needed to establish the investment foundation. The
department has satisfied all of the key practices associated with
ensuring that projects and systems support organizational needs and
meet users' needs. It has satisfied most of the key practices
associated with identifying and collecting investment information,
selecting new proposals[Footnote 20] and reselecting ongoing
investments, and instituting the department's investment review board.
However, because of its limited involvement in overseeing component
agency investments, the department has not executed any of the key
practices related to providing investment oversight.
Table 3 summarizes the status of HHS's critical processes for Stage 2
and shows how many key practices HHS has executed in managing its IT
investments.
Table 3: Summary of Results for Stage 2 Critical Processes and Key
Practices:
Critical process: Instituting the investment board;
Key practices executed: 5;
Total required by critical process: 8;
Percentage of key practices executed: 63.
Critical process: Meeting business needs;
Key practices executed: 7;
Total required by critical process: 7;
Percentage of key practices executed: 100.
Critical process: Selecting an investment;
Key practices executed: 7;
Total required by critical process: 10;
Percentage of key practices executed: 70.
Critical process: Providing investment oversight;
Key practices executed: 0;
Total required by critical process: 7;
Percentage of key practices executed: 0.
Critical process: Capturing investment information;
Key practices executed: 5;
Total required by critical process: 6;
Percentage of key practices executed: 83.
Critical process: Total;
Key practices executed: 24;
Total required by critical process: 38;
Percentage of key practices executed: 63.
Source: GAO.
[End of table]
HHS Has Established an Investment Review Board, but It Is Operating
without a Comprehensive Process Guide:
The establishment of decision-making bodies or boards is a key
component of the IT investment management process. At the Stage 2 level
of maturity, organizations define one or more boards, provide resources
to support the boards' operations, and appoint members who have
expertise in both operational and technical aspects of proposed
investments. The boards should operate according to a written IT
investment process guide that is tailored to the organization's unique
characteristics, thus ensuring that consistent and effective management
practices are implemented across the organization.[Footnote 21] The
organization selects board members to ensure that they are
knowledgeable about policies and procedures for managing investments.
Organizations at the Stage 2 level of maturity also take steps to
ensure that executives and line managers support and carry out the
decisions of the investment board. According to the ITIM, organizations
should (1) use an investment management guide as an authoritative
document to initiate and manage investment processes and (2) provide a
comprehensive foundation for the policies and procedures that are
developed for all of the other related processes. (The complete list of
key practices is provided in table 4.)
The department has executed 5 of the 8 key practices for this critical
process. The department established an IT investment review board as
its corporate-level investment board that consists of senior officials,
including the CIO and the Deputy Assistant Secretaries for Budget,
Finance, and Performance & Planning. The board is adequately resourced,
with most support being provided by the Office of the CIO, whose
responsibilities include developing and modifying the department's
criteria for selecting, controlling, and evaluating potential and
existing IT investments. In addition, the CIO Council reviews the
enterprisewide investments for technical soundness and provides its
recommendations to the board. The Critical Partners and Business Case
Quality Review Team provide additional support to the board by
reviewing and scoring most of their IT investments.
To ensure that the board's decisions are carried out for enterprisewide
investments, the ITIRB approves an accountability agreement document
and business case that identify the benefits, costs, and schedule for
the approved investments. The board then monitors the investments
through the end of development. HHS requires the component agencies to
follow a similar process in accordance with departmental policies and
procedures. We verified that an accountability agreement document was
signed and the business case identified performance expectations for
the two enterprisewide IT investments we reviewed--Public Key
Infrastructure and Enterprise Architecture initiatives.[Footnote 22]
Additionally, the board has oversight of the development and
maintenance of the documented IT investment process through the CPIC
Reengineering/PMT Implementation Team, who provides investment
management policy change recommendations to the board for approval.
Although HHS has implemented these key practices, it does not have a
comprehensive organization-specific process guide to direct the
operations of the investment board. While the Information Resources
Management policy, guidelines, and standard operating procedures
provide general guidance on the organization's investment management
process, they do not reflect the current investment management process.
Moreover, they do not constitute an IT investment process guide because
they do not sufficiently define the investment process. Specifically,
the policies and procedures do not include information on the roles of
the key players such as the CIO Council, Critical Partners, Business
Case Quality Review Team, or the component agency investment review
boards. In addition, they do not identify the manner in which
investment board's processes are to be coordinated with other key
organizational plans and processes (such as the budget formulation
process). HHS has recently drafted a revised investment management
policy addressing many of these weaknesses; however, it has not been
finalized, and HHS officials could not provide a final issuance date.
Without a comprehensive investment management process guide, the
department lacks the assurance that IT investment activities will be
coordinated and performed in a consistent and cost-effective manner.
Moreover, while HHS has established an IT investment board, the board
does not have business representation (that is, mission representation)
from component agencies. Instead, Chief Information Officers represent
the component agencies. According to HHS's CIO, the membership of the
board is adequate for carrying out the investment activities it
currently performs--primarily focusing on enterprisewide IT
investments. However, because allocating resources among major IT
investments may require fundamental trade-offs among a multitude of
business objectives, portfolio management decisions are essentially
business decisions, and therefore require sufficient business
representation on the board. Until the department adjusts its board
membership to include business representation from component agencies,
it will not have assurance that it includes those executives who are in
the best position to make the full range of decisions needed to enable
the agency to meet its mission most effectively, particularly as it
begins to execute its full range of responsibility.
Finally, the HHS ITIRB is not operating according to its assigned
authority and responsibility. The department's investment management
policy and the HHS ITIRB's charter state that the board has oversight
responsibility for both enterprisewide and a defined set of component
agency IT investments, including projects that are high risk,
crosscutting, and require review by the Office of Management and
Budget. However, the board currently oversees only enterprisewide IT
investments. According to HHS officials, the department has delegated
authority to the component agencies to conduct investment reviews;
however, the board does not have a mechanism in place for ensuring that
component agencies are conducting such reviews in accordance with
department policies and procedures. Until the board operates according
to its assigned authority, it cannot ensure that component agency
investments are properly aligned with the organization's objectives or
reviewed by the appropriate board.
Table 4 shows the rating for each key practice required to institute
the investment board. Each of the "executed" ratings shown below
represents instances where, on the basis of the evidence provided by
HHS officials, we concluded that the specific key practices were
executed by the organization.
Table 4: Instituting the Investment Board:
Type of practice: Organizational commitments;
Key practice: 1. An enterprisewide IT investment board composed of
senior executives from IT and business units is responsible for
defining and implementing the organization's IT investment governance
process;
Rating: Not executed;
Summary of evidence: Although HHS has an enterprisewide IT investment
board that is responsible for defining and implementing the
organization's IT investment governance process and consists of the
department's senior executives from IT and other supporting units,
including the CIO, Deputy Assistant Secretaries for Budget, Finance,
Performance & Planning, and the component agencies' CIO, the board does
not have business representation from component agencies.
Key practice: 2. The organization has a documented IT investment
process directing each investment board's operations;
Rating: Not executed;
Summary of evidence: Although the Information Resources Management
policy, guidelines, and standard operating procedures provide general
guidance on the department's investment management process, these
policies and procedures do not reflect the department's current
investment management process. In addition, these documents do not
constitute an investment management process guide in that they do not
(1) include information on the roles of key working groups involved in
the organization's IT investment processes or (2) identify the manner
in which investment board's processes are to be coordinated with other
key organizational plans and processes (such as the budget formulation
process) or component agency investment management processes. HHS is
currently revising its documented IT investment process to reflect its
current investment management practices.
Type of practice: Prerequisites;
Key practice: 1. Adequate resources, including people, funding, and
tools, are provided for supporting the operations of each IT investment
board;
Rating: Executed;
Summary of evidence: Adequate resources are provided to support the
ITIRB's operations. The executive secretariat provides operations
support such as scheduling meetings and managing the flow of IT
investment documentation. The CIO Council performs technical reviews of
enterprisewide IT investments and provides recommendations to the
ITIRB. The Critical Partners rank and score most IT investments from a
functional perspective, while the Business Case Quality Review Team
ranks and scores these investments against the Office of Management and
Budget Exhibit 300 quality criteria.
Key practice: 2. The board members understand the organization's IT
investment management policies and procedures and the tools and
techniques used in the board's decision-making process;
Rating: Executed;
Summary of evidence: HHS ITIRB members understand the investment
board's policies and procedures and the tools and techniques used in
the board's decision-making process. High-level training has been
provided to members during past board meetings on an informal basis.
Key practice: 3. Each board's span of authority and responsibility is
defined to minimize overlaps or gaps among the boards;
Rating: Executed;
Summary of evidence: HHS' investment board, the ITIRB, is responsible
for defining and implementing the organization's IT investment
governance process.
Type of practice: Activities;
Key practice: 1. The enterprisewide investment board has oversight
responsibilities for the development and maintenance of the
organization's documented IT investment process;
Rating: Executed;
Summary of evidence: While the HHS ITIRB does not directly oversee the
development and maintenance of HHS's documented investment process, it
is involved in this process through the CPIC Reengineering/PMT
Implementation Team, who provides investment management policy change
recommendations to the HHS ITIRB for approval.
Key practice: 2. Each investment board operates in accordance with its
assigned authority and responsibility;
Rating: Not executed;
Summary of evidence: While, the HHS ITIRB's charter assigns the board
authority and responsibility for reviewing both the enterprisewide and
a defined set of component agency IT investments, the board primarily
focuses on enterprisewide IT investments.
Key practice: 3. The organization has established management controls
for ensuring that investment boards' decisions are carried out;
Rating: Executed;
Summary of evidence: HHS ITIRB has established management controls such
as the accountability agreement document for ensuring that the board's
decisions regarding the enterprisewide IT investments, which it
directly reviews, are carried out;
For the two enterprisewide projects we reviewed, we verified that
management controls were established through the accountability
agreement document and business cases.
Source: GAO.
[End of table]
HHS Has a Process for Ensuring That Its Investments Support Business
Needs and Meet Users' Needs:
Defining business needs for each IT project helps to ensure that
projects and systems support an organization's business needs and meet
users' needs. This critical process ensures that an organization's
business objectives and its IT management strategy are linked.
According to the ITIM, effectively meeting business needs requires,
among other things, (1) documenting business needs with stated goals
and objectives; (2) identifying specific users and other beneficiaries
of IT projects and systems; (3) providing adequate resources to ensure
that projects and systems support the organization's business needs and
meet users' needs; and (4) periodically evaluating the alignment of IT
projects and systems with the organization's strategic goals and
objectives. (The complete list of key practices is provided in table
5.)
The department has in place all of the key practices for meeting
business needs. Specifically, HHS has policy and procedures that call
for business needs to be identified in the business case or the
portfolio management tool's Select forms for both proposed and ongoing
enterprisewide and component agency IT projects. Resources devoted to
ensuring that IT projects and systems support the organization's
business needs and meet users' needs include the Business Case Quality
Review Team, the Critical Partners, the portfolio management tool, and
detailed procedures and associated templates for developing business
cases. HHS's specific business mission, with stated goals and
objectives, is defined in the HHS Strategic Plan for fiscal years 2004
through 2009.
Further, HHS defines and documents business needs for both proposed and
ongoing enterprisewide and component agency IT projects, and identifies
users and other beneficiaries during its selection activities. In
addition, according to HHS IT officials, end users participate in
project management throughout the IT project's life cycle. For the four
projects we reviewed, we verified that business needs and specific
users and other beneficiaries were identified and documented in the
business case or in the Select forms within HHS's portfolio management
tool. In addition, end users are involved in project management
throughout the life cycle of the enterprisewide investments. For
example, users of HHS's Public Key Infrastructure and Enterprise
Architecture initiatives participate in project management through
integrated project teams, which meet approximately once a month and are
comprised of representatives from the component agencies. Because the
department has executed all of the key practices associated with
identifying business needs, it has increased confidence that its IT
projects will meet both business needs and users' needs.
Table 5 shows the rating for each key practice required to meet
business needs and summarizes the evidence that supports these ratings.
Table 5: Meeting Business Needs:
Type of practice: Organizational commitments;
Key practice: 1. The organization has documented policies and
procedures for identifying IT projects or systems that support the
organization's ongoing and future business needs;
Rating: Executed;
Summary of evidence: HHS has policies and procedures for ensuring that
IT projects and systems support the department's ongoing and future
business needs.
Type of practice: Prerequisites;
Key practice: 1. The organization has a documented business mission
with stated goals and objectives;
Rating: Executed;
Summary of evidence: The HHS Strategic Plan for fiscal years 2004
through 2009 defines the agency's mission goals and objectives.
Key practice: 2. Adequate resources, including people, funding, and
tools, are provided for ensuring that IT projects and systems support
the organization's business needs and meet users' needs;
Rating: Executed;
Summary of evidence: HHS has adequate resources for ensuring that its
IT projects and systems support the organization's business needs and
meet users' needs. They include Business Case Quality Review Team,
Critical Partners, and the portfolio management tool. Also, HHS has
templates for developing business cases and training manuals on the use
of the portfolio management tool.
Type of practice: Activities;
Key practice: 1. The organization defines and documents business needs
for both proposed and ongoing IT projects and systems;
Rating: Executed;
Summary of evidence: HHS policies and procedures call for business
needs for enterprisewide and component agency ongoing and proposed IT
projects and systems to be specified in the business case or Select
forms;
We verified that business needs were defined and documented within the
business case or Select forms in the portfolio management tool for the
four projects we reviewed.
Key practice: 2. The organization identifies specific users and other
beneficiaries of IT projects and systems;
Rating: Executed;
Summary of evidence: HHS policy and procedures call for specific users
and other beneficiaries of both enterprisewide and component agency IT
projects and systems to be identified in the business case and Select
forms;
We verified that customers and stakeholders were defined and documented
within the business case or Select forms in the portfolio management
tool for the four projects we reviewed.
Key practice: 3. Users participate in project management throughout an
IT project's or system's life cycle;
Rating: Executed;
Summary of evidence: According to HHS IT officials, end users
participate in project management throughout an IT project's or
system's life cycle;
We verified that users participated in project management throughout
the life cycle of the two enterprisewide projects we reviewed.
According to HHS Office of the CIO, user participation in project
management is not addressed at the department level for the two
component agency projects we reviewed since it is delegated to the
component agency.
Key practice: 4. The investment board periodically evaluates the
alignment of its IT projects and systems with the organization's
strategic goals and objectives and takes corrective actions when
misalignment occurs;
Rating: Executed;
Summary of evidence: The ITIRB evaluates the alignment of both HHS
enterprisewide and component agency IT systems through the annual
budget formulation process and takes corrective action when
misalignment occurs.
Source: GAO.
[End of table]
HHS Is Selecting New Investments and Reselecting Ongoing Investments,
but Lacks a Fully Documented Process for Doing So:
Selecting new IT proposals and reselecting ongoing investments require
a well-defined and disciplined process to provide the agency's
investment boards, business units, and developers with a common
understanding of the process and the cost, benefit, schedule, and risk
criteria that will be used both to select new projects and to reselect
ongoing projects for continued funding. According to the ITIM, this
critical process requires, among other things, (1) making funding
decisions for new proposals according to an established process; (2)
providing adequate resources for investment selection activities; (3)
using a defined selection process to select new investments and
reselect ongoing investments; (4) establishing criteria for analyzing,
prioritizing, and selecting new IT investments and for reselecting
ongoing investments; and (5) creating a process for ensuring that the
criteria change as organizational objectives change. (The complete list
of key practices is provided in table 6.)
HHS has executed 7 of the 10 key practices associated with selecting an
investment. For example, resources devoted to selection activities
include the Critical Partners, Business Case Quality Review Team, and
portfolio management tool, which contains several forms for selecting
IT projects and systems. HHS also has detailed procedures for using its
portfolio management tool and developing business cases. The criteria
for analyzing, prioritizing, selecting and reselecting new and ongoing
investments address the President's Management Agenda, HHS strategic
goals, and IT strategic goals, value, and risk. They are incorporated
into the department's portfolio management tool and are reviewed by the
investment review board and adjusted within the tool annually at the
beginning of each budget cycle to reflect organizational objectives.
This year, HHS added additional criteria--a quality score.
HHS uses its annual budget formulation process to select both
enterprisewide and component agency proposed and ongoing IT
investments. We verified that the four projects we reviewed were
reselected by the department using the annual budget formulation
process.
Although HHS has the above strengths, the department has not executed
any of the practices associated with documenting policies and
procedures. Specifically, HHS has not fully documented its process for
selecting new IT proposals and reselecting ongoing IT investments.
Although a number of documents address investment selection, they are
not linked to provide decision makers with a clear understanding of the
selection and reselection processes. In addition, they do not define
the roles and responsibilities for all key players involved in these
processes. Moreover, although the HHS Office of the CIO works directly
with the department's Office of the Budget, HHS does not have policies
and procedures documenting the integration of funding with the process
of selecting and reselecting investments. Until the department fully
documents policies and procedures for selecting new IT proposals and
reselecting ongoing IT investments, the department will not be
adequately certain that it is consistently and objectively selecting
and reselecting investments that best meet the needs and priorities of
the department.
Table 6 shows the rating for each key practice required to select an
investment and summarizes the evidence that supports these ratings.
Table 6: Selecting an Investment:
Type of practice: Organizational commitments;
Key practice: 1. The organization has documented policies and
procedures for selecting new IT proposals;
Rating: Not executed;
Summary of evidence: Although HHS has a number of documents that
address investment selection, they are not linked to provide decision
makers with a common understanding of the selection process. In
addition, these documents do not define the roles and responsibilities
for each participating unit involved in the project selection process.
Key practice: 2. The organization has documented policies and
procedures for reselecting[A] ongoing IT investments;
Rating: Not executed;
Summary of evidence: Although HHS has a number of documents that
address investment reselection, they are not linked to provide the
decision makers with a common understanding of the selection process.
In addition, these documents do not define the roles and
responsibilities for each participating unit involved in the project
selection process.
Key practice: 3. The organization has policies and procedures for
integrating funding with the process of selecting an investment;
Rating: Not executed;
Summary of evidence: Although the HHS Office of the CIO works directly
with the department's Office of the Budget, HHS does not have policies
and procedures documenting the integration of funding with the process
of selecting and reselecting investments.
Type of practice: Prerequisites;
Key practice: 1. Adequate resources, including people, funding, and
tools, are provided for identifying and selecting IT projects and
systems;
Rating: Executed;
Summary of evidence: Adequate resources are provided for identifying
and selecting IT projects and systems. They include the Critical
Partners, Business Case Quality Review Team, and the department's
portfolio management tool, which contains several forms for selecting
IT projects and systems.
Key practice: 2. Criteria for analyzing, prioritizing, and selecting
new IT investment opportunities have been established;
Rating: Executed;
Summary of evidence: HHS has established criteria for analyzing,
prioritizing, and selecting enterprisewide and component agency new IT
investments. The department selects new IT proposals and reselects
ongoing investments using the same criteria, which are incorporated
into its portfolio management tool.
Key practice: 3. Criteria for analyzing, prioritizing, and reselecting
IT investment opportunities have been established;
Rating: Executed;
Summary of evidence: HHS has established criteria for analyzing,
prioritizing, and reselecting both enterprisewide and component agency
IT investments. The department selects new IT proposals and reselects
ongoing investments using the same criteria, which are incorporated
into its portfolio management tool.
Key practice: 4. A mechanism exists to ensure that the criteria
continue to reflect organizational objectives;
Rating: Executed;
Summary of evidence: The HHS ITIRB reviews and adjusts criteria
annually at the start of each budget cycle and updates the portfolio
management tool to reflect HHS's objectives.
Type of practice: Activities;
Key practice: 1. The organization uses its defined selection process,
including predefined selection criteria, to select new IT investments;
Rating: Executed;
Summary of evidence: HHS uses its annual budget formulation process to
select new IT investments;
We verified that the four projects we reviewed were selected using the
annual budget formulation activities.
Key practice: 2. The organization uses the defined selection process,
including predefined selection criteria, to reselect ongoing IT
investments;
Rating: Executed;
Summary of evidence: HHS uses its annual budget formulation process to
reselect ongoing IT investments;
We verified that the four projects we reviewed were reselected using
the annual budget formulation activities.
Key practice: 3. Executives' funding decisions are aligned with
selection decisions;
Rating: Executed;
Summary of evidence: The HHS ITIRB makes funding decisions for new and
ongoing IT investments through the department's budget formulation
process, which is used to select both enterprisewide and component
agency investments.
Source: GAO.
[A] According to the GAO ITIM framework, reselecting is the periodic
reconsideration of an investment's continuing value to the organization
and the decision to continue funding. It is a recurring process that
continues for as long as a project is receiving funding.
[End of table]
HHS Does Not Have a Process for Effectively Overseeing Its Component
Agency IT Investments:
An organization should effectively oversee its IT projects throughout
all phases of their life cycles. Its investment board should observe
each project's performance and progress toward predefined cost and
schedule expectations as well as each project's anticipated benefits
and risk exposure. This does not mean that a departmental board, such
as the ITIRB, should micromanage each project to provide effective
oversight; rather it means that the departmental board should be
actively involved in all IT investments and proposals that are high
cost or high risk or have significant scope and duration and at a
minimum, should, have a mechanism for maintaining visibility of other
investments. The board should also employ early warning systems that
enable it to take corrective actions at the first sign of cost,
schedule, and performance slippages. According to the ITIM, effective
project oversight requires, among other things, (1) having written
policies and procedures for management oversight; (2) developing and
maintaining an approved management plan for each IT project; (3) making
up-to-date cost and schedule data for each project available to the
oversight boards; (4) having regular reviews by each investment board
of each project's performance against stated expectations; and (5)
ensuring that corrective actions for each underperforming project are
documented, agreed to, implemented, and tracked until the desired
outcome is achieved. (The complete list of key practices is provided in
table 7.)
The department has not executed any of the seven key practices
associated with effective project oversight, primarily because of its
limited role in overseeing component agency IT investments.
Specifically, while the department has documented standard operating
procedures and instructional memorandums for oversight of
enterprisewide IT investments, they are not comprehensive in that they
do not specify the board's responsibilities for investment oversight;
procedural rules for the ITIRB operations and decision making during
project oversight; or policies and procedures for overseeing component
agency IT investments.
The HHS ITIRB is currently performing regular reviews[Footnote 23] of
enterprisewide IT projects and systems against stated expectations
through reports that are available to decision makers on the HHS
Intranet. However, the department is not regularly reviewing component
agency investments that are high risk, crosscutting, and require review
by the Office of Management and Budget, although their policy calls for
it. The board also does not have a mechanism for maintaining visibility
of other component agency investments.
The department delegates oversight of these investments to the
component agencies but believes it is nonetheless effectively
overseeing component agency investments through (1) reviews of these
investments as part of the annual Critical Partner and Business Case
Quality reviews performed during the annual selection process and the
use of (2) earned value management data.[Footnote 24] Although the
annual reviews may provide insight into the status of investments, they
are not frequent enough to allow for timely identification of problems.
Moreover, while HHS officials told us that staff responsible for
collecting earned value management data on component agency investments
share significant concerns about the data with the ITIRB, they did not
have formal documentation clearly supporting this issue. In addition,
formal procedures for elevating issues to the board have not been
developed. In the absence of effective board oversight, HHS executives
will not have the information they need to determine whether component
agency projects are being developed on schedule and within budget. In
addition, the department will run the risk that underperforming
component agency projects will not be identified in time for corrective
actions to be taken.
We verified that HHS provided oversight for the two enterprisewide
investments, but had delegated oversight activities for the two
component agency investments we reviewed.
Table 7 shows the rating for each key practice required to provide
investment oversight and summarizes the evidence that supports these
ratings.
Table 7: Providing Investment Oversight:
Type of practice: Organizational commitment;
Key practice: 1. The organization has documented policies and
procedures for management oversight of IT projects and systems;
Rating: Not executed;
Summary of evidence: Although HHS has developed standard operating
procedures and instructional memorandums for oversight of
enterprisewide IT projects and systems, they do not (1) specify the HHS
ITIRB's responsibilities when providing investment oversight within its
domain or (2) procedural rules for the ITIRB's operations and for
decision making during project oversight. In addition, HHS does not
have policies and procedures for management oversight of component
agency investments.
Type of practice: Prerequisites;
Key practice: 1. Adequate resources, including people, funding, and
tools, are provided for IT project oversight;
Rating: Not executed;
Summary of evidence: Although HHS has adequate resources for providing
oversight for enterprisewide IT investments, the department does not
have adequate resources for providing oversight for component agency IT
investments.
Key practice: 2. IT projects and systems, including those in steady
state (operations and maintenance), maintain approved project
management plans that include expected cost and schedule milestones and
measurable benefit and risk expectations;
Rating: Not executed;
Summary of evidence: HHS's policy calls for an accountability agreement
document and business case, including cost, benefit, schedule, and risk
expectations, to be available to the ITIRB after approval of an
enterprisewide IT projects and systems, but there is no similar
requirement for component agency IT projects and systems;
We verified that HHS provided oversight for the two enterprisewide
investments, but had delegated oversight activities for the two
component agency investments we reviewed.
Type of practice: Activities;
Key practice: 1. Data on actual performance (including cost, schedule,
benefit, and risk performance) are provided to the appropriate IT
investment board;
Rating: Not executed;
Summary of evidence: Data on actual performance of enterprisewide IT
investments are provided to the HHS ITIRB;
however, the ITIRB does not regularly receive data on actual
performance of a defined set of component agencies' IT investments and
maintain visibility of other investments;
We verified that the two enterprisewide projects provide quarterly
reports to the ITIRB. For the component agency projects we reviewed,
this activity is delegated to the component agency and is not addressed
at the department level.
Key practice: 2. Using verified data, each investment board regularly
reviews the performance of IT projects and systems against stated
expectations;
Rating: Not executed;
Summary of evidence: HHS ITIRB quarterly reviews performance of
enterprisewide IT investments under development and annually reviews
enterprisewide IT investment in their operational phase of their life
cycles;
however, the investment board does not have a process for regularly
reviewing the performance of a defined set of component agency
investments and maintaining visibility of other investments.
Key practice: 3. For each underperforming IT project or system,
appropriate actions are taken to correct or terminate the project or
system in accordance with defined criteria and the documented policies
and procedures for management oversight;
Rating: Not executed;
Summary of evidence: The HHS ITIRB takes appropriate actions to correct
or terminate the enterprisewide IT projects or systems. However, it
does not take actions to correct or terminate underperforming component
agency investments because it does not regularly review these
investments' performance.
Key practice: 4. The investment board regularly tracks the
implementation of corrective actions for each underperforming project
until the actions are completed;
Rating: Not executed;
Summary of evidence: The HHS ITIRB maintains meeting minutes for
enterprisewide IT investments to ensure that corrective actions are
implemented and tracked until the desired outcome is achieved. However,
it does not take actions to correct or terminate underperforming
component agency investments because it does not regularly review these
investments' performance.
Source: GAO.
[End of table]
HHS Has a Defined Process for Capturing Investment Information:
To make good IT investment decisions, an organization must be able to
acquire pertinent information about each investment and store that
information in a retrievable format. During this critical process, an
organization identifies its IT assets and creates a comprehensive
repository of investment information. This repository provides
information to investment decision makers to help them evaluate the
potential impacts and opportunities created by proposed or continuing
investments. It can provide insights into major IT cost and management
drivers and trends. The repository can take many forms and need not be
centrally located, but the collection method should, at a minimum,
identify each IT investment and its associated components. This
critical process may be satisfied by the information contained in the
organization's current enterprise architecture, augmented by additional
information--such as financial information and information on risk and
benefits--that the investment board may require to ensure that informed
decisions are being made. According to the ITIM, effectively managing
this repository requires, among other things, (1) developing written
policies and procedures for identifying and collecting the information;
(2) assigning responsibilities for ensuring that the information being
collected meets the needs of the investment management process; (3)
identifying IT projects and systems and collecting relevant information
to support decisions about them; and (4) making the information easily
accessible to decision makers and others. (The complete list of key
practices is provided in table 8.)
HHS has executed 5 of the 6 key practices for capturing investment
information. For example, the department has several documents that
define the policies and procedures for identifying and collecting
investment information in its repositories and also assign
responsibility to the HHS CIO for ensuring that the information
collected during project and systems identification meets the needs of
the investment management process. HHS maintains a portfolio management
tool, which serves as the primary repository for identifying and
collecting information about both department and component agency IT
projects and systems. The department's portfolio management tool is
easily accessible to decision makers at both the department and
component level and the Office of the CIO has provided decision makers
with various training manuals and guidance memorandums. In addition,
the department also identifies and collects information about
enterprisewide IT investments using its Intranet. Further, the
department recently began collecting earned value information through
spreadsheets on major HHS IT investments that compares planned and
actual cost and schedule information. These repositories are easily
accessible to the board members.
The key practice HHS has not executed has to do with the captured
investment information not yet being used by the HHS ITIRB to fully
support decisions about component agency investments. For example, the
earned value investment data received from each component agency has
not been used by the HHS ITIRB for control and evaluation decisions.
According to agency officials, the department has recently begun
monitoring the earned value data to identify investments that report
cost and schedule variances and these officials acknowledge a need to
formalize the process for doing so. Until HHS's decision makers use the
information in the repository to fully support the investment
management process, it will be unable to effectively evaluate the
impacts and opportunities created by proposed or continuing
investments.
Table 8 shows the rating for each key practice required to capture
investment information and summarizes the evidence that supports these
ratings.
Table 8: Capturing Investment Information:
Type of practice: Organizational commitments;
Key practice: 1. The organization has documented policies and
procedures for identifying and collecting information about IT projects
and systems to support the investment management process;
Rating: Executed;
Summary of evidence: The department has documented policies and
procedures for identifying and collecting information about IT projects
and systems to support the investment management process.
Key practice: 2. An official is assigned responsibility for ensuring
that the information collected during project and systems
identification meets the needs of the investment management process;
Rating: Executed;
Summary of evidence: The HHS CIO is responsible for ensuring that the
information collected during project and systems identification meets
the needs of the investment management process.
Type of practice: Prerequisite;
Key practice: 1. Adequate resources, including people, funding, and
tools, are provided for identifying IT projects and systems and
collecting relevant investment information about them;
Rating: Executed;
Summary of evidence: According to the HHS IT officials, adequate
resources are provided for identifying IT projects and systems and
collecting relevant investment information about them.
Type of practice: Activities;
Key practice: 1. The organization's IT projects and systems are
identified, and specific information is collected to support decisions
about them;
Rating: Executed;
Summary of evidence: HHS's portfolio management tool identifies and
collects information about both department and component agency IT
projects and systems to support the investment management process as it
currently exists. The department also identifies and collects relevant
investment information for the enterprisewide IT investments through
the HHS Intranet and component agency IT investments through
spreadsheets that capture earned value data;
We verified that HHS's portfolio management tool identifies and
contains investment information for the four projects we reviewed.
Key practice: 2. The information that has been collected is easily
accessible and understandable to decision makers and others;
Rating: Executed;
Summary of evidence: IT investment decision makers at both the
department and component agency level have access to HHS's portfolio
management tool that is used to capture IT project and system
information. Instructions on the use and navigation through the
portfolio management system are available to investment management
decision makers. In addition, the HHS ITIRB can also access the
enterprisewide IT investment information posted on the HHS Intranet.
Key practice: 3. The information repository is used by investment
decision makers and others to support investment management;
Rating: Not executed;
Summary of evidence: While HHS identifies and collects information
about IT projects and systems to support the investment management
process, this information has not been used by the HHS ITIRB to fully
support the control and evaluate decisions for component agency IT
investments.
Source: GAO.
[End of table]
HHS Has Some of the Capabilities Needed to Manage IT Investments as a
Portfolio:
Once an agency has attained Stage 2 maturity, it needs to implement
critical processes for managing its investments as a portfolio (Stage
3). An IT investment portfolio is an integrated, agencywide collection
of investments that are assessed and managed collectively based on
common criteria. Managing investments as a portfolio is a conscious,
continuous, and proactive approach to allocating limited resources
among an organization's competing initiatives in light of the relative
benefits expected from these investments. Taking an agencywide
perspective enables an organization to consider its investments
comprehensively, so that collectively the investments optimally address
the organization's missions, strategic goals, and objectives. Managing
IT investments as a portfolio also allows an organization to determine
its priorities and make decisions about which projects to fund and
continue to fund based on analyses of the relative organizational value
and risks of all projects, including projects that are proposed, under
development, and in operation. Although investments may initially be
organized into subordinate portfolios--based on, for example, business
lines or life cycle stages--and managed by subordinate investment
boards; they should ultimately be aggregated into this enterprise-level
portfolio.
According to the ITIM framework, Stage 3 maturity includes (1) defining
the portfolio criteria, (2) creating the portfolio, (3) evaluating the
portfolio, and (4) conducting postimplementation reviews. Table 9
summarizes the purpose of each critical process in Stage 3.
Table 9: Stage 3 Critical Processes--Developing a Complete Investment
Portfolio:
Critical process: Defining the portfolio criteria;
Purpose: To ensure that the organization develops and maintains IT
portfolio selection criteria that support its mission, organizational
strategies, and business priorities.
Critical process: Creating the portfolio;
Purpose: To ensure that IT investments are analyzed according to the
organization's portfolio selection criteria and to ensure that an
optimal IT investment portfolio with manageable risks and returns is
selected and funded.
Critical process: Evaluating the portfolio;
Purpose: To review the performance of the organization's investment
portfolio(s) at agreed- upon intervals and to adjust the allocation of
resources among investments as necessary.
Critical process: Conducting postimplementation reviews;
Purpose: To compare the results of recently implemented investments
with the expectations that were set for them and to develop a set of
lessons learned from these reviews.
Source: GAO.
[End of table]
HHS has executed 8 of the 27 key practices required by Stage 3. For
example, the department's core IT portfolio selection criteria,
including cost, benefit, schedule, and risk are approved by the HHS
ITIRB. In addition, the investment board examines the mix of new and
ongoing investments and their respective data and analyses to select
investments to fund. However, many key practices still need to be
executed before HHS can effectively manage its IT investments from a
portfolio perspective. For example, HHS has not addressed any of the
key practices related to evaluating the portfolio or conducting PIRs.
Until HHS fully implements the critical processes associated with
managing its investments as a complete portfolio, it will not have the
data it needs to make informed decisions about competing investments.
Table 10 summarizes the status of HHS's critical processes for Stage 3,
showing how many associated key practices it has executed.
Table 10: Summary of Results for Stage 3 Critical Processes and Key
Practices:
Critical process: Defining the portfolio criteria;
Key practices executed: 5;
Total required by critical process: 7;
Percentage of key practices executed: 71.
Critical process: Creating the portfolio;
Key practices executed: 3;
Total required by critical process: 7;
Percentage of key practices executed: 43.
Critical process: Evaluating the portfolio;
Key practices executed: 0;
Total required by critical process: 7;
Percentage of key practices executed: 0.
Critical process: Conducting postimplementation reviews;
Key practices executed: 0;
Total required by critical process: 6;
Percentage of key practices executed: 0.
Critical process: Total;
Key practices executed: 8;
Total required by critical process: 27;
Percentage of key practices executed: 30.
Source: GAO.
[End of table]
Process for Modifying IT Portfolio Selection Criteria Is Not
Institutionalized:
To manage IT investments effectively, an organization needs to
establish rules or "portfolio selection criteria" for determining how
to allocate scarce funding to existing and proposed investments. Thus,
developing an IT investment portfolio requires defining appropriate
cost, benefit, schedule, and risk criteria with which to evaluate
individual investments in the context of all other investments. To
ensure that the organization's strategic goals, objectives, and mission
will be satisfied by its investments, the criteria should have an
enterprisewide perspective. Further, if an organization's mission or
business needs and strategies change, criteria for selecting
investments should be reexamined and modified as appropriate. Portfolio
selection criteria should be disseminated throughout the organization
to ensure that decisions concerning investments are made in a
consistent manner and that this critical process is institutionalized.
To achieve this result, project management personnel and others should
be aware of the criteria and address the criteria in funding
submissions for projects. Resources required for this critical process
typically include the time and attention of executives involved in the
process, adequate funding, and supporting tools. (The complete list of
key practices is provided in table 11.)
The department has executed 5 of the 7 key practices for this critical
process. For example, responsibility has been assigned to the HHS Lead
Capital Planner for managing the development and modification of the IT
portfolio selection criteria, and adequate resources have been
committed for portfolio selection activities, including the Critical
Partners, portfolio management tool project manager, and the Office of
the CIO staff. Moreover, the project management personal and other
stakeholders are aware of the portfolio selection criteria that are
embedded into the department's portfolio management tool and also
contained within policies and procedures.
Finally, the HHS ITIRB approves the core IT selection criteria,
including cost, benefit, schedule, and risk criteria, based on the
organization's mission, goals, strategies, and priorities. Beginning in
fiscal year 2004, HHS began scoring and ranking approximately 80
percent of its IT investments against alignment, value, and risk
criteria in order to determine a priority score, which is the sum of
alignment, value, and risk criteria scores, weighted for relative
importance. Similarly, for the fiscal year 2007 budget formulation
process, HHS began collecting investment information on the business
case quality, Critical Partner reviews, and cost and schedule variance
to determine a quality score, which is the sum of the business case
quality, Critical Partner reviews, and cost and schedule variance
scores, weighted for relative importance. The HHS ITIRB evaluates and
annually adjusts its portfolio selection criteria within the portfolio
management tool.
Despite these important steps in defining portfolio selection criteria,
weaknesses remain. The department has not developed policies or
procedures for modifying the portfolio selection criteria to reflect
changes to HHS mission, goals, strategies, and priorities. In addition,
the HHS ITIRB began reviewing the IT portfolio selection criteria this
year. However, the process for modifying portfolio selection criteria
is not institutionalized because the process to do so was only used
once and there are no documented policies and procedures to ensure that
it will be used again. Until HHS defines and implements the practices
required for defining the portfolio criteria definition, it will not
have the tool it needs to select investments that support its mission,
organizational strategies, and business priorities.
Table 11 shows the rating for each key practice required to define
portfolio selection criteria and summarizes the evidence that supports
these ratings.
Table 11: Defining the Portfolio Criteria:
Type of practice: Organizational commitments;
Key practice: 1. The organization has documented policies and
procedures for creating and modifying IT portfolio selection criteria;
Rating: Not executed;
Summary of evidence: While HHS has policies and procedures for creating
IT portfolio selection criteria, the department lacks policies and
procedures for modifying the portfolio selection criteria.
Key practice: 2. Responsibility is assigned to an individual or group
for managing the development and modification of the IT portfolio
selection criteria;
Rating: Executed;
Summary of evidence: The HHS Lead Capital Planner is responsible for
managing the development and modification of the IT portfolio selection
criteria.
Type of practice: Prerequisites;
Key practice: 1. Adequate resources, including people, funding, and
tools, have been committed for portfolio selection criteria activities;
Rating: Executed;
Summary of evidence: Adequate resources have been committed for
portfolio selection criteria activities. They include the Critical
Partners, portfolio management tool project manager, and the Office of
the CIO staff.
Key practice: 2. A working group has been designated to be responsible
for developing and modifying the IT portfolio selection criteria;
Rating: Executed;
Summary of evidence: The CPIC Reengineering/PMT Implementation Team
conducts weekly teleconferences with HHS component agencies to
coordinate investment management issues, including the development and
modification of IT portfolio selection criteria. According to HHS IT
officials, this group will evolve into the Policy Advisory Board,
which, among other things, will formalize the IT portfolio selection
criteria activities.
Type of practice: Activities;
Key practice: 1. The enterprisewide investment board approves the core
IT portfolio selection criteria, including CBSR criteria, based on the
organization's mission, goals, strategies, and priorities;
Rating: Executed;
Summary of evidence: The HHS ITIRB approves the core IT portfolio
selection criteria, including cost, benefit, schedule, and risk
criteria, based on the organization's mission, goals, strategies, and
priorities.
Key practice: 2. Project management personnel and other stakeholders
are aware of the portfolio selection criteria;
Rating: Executed;
Summary of evidence: Project management personnel and other
stakeholders are aware of the portfolio selection criteria, which are
embedded into HHS's portfolio management tool and contained in policies
and procedures.
Key practice: 3. The enterprisewide investment board regularly reviews
the IT portfolio selection criteria, using cumulative experience and
event-driven data, and modifies the criteria as appropriate;
Rating: Not executed;
Summary of evidence: The HHS ITIRB began reviewing the IT portfolio
selection criteria this year. However, the process for modifying the
portfolio selection criteria is not institutionalized because it was
only used once and there are no documented policies and procedures to
ensure that it will be used again.
Source: GAO.
[End of table]
Process for Creating a Portfolio Is Not Documented:
At Stage 3, organizations create a portfolio of IT investments to
ensure that IT investments are analyzed according to the organization's
portfolio selection criteria and to ensure that an optimal IT
investment portfolio with manageable risks and returns is selected and
funded. According to ITIM, creating the portfolio requires
organizations to, among other things, document policies and procedures
for analyzing, selecting, and maintaining the portfolio; provide
adequate resources, including people, funding, and tools for creating
the portfolio; and capture the information used to select, control, and
evaluate the portfolio and maintain it for future reference. In
creating the portfolio, the investment board must also (1) examine the
mix of new and ongoing investments, and their respective data and
analyses and select investments for funding and (2) approve or modify
the performance expectations for the IT investments they have selected.
(The complete list of key practices is provided in table 12.)
HHS has executed 3 of the 7 key practices associated with creating the
portfolio. Beginning in fiscal year 2004, the department began to
create a portfolio by using its portfolio management tool to collect
cost, benefit, schedule, risk, strategic alignment, and enterprise
architecture information on investments accounting for 80 percent of
the dollar value of the HHS IT investment portfolio. Each component
agency's IT portfolio is displayed in priority order along with where
each investment falls within the overall IT portfolio. Further,
according to HHS IT officials, the agency has adequate resources for
portfolio selection activities, including the Critical Partners, the
portfolio management tool project manager, and the Office of the CIO
staff. These officials also stated that HHS ITIRB members are also
knowledgeable about the process of creating a portfolio.
Nevertheless, HHS has a number of significant weaknesses in the way it
creates a portfolio. First, it does not have policies and procedures
that sufficiently address this critical process. Although the
department has policies and procedures for creating IT portfolio
selection criteria, they lack policies and procedures for using these
criteria to analyze, select, and maintain the investment portfolio.
Second, even though the HHS ITIRB has quarterly reviews to compare
project and system performance with expectations for enterprisewide IT
investments, the board is not provided with information comparing the
performance of component agency investments against expectations. In
addition, the board approves or modifies the performance expectations
for the enterprisewide IT investments it has selected, but does not
regularly approve or modify the performance expectations for component
agency IT investments or ensure that this is done. Moreover, as
previously mentioned, investment information has not been used to fully
support control and evaluate decisions for component agency
investments. Unless HHS defines and implements the practices for
creating a comprehensive portfolio of IT investments, it will not be
able to determine whether it has selected the mix of investments that
best meets its needs considering resource and funding constraints.
Table 12 shows the rating for each key practice required to create a
portfolio and summarizes the evidence that supports these ratings.
Table 12: Creating the Portfolio:
Type of practice: Organizational commitment;
Key practice: 1. The organization has documented policies and
procedures for analyzing, selecting, and maintaining the investment
portfolio;
Rating: Not executed;
Summary of evidence: While HHS has policies and procedures for creating
IT portfolio selection criteria, the department lacks policies and
procedures for using these criteria to analyze, select, and maintain
the investment portfolio.
Type of practice: Prerequisites;
Key practice: 1. Adequate resources, including people, funding, and
tools, are provided for the process of creating the portfolio;
Rating: Executed;
Summary of evidence: According to HHS IT officials, adequate resources
have been committed for portfolio selection criteria activities. They
include the Critical Partners, portfolio management tool project
manager, and Office of the CIO staff.
Key practice: 2. Board members are knowledgeable about the process of
creating a portfolio;
Rating: Executed;
Summary of evidence: HHS ITIRB members are knowledgeable about the
process of creating a portfolio;
they have now gone through the process twice.
Key practice: 3. The investment board is provided with information
comparing project and system performance with expectations;
Rating: Not executed;
Summary of evidence: While the investment board is provided with
information comparing HHS enterprisewide project and system performance
with expectations, it is not provided with information comparing the
performance of component agency investments against expectations.
Type of practice: Activities;
Key practice: 1. Each IT investment board examines the mix of new and
ongoing investments and their respective data and analyses and selects
investments for funding;
Rating: Executed;
Summary of evidence: The ITIRB examines a mix of new and ongoing
investments through the department's portfolio management tool, which
is used to analyze, prioritize, and select investments for funding.
Key practice: 2. Each investment board approves or modifies the
performance expectations for its selected IT investments;
Rating: Not executed;
Summary of evidence: While the HHS ITIRB approves the performance
expectations for its enterprisewide IT investments, it does not have a
similar process for approving the performance expectations for
component agency IT investments or ensuring that this is done.
Key practice: 3. Information used to select, control, and evaluate the
portfolio is captured and maintained for future reference;
Rating: Not executed;
Summary of evidence: Although HHS is capturing investment information,
the information is not yet used to fully support control and evaluate
decisions about component agency investments.
Source: GAO.
[End of table]
Criteria for Portfolio Performance Evaluations Are Not Yet Developed or
Regularly Modified:
This critical process builds upon the Stage 2 critical process,
Providing Investment Oversight, by adding the elements of portfolio
performance to an organization's investment control capacity. Compared
with less mature organizations, Stage 3 organizations will have the
foundation they need to control the risks faced by each investment and
to deliver benefits that are linked to mission performance. In
addition, a Stage 3 organization will have the benefit of performance
data generated by Stage 2 processes. Executive-level oversight of risk
management outcomes and incremental benefit accumulation provides the
organization with increased assurance that each IT investment will
achieve the desired results. (The complete list of key practices is
provided in table 13.)
HHS has not executed any of the seven key practices for evaluating a
portfolio. It has yet to develop policies and procedures that address
performance oversight from a portfolio perspective. Moreover, while the
department annually reviews its portfolio as part of its selection
process, it does not evaluate the investment portfolio on a continuing
basis to assess its performance. Finally, the results of Providing
Investment Oversight reviews from Stage 2 are important to this
critical process. However, as previously mentioned, while the HHS ITIRB
has oversight of enterprisewide investments, it does not regularly
review a defined set of component agencies' investments and maintain
visibility of other investments. Although the department's portfolio
management tool has the ability to summarize performance metrics for
each investment and quickly understand the status of each investment
and any potential emerging problem area, the tool is currently only
being used on an ad hoc basis to make portfolio oversight decisions.
Defining and implementing processes to evaluate the performance of its
entire portfolio would provide HHS with greater assurance that it is
controlling the risks and achieving the benefits associated with the
mix of investments it has selected.
Table 13 shows the rating for each key practice required to evaluate
the portfolio and summarizes the evidence that supports these ratings.
Table 13: Evaluating the Portfolio:
Type of practice: Organizational commitment;
Key practice: 1. The organization has documented policies and
procedures for reviewing, evaluating, and improving the performance of
its portfolio(s);
Rating: Not executed;
Summary of evidence: HHS does not have policies and procedures for
reviewing, evaluating, and improving the performance of its portfolio.
Type of practice: Prerequisites;
Key practice: 1. Adequate resources, including people, funding, and
tools have been provided for reviewing the investment portfolio and its
projects;
Rating: Not executed;
Summary of evidence: Although HHS annually reviews its portfolio as
part of its selection process, it does not evaluate the performance on
a continuing basis.
Key practice: 2. Board members are familiar with the process for
evaluating and improving the portfolio's performance;
Rating: Not executed;
Summary of evidence: Although HHS annually reviews its portfolio as
part of its selection process, it does not evaluate the performance on
a continuing basis.
Key practice: 3. Results of relevant Providing Investment Oversight
reviews from Stage 2 are provided to the investment board;
Rating: Not executed;
Summary of evidence: While the HHS ITIRB has oversight of
enterprisewide investments, it does not effectively oversee its
component agency IT investments.
Key practice: 4. Criteria for assessing portfolio performance are
developed, reviewed, and modified at regular intervals to reflect
current performance expectations;
Rating: Not executed;
Summary of evidence: HHS does not have criteria for assessing portfolio
performance.
Type of practice: Activities;
Key practice: 1. IT portfolio performance measurement data are defined
and collected consistent with portfolio performance criteria;
Rating: Not executed;
Summary of evidence: HHS does not have criteria for assessing portfolio
performance.
Key practice: 2. Adjustments to the IT investment portfolio are
executed in response to actual portfolio performance;
Rating: Not executed;
Summary of evidence: Although HHS annually reviews its portfolio as
part of its selection process, it does not evaluate the performance on
a continuing basis.
Source: GAO.
[End of table]
Process for Conducting Postimplementation Reviews Is Not Defined:
The purpose of a PIR is to evaluate an investment after it has
completed development (that is, after its transition from the
implementation phase to the operations and maintenance phase) in order
to validate actual investment results. This review is conducted to (1)
examine differences between estimated and actual investment costs and
benefits and possible ramifications for unplanned funding needs in the
future and (2) extract "lessons learned" about the investment selection
and control processes that can be used as the basis for management
improvements. Similarly, PIRs should be conducted for investment
projects that were terminated before completion, to readily identify
potential management and process improvements. (The complete list of
key practices is provided in table 14.)
HHS has not executed the six key practices for conducting PIRs.
Although its policy calls for postimplementation reviews of IT
investments that have recently completed implementation of the entire
investment or a significant phase of the investment, the department
does not have specific procedures for conducting such reviews,
including specifying who conducts and participates in the PIR, what
information is presented in a PIR, or how results are to be
disseminated to decision makers. To date, HHS has conducted closeout
reviews of two enterprisewide investments following their
implementation;
however, while these reports do cover investment cost expectations,
they cannot be considered PIRs because the reports do not address
general conclusions, lessons learned, or schedule deviations. Unless
PIRs are conducted on a regular basis, HHS will not be able to
effectively evaluate the results of its IT investments to determine
whether continuation, modification, or termination of an IT investment
would be necessary in order to meet stated HHS mission objectives.
Table 14 shows the rating for each key practice required to conduct
PIRs and summarizes the evidence that supports these ratings.
Table 14: Conducting Postimplementation Reviews:
Type of practice: Organizational commitment;
Key practice: 1. The organization has documented policies and
procedures for conducting PIRs;
Rating: Not executed;
Summary of evidence: Although, HHS has policy for conducting PIRs, the
department does not have associated procedures for conducting such
reviews.
Type of practice: Prerequisites;
Key practice: 1. Adequate resources, including people, funding, and
tools, have been provided for conducting PIRs;
Rating: Not executed;
Summary of evidence: HHS is not conducting PIRs.
Key practice: 2. Individuals assigned to the investment board to
conduct PIRs should be familiar with both the policies and the
procedures for conducting such reviews;
Rating: Not executed;
Summary of evidence: HHS is not conducting PIRs.
Type of practice: Activities;
Key practice: 1. The investment board identifies which projects will
have a PIR conducted;
Rating: Not executed;
Summary of evidence: HHS is not conducting PIRs.
Key practice: 2. Quantitative and qualitative investment data are
collected, evaluated for reliability, and analyzed during the PIRs;
Rating: Not executed;
Summary of evidence: HHS is not conducting PIRs.
Key practice: 3. Lessons learned and recommendations for improving the
investment process are developed during the PIR, documented, and then
distributed to all stakeholders;
Rating: Not executed;
Summary of evidence: HHS is not conducting PIRs.
Source: GAO.
[End of table]
HHS Has Provided Limited Guidance to and Oversight of Component
Agencies' Investment Management Processes:
The ability of a department-level CIO to effectively oversee IT
investment management processes throughout the agency depends on the
existence of appropriate management structures with adequate
authorities and sufficient guidance. Under the Clinger-Cohen Act of
1996, the CIO of each agency is responsible for effectively managing
all of the agency's IT resources. To comply with the act, HHS
designates its CIO to be responsible for ensuring that the component
agencies are defining and implementing effective investment management
processes that are appropriately aligned with the department's
processes.
Although each component agency has staff responsible for gathering,
maintaining, and analyzing IT investment information, the HHS Office of
the CIO has the responsibility to define and implement overall HHS IT
investment management practices, and monitor component agency
investment management practices to ensure a cohesive departmental
process and the capability exists to carry out the process. In
accordance with this, the department's investment management policies
and guidelines state that the component agencies are to establish and
manage investment management processes and governance structures that
are aligned with the department's policies and procedures. However, as
mentioned in previous sections, the department's investment management
policies and procedures have several weaknesses. For example, HHS does
not have a set of documented procedures that provide decision makers
with a clear understanding of the selection and reselection process.
Moreover, HHS currently has no structured mechanism in place to ensure
that the component agencies are adhering to the department's policies
and procedures. According to HHS officials, the CIO has the authority
to audit a component agencies IT investment management process.
However, they were unable to provide us evidence of having performed
any such audits. These officials also stated that the department's
portfolio management tool is another method that will enable HHS to
oversee component-level investment management processes. However, since
not all component agencies are using the portfolio management tool to
individually make select, control, and evaluate decisions, its
usefulness in this regard is limited. Until the department develops a
mechanism for ensuring that component agencies define and implement
investment management processes that align with those of the
department, it is running the risk that effective processes are being
institutionalized at both the department and the component agency
level. In addition, the department will be unable to ensure that it is
optimizing its investments in IT and effectively assessing and managing
the risks of these investments.
HHS Does Not Have a Plan to Coordinate and Guide Improvement Efforts:
HHS has initiated several efforts to improve its investment management
process. Specifically, it has drafted a revised investment management
guide that addresses the weaknesses with current guidance that we
identify in this report. In addition, in February 2005, HHS
incorporated capabilities into its portfolio management tool to enhance
performance of control and evaluate functions. Specifically, the tool
now has the capabilities to produce (1) scorecards to provide data for
each investment in a portfolio, allowing cross investment comparisons
on data elements collected; (2) investor maps to provide a graphical
depiction of a portfolio in terms of up to six data categories, with
the ability to show target and actual values; and (3) a workbook module
to track the identification and resolution of issues that may arise
regarding the management of an investment or set of investments.
Although HHS has initiated these efforts, they only fully address 2 of
the 14 Stage 2 key practices the department did not execute.
* The draft investment management guidance, when finalized, will
address weaknesses associated with one of the key practices for
instituting the investment board by reflecting the current management
process, including information on the roles of key working groups
involved in the organization's IT investment processes, and identifying
the manner in which investments board's processes are to be coordinated
with other key organizational plans and processes. The guidance will
also address the integration of the funding and selection processes, a
key practice the department has not executed that is associated with
selecting an investment.
* The enhanced portfolio management tool capabilities will enhance the
department's ability to oversee investments' performance and position
the board to perform portfolio evaluation activities, but they will not
fully address any of the weaknesses we identify.
HHS has not coordinated these and additional efforts that would address
the weaknesses we identify in this report in a comprehensive plan that
(1) specifies measurable goals, objectives, and milestones; (2)
specifies needed resources; (3) assigns clear responsibility and
accountability for accomplishing tasks; and (4) is approved by senior
management. We have previously reported that such a plan is
instrumental in helping agencies coordinate and guide improvement
efforts. Until HHS develops a plan that would allow for the systematic
prioritization, sequencing, and evaluation of improvement efforts, the
agency risks not being able to effectively establish the mature
investment management processes that result in greater certainty about
the outcomes of future IT investments.
Conclusions:
Because of the attention that has been given to investment management,
HHS has established several of the practices needed to effectively
manage its investments. These practices have strengthened the
department's basic capabilities for selecting and controlling projects
and begun to equip the department with the capabilities it needs to
make informed decisions about competing investments. However, several
significant weaknesses remain in the foundational practices needed to
manage individual investments, the portfolio-level investments needed
to manage investments as a collection, and in the level of guidance and
oversight provided to component agency investment management processes.
These weaknesses hamper the department's ability to ensure that it is
managing the mix of investments that will maximize returns to the
organization, taking into account the appropriate level of risk.
Critical to HHS's success, going forward will be the development of an
implementation plan that (1) is based on an assessment of strengths and
weaknesses; (2) specifies measurable goals, objectives, and milestones;
(3) specifies needed resources; (4) assigns clear responsibility and
accountability for accomplishing tasks; and (5) is approved by senior
management. Although the department has initiated improvement efforts,
it has not developed a comprehensive plan to guide these and other
efforts needed to improve its investment management process. Without
such a plan and procedures for implementing it, it is unlikely that the
department will effectively establish mature investment management
capability. As a result, HHS will continue to be challenged in its
ability to make informed and prudent investment decisions in managing
its annual multibillion-dollar IT budget.
Recommendations for Executive Action:
To strengthen HHS's investment management capability and address the
weaknesses discussed in this report, we recommend that the Secretary of
the Department of Health and Human Services direct the Chief
Information Officer to develop and implement a plan for improving the
department's IT investment management processes. The plan should
address the weaknesses described in this report, beginning with those
we identified in our Stage 2 analysis and continuing with those we
identified in our Stage 3 analysis. The plan should, at a minimum,
provide for accomplishing the following:
In Stage 2:
* Develop comprehensive guidance and additional supporting guidance
that defines and describes the complete investment management process,
unifies existing processes enterprisewide, reflects changes in
processes as they occur; define the operations and decision-making
processes of the HHS investment review board and other management
entities, such as the component agencies, involved in managing IT
investments.
* Ensure that HHS's investment review board's membership includes
business representation of its component agencies as it begins to
execute its full range of responsibilities.
* Develop well-defined and disciplined written procedures that outline
the process for selecting new IT proposals, reselecting ongoing IT
investments, and integrating funding with the process of selecting an
investment.
* Establish a process for the investment board to regularly review and
track the performance of a defined set of component agency IT systems
against expectations, and take corrective actions when these
expectations are not being met; and establish a mechanism for
maintaining visibility into other investments.
In Stage 3:
* Develop and implement policies and procedures for modifying IT
portfolio selection criteria.
* Develop policies and procedures for using the portfolio selection
criteria to create its portfolio.
* Develop, review, and modify criteria for assessing portfolio
performance at regular intervals to reflect current performance
expectations.
* Define and implement processes for carrying out PIRs for all IT
investments.
We also recommend that the HHS Secretary direct the CIO to ensure that
the plan draws together ongoing efforts and additional efforts that are
needed to address the weaknesses identified in this report. The plan
should also (1) specify measurable goals, objectives, and milestones;
(2) specify needed resources; (3) assign clear responsibility and
accountability for accomplishing tasks; and (4) be approved by senior
management.
Finally, to improve the department oversight of its component agency
investment management process, we are recommending that the HHS
Secretary direct the HHS CIO to establish a mechanism for ensuring
component agencies define and implement investment management processes
that are aligned with those of the department.
Agency Comments:
The Department of Health and Human Services's Inspector General
provided written comments on a draft of this report (reprinted in app.
II). In these comments, HHS generally agreed with our findings and
recommendations and stated that the report represented a fair
assessment of the department's progress in IT investment management.
The department added that it will leverage the report in its efforts to
improve its investment management processes.
HHS expressed differing perspectives on the inclusion of component
agency business representation on the investment review board and the
performance of postimplementation reviews. Specifically, regarding
business representation on the board, the department commented that it
used a hierarchy of investment reviews (with the first review occurring
at the component agency) combined with ITIRB members representing
mission support areas, such as Finance, Acquisition, and Human
Resources, to provide a structure for making the business decisions
regarding the department's investments. We disagree with the department
that this arrangement provides an adequate structure for managing the
department's investments. Because allocating resources among major IT
investments may require fundamental trade-offs among a multitude of
business objectives, portfolio management decisions are essentially
business decisions, and therefore require sufficient business
representation on the board. CIOs and executives responsible for
mission-support functions do not constitute sufficient business
representation because, by virtue of their responsibilities, they are
not in the best position to make business decisions. Portfolio
management decisions are better made by executives with business line
decision-making authority.
Regarding PIRs, HHS commented that it was currently informally
performing them by conducting closeout reviews of recently implemented
investments and annual reviews of systems in operations and
maintenance. PIRs are conducted to determine whether cost, benefit,
schedule, and risk expectations that were set for investments were
achieved and develop lessons learned about the investment selection and
control processes that can be used as the basis for management
improvements. However, neither the closeout reviews, nor the reviews of
systems in operations and maintenance, are addressing all these
elements. Specifically, as we stated in our report, the closeout
reviews do not address schedule deviations, determine whether the
benefits were achieved, or identify lessons learned. In addition, the
reviews of projects in operations and maintenance do not capture the
benefits realized or identify lessons learned.
Commenting on departmental-level oversight of component agency
investments, HHS stated that it agrees with our recommendation to
improve its oversight of component agency investments. It stated that
it would use a number of mechanisms to do this, including performing
audits to ensure alignment of component agency's processes with those
of the department, using earned value management data to identify
potential performance problems with most investments, and directly
reviewing investments determined to be of high priority. We agree with
HHS that these steps would help address some of the weaknesses in
project oversight that we identify in this report.
As agreed with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days
from the date of this report. At that time, we will send copies to
other interested congressional committees, the Secretary of Health and
Human Services, and other interested parties. We will also make copies
available to others upon request. In addition, the report will be
available at no charge on the GAO Web site at [Hyperlink,
http://www.gao.gov].
Should you or your offices have questions on matters discussed in this
report, please contact me at (202) 512-9286 or [Hyperlink,
pownerd@gao.gov]. Contact points for our Offices of Congressional
Relations and Public Affairs may be found on the last page of this
report. GAO staff who made major contributions to this report are
listed in appendix III.
Sincerely yours,
Signed by:
David A. Powner:
Director, Information Technology Management Issues:
[End of section]
Appendixes:
Appendix I: Objectives, Scope, and Methodology:
The objectives of our review were to (1) assess the Department of
Health and Human Services's capabilities for managing its IT
investments and (2) determine any plans HHS might have for improving
those capabilities.
To address our first objective, we reviewed the results of the
department's self-assessment of Stages 2 and 3 practices using our ITIM
framework and validated and updated the results of the self-assessment
through document reviews and interviews with officials. We reviewed
written policies, procedures, and guidance and other documentation
providing evidence of executed practices, including HHS's Capital
Planning and Investment Control Policy and Guidelines, standard
operating procedures, portfolio management tool training manuals, and
various instructional memorandums. We also reviewed the HHS ITIRB
meeting materials, including quarterly status reports, meeting minutes,
and records of decisions. We did not assess progress in establishing
the capabilities found in Stages 4 and 5 because the department
acknowledged that it had not executed any of the key practices in
higher maturity stages. In addition, we conducted interviews with
officials from the Office of the CIO, whose main responsibility is to
oversee and ensure that HHS's IT investment management process is
implemented and followed to determine the level of oversight and
guidance the department is providing to its component agencies. We also
interviewed the Centers for Medicare & Medicaid's Director for
Investment Tracking and Assessment to determine the level of investment
management guidance and oversight that is provided by the department.
As part of our analysis, we selected two HHS enterprisewide and two
component agency IT projects as case studies to verify that the
critical processes and key practices were being applied. The projects
selected (1) are recognized as major systems, (2) were in different
life cycle phases, (3) represent a mix of headquarters and component
agency investments, (4) support different functional areas, and (5)
required different levels of funding. The four projects are described
below:
* HHS Public Key Infrastructure--This project supports digital
signatures and other public key-enabled security services; it is
intended to be the underlying architecture to support secure
transmissions of electronic communication, such as encrypted email, by
linking a digital key to a specific person, and issues and manages
digital certificates. The intent of the project is to provide an
identity proofing process that is both fast and certificate authority
neutral. It is an agencywide strategic initiative that provides
security services. The project is a major enterprisewide investment and
is in the operations and maintenance phase. The project has a planned
completion date of July 2011 and is estimated to spend $7.7 million for
fiscal year 2006.
* HHS Enterprise Architecture Initiative--This initiative is to provide
the overall framework for planning and managing the technology-
supported information assets of HHS and give the department the ability
to identify data and process redundancies and inefficiencies in its
information systems. The program's objectives focus on development of
operational policies and support that enable identification, analysis
and ongoing management of the business, and information and related
technology architectures. It is to provide leadership, direction, and
support to HHS's component agencies in planning and implementing
information systems to support required business processes. As of
fiscal year 2005, the initiative is a major enterprisewide program
investment and is estimated to spend $15.0 million for fiscal year
2006.
* National Institutes of Health's Electronic Research Administration--
This initiative is the National Institutes of Health's infrastructure
for conducting interactive electronic transactions for the receipt,
review, monitoring, and administration of grant awards to biomedical
investigators worldwide. It is to provide the technology capabilities
for the agency to efficiently and effectively perform grants
administration functions. The system is to provide end-to-end support
of the grants administration process, including receipt of
applications, review and selection of grantees, financial and progress
reporting, issuance of final reports and grant dole-out, invention
reporting, and interface with accounting systems. It is a major
component agency investment and is expected to have a useful life of 13
years. The project is estimated to spend $42.1 million for fiscal year
2006.
* Food and Drug Administration's Mission Accomplishment and Regulatory
Compliance Services--This program is a comprehensive redesign and
reengineering of core mission-critical systems at the agency, including
the Field Accomplishments and Compliance Tracking System and the
Operation and Administration Support System. The first of these systems
is to support the investigation, tracking of compliance, and laboratory
operations related to domestic operations under the agency's purview;
the second is to primarily support the review and decision-making
process of products imported into the United States. Both are legacy
systems that execute on client-server platforms; while currently
viable, the current systems cannot address many of the business needs
due to the exponential growth in functionality on a rigid platform that
was not designed to support the extent of change that has been
required. The Mission Accomplishment and Regulatory Compliance Services
is a major component agency investment and is expected to move to
production in September 2007 and have a useful life of 10 years. The
project is estimated to spend $10.2 million for fiscal year 2006.
For these projects, we reviewed project management documentation, such
as business cases, status reports, and meeting minutes. We also
interviewed officials from the Office of the CIO for the two component
agency investments and the project managers for the two HHS
enterprisewide projects.
We compared the evidence collected from our document reviews and
interviews to the key practices in ITIM. We rated the key practices as
"executed" on the basis of whether the agency demonstrated (by
providing evidence of performance) that it had met the criteria of the
key practice. A key practice was rated as "not executed" when we found
insufficient evidence of a practice during the review or when we
determined that there were significant weaknesses in HHS's execution of
the key practice. In addition, HHS was provided the opportunity to
produce evidence for key practices rated as "not executed."
To address our second objective, we obtained and evaluated documents
showing what management actions had been taken and what initiatives had
been planned by the agency. This documentation included the Policy
Advisory Board charter, draft investment management policies and
procedures, as well as procedures and guidance for control and evaluate
functionalities within HHS's portfolio management tool. We also
interviewed officials from the Office of the CIO to determine efforts
undertaken to improve IT investment management processes.
We conducted our work at HHS headquarters in Washington, D.C., from
January through September 2005, in accordance with generally accepted
government auditing standards.
[End of section]
Appendix II Comments from the Department of Health and Human Services:
Department Of Health & Human Services:
Office of Inspector General:
Washington, D.C. 20201:
October 4, 2005:
Mr. David A. Powner:
Director:
Information Technology Management Issues: U.S. Government
Accountability Office: Washington, DC 20548:
Dear Mr. Powner:
Enclosed are the Department's comments on the U.S. Government
Accountability Office's (GAO's) draft report entitled, "INFORMATION
TECHNOLOGY: HHS Has Several Investment Management Capabilities in
Place, but Needs to Address Key Weaknesses" (GAO-06-11). These comments
represent the tentative position of the Department and are subject to
reevaluation when the final version of this report is received.
The Department appreciates the opportunity to comment on this draft
report before its publication.
Sincerely,
Signed by:
Daniel R. Levinson:
Inspector General:
Enclosure:
The Office of Inspector General (OIG) is transmitting the Department's
response to this draft city' as the Department's designated focal point
and coordinator for U.S. Government Accountability Office reports. OIG
has not conducted an assessment of these comments and therefore
expresses no opinion:
COMMENTS OF THE U.S. DEPARTMENT OF HEALTH AND HUMAN SERVICES ON THE
U.S. GOVERNMENT ACCOUNTABILITY OFFICE'S DRAFT REPORT ENTITLED,
"INFORMATION TECHNOLOGY: HHS HAS SEVERAL INVESTMENT MANAGEMENT
CAPABILITIES IN PLACE, BUT NEEDS TO ADDRESS KEY WEAKNESSES" (GAO-06-
11):
The Department of Health and Human Services (HHS) appreciates GAO's
efforts to independently assess the Department's Information Technology
(IT) investment management capabilities and the opportunity to respond
to your draft report.
The GAO report acknowledges the management attention and significant
improvements HHS has made in this area over the past two years. During
that time, HHS has implemented an IT portfolio management tool (PMT)
and begun reengineering its Capital Planning and Investment Control
(CPIC) processes. Particular strides have been made in integrating the
CPIC process with budget formulation and prioritizing the Department's
IT investments in terms of strategic alignment, value, risk, and
performance during fiscal years (FY) 2006 and 2007 budget cycles.
HHS has taken what is essentially a rapid prototype development
approach to improving its IT investment management. We have focused on
changing actual practices and leveraging the information sharing and
analytical capabilities available through the PMT. These efforts were
applied in the FY 2006 budget process and lessons learned were applied
in the FY 2007 budget cycle. We have deliberately postponed formal
documentation of the process until some experience was gained in using
the process.
Many of GAO's recommendations to HHS center on providing that
documentation.
Now that we have had the benefit of two years' experience with improved
processes, HHS intends to issue policies and procedures in the near
term. We agree with GAO that better documentation of evolving policies
and procedures will help to institutionalize the processes and better
ensure consistent optimal decisionmaking regarding IT investments.
GAO's assessment will be helpful to HHS in preparing the documentation
and focusing our efforts as the Department continues to improve its IT
investment management processes.
Although the Department is in agreement with the majority of GAO's
findings and recommendations, we offer some differing perspectives in
the following areas:
* Inclusion of Operating Division (OPDIV) business representation on
the Department-level IT Investment Review Board (ITIRB). HHS believes
that the intent of this recommendation is to ensure that subject matter
expertise is available in the targeted areas of investment that come
before the ITIRB to provide perspective on the efficacy of the approach
being proposed in that investment and to further ensure that the
subject approach will have a reasonable opportunity to produce the
benefits for which the investment is being made. HHS concurs with the
intent of this recommendation but has chosen to pursue the intended
result using a different approach due to the size and diversity of
business/mission activities for which the agencies that compose HHS
hold responsibility. HHS has hundreds of business/mission programs with
an extremely diverse mix. To have subject matter expertise in each and
every business/mission area that is the responsibility of HHS would
make the Department level ITIRB so large as to become unmanageable and
ineffective. No substantial discussion would be so relevant to the
entire group that any level of detail for a particular investment could
be understood, nor would the majority of the group understand how their
business/mission related to the investment being discussed. To achieve
relevancy of discussion, alignment to business goals and objectives,
and understanding of impact and relationship to supporting investments
requires subject matter expertise that is conversant in the subject at
hand. The HHS approach of establishing a hierarchy of reviews allows
the first level of review to occur in the agency that has direct
responsibility for the success of that investment in support of the
business/mission for which they themselves are the owners. This allows
for a number of subject matter experts that have a vested interest in
the outcomes being pursued by a particular investment and who fully
understand the impact of a particular approach to evaluate its efficacy
at a detailed level. As major investments move up to the Departmental
ITIRB, business decisions regarding the mix of investments to be made
in support of particular goals and objectives can be made with an
assurance that the efficacy of the approach has been validated by
subject matter experts. The Departmental ITIRB reserves the authority
to call these experts before them to answer any questions. This allows
the Departmental ITIRB to evaluate whether it is a good business
decision to make an investment based on its relative value to the
Department, which is why the Departmental ITIRB is composed of, in
addition to each Chief Information Officer of the agencies that make up
HHS, the Departmental executives for Finance, Acquisition, Human
Resources, Budget, etc. HHS fully agrees with the recommendation that
the HHS CIO should provide periodic reports on IT investment portfolio
priorities and performance to senior Department executives, to include
the OPDIV heads, and will work to that end.
Department-level review and tracking of the performance of a defined
set of OPDIV IT systems. The Department agrees with GAO's
recommendation that the Department should provide improved oversight
over OPDIV IT investment management processes. HHS further agrees that
the Department should review any high risk or under performing OPDIV-
specific IT investments. If an appropriate and aligned OPDIV process is
established and adequate audits are in place to ensure continued
compliance, then the Department should be able to generally rely on
that process to provide adequate oversight to OPDIV-level investments.
HHS collects and analyzes earned value data on all HHS Departmental and
OPDIV major and tactical IT investments. The Department ITIRB will use
that information to identify potential performance problems in OPDIV IT
investments. HHS intends to manage OPDIV-level investments by
exception. Typically, specific investments that are of sufficiently
high priority or that have performance problems that place them on a
Departmental "watch list" will be elevated for detailed Department-
level review. This would allow for the set of OPDIV IT systems under
review at the Departmental level to evolve as corrective actions are
successful and project performance improves. Using this approach the
Departmental ITIRB can focus its attention where it is needed most.
Define and implement processes for carrying out Post Implementation
Review (PIRs). HHS concurs with the recommendation to better document
the policies and procedures regarding PIRs and the evaluation of steady-
State IT investments. However, HHS believes that the implication that
HHS does not perform those functions now is incomplete. Although the
process is less formal than it should be, closeout reviews of recently
implemented investments are conducted by the HHS ITIRB to identify
lessons learned for application to future investments. Each steady-
State investment is also required to provide an annual report to the
ITIRB discussing its ability to meet continuing or evolving business
needs, the ability and need for technology upgrades or enhanced
functionality, cost/benefit analysis, and a number of other aspects
that are appropriate for managing ongoing investments. HHS does agree
that there is opportunity for much improvement in this area, which has
always been the HHS plan, but feels that the Department is already
doing some of this activity in an informal manner.
Overall, HHS finds the GAO's report on HHS IT Investment Management
capabilities to represent a fair assessment of the Department's
progress in this area. HHS will leverage this report in the
Department's continuing efforts to improve IT investment management.
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
David A. Powner (202) 512-9286, [Hyperlink, pownerd@gao.gov]
Staff Acknowledgments:
In addition to the person named above, Neil Doherty, Joanne Fiorino,
Sabine Paul, Nik Rapelje, Niti Tandon, and Amos Tevelow made key
contributions to this report.
(310451):
FOOTNOTES
[1] Office of Management and Budget, Budget of the U.S. Government,
Fiscal Year 2006, Report on IT Spending for the Federal Government for
Fiscal Years 2004, 2005, and 2006. We did not verify these data.
[2] Our second report, GAO, Information Technology: Centers for
Medicare & Medicaid Services Needs to Establish Critical Investment
Management Capabilities, GAO-06-12 (Washington, D.C.: Oct. 28, 2005),
addresses (1) the agency's capabilities for managing its IT
investments, (2) determining any plans the agency might have for
improving these capabilities, and (3) examining the agency's process
for approving and monitoring the state Medicaid management systems it
funds.
[3] GAO, Information Technology Investment Management: A Framework for
Assessing and Improving Process Maturity, GAO-04-394G (Washington,
D.C.: March 2004).
[4] HHS refers to its component agencies as operating divisions.
[5] Enterprisewide initiatives are mission-support and administrative
systems that are used by all component agencies.
[6] GAO, Financial Management Systems: Lack of Disciplined Processes
Puts Implementation of HHS' Financial System at Risk, GAO-04-1008
(Washington, D.C.: Sept. 23, 2004).
[7] GAO, Information Technology: Federal Agencies Face Challenge in
Implementing Initiatives to Improve Public Health Infrastructure, GAO-
05-308 (Washington, D.C.: June 10, 2005).
[8] GAO, Information Technology Management: Governmentwide Strategic
Planning, Performance, Measurement, and Investment Management Can Be
Further Improved, GAO-04-49 (Washington, D.C.: Jan. 12, 2004).
[9] GAO-05-308.
[10] We did not evaluate HHS administrative processes for managing IT
grants to states because according to officials, both the department
and component agencies CIOs are not directly involved in the approval
or oversight of those IT investments.
[11] According to HHS IT officials, for the fiscal year 2006 budget
formulation, the business cases and Select forms were updated for
investments that represented 80 percent of the entire HHS IT portfolio
dollar value. The remaining 20 percent are nonmajor investments
requesting less than $4.5 million in fiscal year 2006.
[12] These business cases are generally referred to as "exhibit 300s."
[13] The Office of Management and Budget evaluates the business cases
against the following 10 criteria: acquisition strategy, project
(investment) management, enterprise architecture, alternatives
analysis, risk management, performance goals, security and privacy,
performance-based management system, life-cycle costs formulation, and
support the President's Management Agenda.
[14] The department's portfolio management tool was implemented in May
2004 and has not been used yet to support the entire investment
management process.
[15] GAO-04-394G.
[16] GAO, Information Technology: DLA Needs to Strengthen Its
Investment Management Capability, GAO-02-314 (Washington, D.C.: Mar.
15, 2002); GAO, United States Postal Service: Opportunities to
Strengthen IT Investment Management Capabilities, GAO-03-3 (Washington,
D.C.: Oct. 15, 2002); GAO, Information Technology: Departmental
Leadership Crucial to Success of Investment Reforms at Interior, GAO-03-
1028 (Washington, D.C.: Sept. 12, 2003); GAO, Bureau of Land
Management: Plan Needed to Sustain Progress in Establishing IT
Investment Management Capabilities, GAO-03-1025 (Washington, D.C.:
Sept. 12, 2003); and GAO, Information Technology: FAA Has Many
Investment Management Capabilities in Place, but More Oversight of
Operational Systems Is Needed, GAO-04-822 (Washington, D.C.: Aug. 20,
2004).
[17] The Clinger-Cohen Act of 1996, 40 U.S.C. §§ 11311-11313.
[18] An IT investment board is a decision-making body, made up of
senior program, financial, and information managers, that is
responsible for making decisions about IT projects and systems on the
basis of comparisons and trade-offs among competing projects, with an
emphasis on meeting mission goals.
[19] 40 U.S.C. § 11312(b)(1).
[20] According to the ITIM, "new" proposals include both (1) previously
submitted IT proposals that were not originally selected for funding
and (2) IT proposals that have never been submitted.
[21] According to the ITIM, a process is a sequence of steps performed
for a given purpose, and a process guide is a document that
specifically defines the manner in which the general IT investment
guidance will be implemented within the organization.
[22] We reviewed two enterprisewide projects--HHS Public Key
Infrastructure and HHS Enterprise Architecture initiative, and two
component agency projects--National Institutes of Health's Electronic
Research Administration and Food and Drug Administration's Mission
Accomplishment and Regulatory Compliance Services. The projects are
described in appendix I.
[23] HHS conducts quarterly reviews on its enterprisewide investments
during the period of development and annual reviews of its steady state
enterprisewide investments, that is, those systems that have completed
development and become operational.
[24] Earned value management is a project management tool that
integrates the investment scope of work with schedule and cost elements
for investment planning and control. This method compares the value of
work accomplished during a given period with that of the work expected
in the period. Differences in expectations are measured in both cost
and schedule variances.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
