Department of Defense
Further Actions Needed to Institutionalize Key Business System Modernization Management Controls Gao ID: GAO-11-684 June 29, 2011For decades, the Department of Defense (DOD) has been challenged in modernizing its timeworn business systems. Since 1995, GAO has designated DOD's business systems modernization program as high risk. Between 2001 and 2005, GAO reported that the modernization program had spent hundreds of millions of dollars on an enterprise architecture and investment management structures that had limited value. Accordingly, GAO made explicit architecture and investment management-related recommendations. Congress included provisions in the Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005 that were consistent with GAO's recommendations and required GAO to assess DOD's actions to comply with these provisions. To do so, GAO reviewed documents and interviewed military officials on the progress the military departments have made relative to developing their respective parts of the federated business enterprise architecture and establishing investment management structures and processes.
DOD continues to take steps to comply with the act's provisions and to satisfy relevant system modernization management guidance. Collectively, these steps address several statutory provisions and best practices concerning the business enterprise architecture, budgetary disclosure, and review of systems costing in excess of $1 million. However, long-standing challenges that GAO previously identified remain to be addressed in order for DOD to be in compliance with guidance and the act. In particular, (1) While DOD continues to release updates to its enterprise architecture, the architecture has yet to be augmented by a coherent family of component architectures. In this regard, each of the military departments has made progress in managing its respective enterprise architecture program since GAO last reported in 2008. However, each has yet to address key elements, including developing the architecture content, to advance to a level that could be considered mature. For example, while each department has established or is in the process of establishing an executive committee with responsibility and accountability for the enterprise architecture, none has fully developed an enterprise architecture methodology or a well-defined business enterprise architecture and transition plan to guide and constrain business transformation initiatives. (2) DOD continues to establish investment management processes, but neither DOD-level organizations nor the military departments have defined the full range of project-level and portfolio-based IT investment management policies and procedures that are necessary to meet the investment selection and control provisions of the Clinger-Cohen Act of 1996. Specifically, with regard to project-level practices, DOD enterprise, Air Force, and Navy have yet to fully define 56 percent of the practices, and Army has yet to do so for 78 percent of the practices. With regard to the portfolio-level practices, DOD enterprise, Air Force, and Navy have yet to fully define 80 percent and Army has yet to do so for any of the practices. In addition, while DOD largely followed its certification and oversight processes, key steps were not performed. For example, as part of the certification process, DOD performed three process assessments specified in DOD guidance, such as assessing investment alignment with the architecture, but did not validate the results of the assessment, thus increasing the risk that certification decisionmaking was based on inaccurate and unreliable information. It is essential that DOD address GAO's existing recommendations aimed at addressing these long-standing challenges, as doing so is critical to the department's ability to establish the full range of institutional management controls needed to address its business systems modernization high-risk program. Department officials attributed the state of progress in part to the uncertainty and pending decisions surrounding the roles and responsibilities of key organizations and senior leadership positions as well as the lack of resources (i.e., people and funding). Because GAO has existing recommendations that address the long-standing challenges discussed in this report, it is making no further recommendations in these areas. GAO is recommending that DOD complete the implementation of the reorganization of key organizations. DOD agreed with GAO's recommendation.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Valerie C. Melvin Team: Government Accountability Office: Information Technology Phone: (202) 512-6304GAO-11-684, Department of Defense: Further Actions Needed to Institutionalize Key Business System Modernization Management Controls
This is the accessible text file for GAO report number GAO-11-684
entitled 'Department of Defense: Further Actions Needed to
Institutionalize Key Business System Modernization Management
Controls' which was released on June 29, 2011.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
United States Government Accountability Office:
GAO:
Report to Congressional Committees:
June 2011:
Department of Defense:
Further Actions Needed to Institutionalize Key Business System
Modernization Management Controls:
GAO-11-684:
GAO Highlights:
Highlights of GAO-11-684, a report to congressional committees.
Why GAO Did This Study:
For decades, the Department of Defense (DOD) has been challenged in
modernizing its timeworn business systems. Since 1995, GAO has
designated DOD‘s business systems modernization program as high risk.
Between 2001 and 2005, GAO reported that the modernization program had
spent hundreds of millions of dollars on an enterprise architecture
and investment management structures that had limited value.
Accordingly, GAO made explicit architecture and investment management-
related recommendations. Congress included provisions in the Ronald W.
Reagan National Defense Authorization Act for Fiscal Year 2005 that
were consistent with GAO‘s recommendations and required GAO to assess
DOD‘s actions to comply with these provisions. To do so, GAO reviewed
documents and interviewed military officials on the progress the
military departments have made relative to developing their respective
parts of the federated business enterprise architecture and
establishing investment management structures and processes.
What GAO Found:
DOD continues to take steps to comply with the act‘s provisions and to
satisfy relevant system modernization management guidance.
Collectively, these steps address several statutory provisions and
best practices concerning the business enterprise architecture,
budgetary disclosure, and review of systems costing in excess of $1
million. However, long-standing challenges that GAO previously
identified remain to be addressed in order for DOD to be in compliance
with guidance and the act. In particular,
* While DOD continues to release updates to its enterprise
architecture, the architecture has yet to be augmented by a coherent
family of component architectures. In this regard, each of the
military departments has made progress in managing its respective
enterprise architecture program since GAO last reported in 2008.
However, each has yet to address key elements, including developing
the architecture content, to advance to a level that could be
considered mature. For example, while each department has established
or is in the process of establishing an executive committee with
responsibility and accountability for the enterprise architecture,
none has fully developed an enterprise architecture methodology or a
well-defined business enterprise architecture and transition plan to
guide and constrain business transformation initiatives.
* DOD continues to establish investment management processes, but
neither DOD-level organizations nor the military departments have
defined the full range of project-level and portfolio-based IT
investment management policies and procedures that are necessary to
meet the investment selection and control provisions of the Clinger-
Cohen Act of 1996. Specifically, with regard to project-level
practices, DOD enterprise, Air Force, and Navy have yet to fully
define 56 percent of the practices, and Army has yet to do so for 78
percent of the practices. With regard to the portfolio-level
practices, DOD enterprise, Air Force, and Navy have yet to fully
define 80 percent and Army has yet to do so for any of the practices.
In addition, while DOD largely followed its certification and
oversight processes, key steps were not performed. For example, as
part of the certification process, DOD performed three process
assessments specified in DOD guidance, such as assessing investment
alignment with the architecture, but did not validate the results of
the assessment, thus increasing the risk that certification
decisionmaking was based on inaccurate and unreliable information.
It is essential that DOD address GAO‘s existing recommendations aimed
at addressing these long-standing challenges, as doing so is critical
to the department‘s ability to establish the full range of
institutional management controls needed to address its business
systems modernization high-risk program. Department officials
attributed the state of progress in part to the uncertainty and
pending decisions surrounding the roles and responsibilities of key
organizations and senior leadership positions as well as the lack of
resources (i.e., people and funding).
What GAO Recommends:
Because GAO has existing recommendations that address the long-
standing challenges discussed in this report, it is making no further
recommendations in these areas. GAO is recommending that DOD complete
the implementation of the reorganization of key organizations. DOD
agreed with GAO‘s recommendation.
View [hyperlink, http://www.gao.gov/products/GAO-11-684] or key
components. For more information, contact Valerie C. Melvin at (202)
512-6304 or melvinv@gao.gov.
[End of section]
Contents:
Letter:
Background:
DOD Continues to Strengthen Management of Its Business Systems
Modernization, but Long-standing Challenges Remain:
Conclusions:
Recommendation for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Objective, Scope, and Methodology:
Appendix II: Comments from the Department of Defense:
Appendix III: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Stage 2 Critical Processes and Associated Key Practices:
Table 2: Stage 3 Critical Processes and Associated Key Practices:
Table 3: DOD Business Systems Modernization Governance Entities'
Roles, Responsibilities, and Composition:
Table 4: DOD Investment Tiers:
Table 5: Description of Progress for Enterprise Architecture Core
Elements Previously Reported as Not Fully Satisfied by One or More
Military Departments:
Table 6: Summary of Key Practices for Stage 2 Critical Processes-
Building the Investment Foundation:
Table 7: Summary of Key Practices for Stage 3 Critical Processes-
Developing a Complete Investment Portfolio:
Figures:
Figure 1: Simplified View of DOD Organizational Structure:
Figure 2: The Five ITIM Stages of Maturity with Critical Processes:
Abbreviations:
ASD(NII)/DOD CIO: Assistant Secretary of Defense for Networks and
Information Integration/Department of Defense Chief Information
Officer:
BEA: business enterprise architecture:
CIO: chief information officer:
DITPR: Defense Information Technology Portfolio Repository:
DOD: Department of Defense:
IRB: investment review board:
IT: information technology:
ITIM: Information Technology Investment Management:
IV&V: independent verification and validation:
NDAA: National Defense Authorization Act:
OMB: Office of Management and Budget:
PMRT: Project Management Resource Tool:
SNAP-IT: Select and Native Programming Data Input System--Information
Technology:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
June 29, 2011:
Congressional Committees:
For decades, the Department of Defense (DOD) has been challenged in
modernizing its timeworn business systems.[Footnote 1] In 1995, we
designated the department's business systems modernization program as
high risk, and we continue to designate it as such today.[Footnote 2]
As our research on public and private sector organizations has shown,
two essential ingredients to a successful systems modernization
program are an effective institutional approach to managing
information technology (IT) investments and a well-defined enterprise
architecture.[Footnote 3] For its business systems modernization, DOD
is developing and using a federated business enterprise architecture,
which is a coherent family of parent and subsidiary architectures, to
help modernize its nonintegrated and duplicative business operations
and the systems that support them.
In May 2001,[Footnote 4] we recommended that the Secretary of Defense
establish the means for effectively developing an enterprise
architecture and a corporate, architecture-centric approach to
investment control and decision making. Yet, between 2001 and 2005, we
reported that the department's business systems modernization program
continued to lack both of these approaches, concluding in 2005 that
hundreds of millions of dollars had been spent on a business
enterprise architecture and investment management structures that had
limited value.[Footnote 5] Accordingly, we made additional, explicit
architecture and investment management-related recommendations to
address these continuing deficiencies.
To further assist DOD in addressing these modernization management
challenges, Congress included provisions in the Ronald W. Reagan
National Defense Authorization Act (NDAA) for Fiscal Year 2005
[Footnote 6] that were consistent with our recommendations. More
specifically, section 332 of the act required the department to, among
other things, (1) develop a business enterprise architecture and a
transition plan for implementing the architecture, (2) identify
systems information in its annual budget submission, (3) establish a
system investment approval and accountability structure along with an
investment review process, and (4) certify and approve any system
modernizations costing in excess of $1 million. The act[Footnote 7]
further required that the Secretary of Defense submit an annual report
to congressional defense committees on DOD's compliance with certain
requirements of the act not later than March 15 of each year, from
2005 through 2013. Additionally, the act directed us to submit to
these congressional committees--within 60 days of DOD's report
submission--an assessment of the department's actions to comply with
these requirements.
Accordingly, as agreed with your office, the objective of our review
was to assess the actions by DOD to comply with the above four
provisions of section 332 of the act. To address the provisions of the
act related to enterprise architecture and investment management, we
focused on the progress the military departments have made relative to
developing their respective parts of the federated business enterprise
architecture and establishing investment management structures and
processes as required by statute, using the results of our prior
reports as a baseline.[Footnote 8] To address the budgetary disclosure
and certification provisions of the act, we reviewed the department's
report to Congress, which was submitted on May 4, 2011, and evaluated
the information used to satisfy the budget submission and investment
review, certification, and approval aspects of the act.
We conducted this performance audit at DOD and military department
offices in Arlington, Virginia, from January to June 2011, in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives. Details on our
objective, scope, and methodology are contained in appendix I.
Background:
DOD is a massive and complex organization entrusted with more taxpayer
dollars than any other federal department or agency. Organizationally,
the department includes the Office of the Secretary of Defense, the
Joint Chiefs of Staff, the military departments, numerous defense
agencies and field activities, and various unified combatant commands
that are responsible for either specific geographic regions or
specific functions. (See figure 1 for a simplified depiction of DOD's
organizational structure.)
Figure 1: Simplified View of DOD Organizational Structure:
[Refer to PDF for image: organizational chart]
Top level:
Secretary of Defense:
Deputy Secretary of Defense[A].
Second level, reporting to: Secretary of Defense; Deputy Secretary of
Defense:
Department of the Army;
Department of the Navy;
Department of the Air Force;
Office of the Secretary of Defense:
- DOD field activities;
- Defense agencies;
Inspector General;
Joint Chiefs of Staff;
Combatant commands[B].
Source: GAO based on DOD documentation.
[A] The Deputy Secretary of Defense serves as the Chief Management
Officer, who provides focused and sustained leadership over DOD's
business transformation efforts.
[B] The Chairman of the Joint Chiefs of Staff serves as the
spokesperson for the commanders of the combatant commands, especially
on the administrative requirements of the commands.
[End of figure]
In support of its military operations, DOD performs an assortment of
interrelated and interdependent business functions, such as logistics
management, procurement, health care management, and financial
management. As we have previously reported,[Footnote 9] the DOD
systems environment that supports these business functions is overly
complex and error prone, and is characterized by (1) little
standardization across the department, (2) multiple systems performing
the same tasks, (3) the same data stored in multiple systems, and (4)
the need for data to be entered manually into multiple systems. The
department recently requested about $17.3 billion for its business
systems environment and IT infrastructure investments for fiscal year
2012. According to the department's systems inventory, this
environment is composed of 2,258 business systems and includes 335
financial management, 709 human resource management, 645 logistics,
243 real property and installation, and 281 weapon acquisition
management systems.
DOD currently bears responsibility, in whole or in part, for 14 of the
30 programs across the federal government that we have designated as
high risk because they are highly susceptible to fraud, waste, abuse,
and mismanagement.[Footnote 10] Seven of these areas are specific to
the department,[Footnote 11] and seven other high-risk areas are
shared with other federal agencies.[Footnote 12] Collectively, these
high-risk areas relate to DOD's major business operations that are
inextricably linked to the department's ability to perform its overall
mission and directly affect the readiness and capabilities of U.S.
military forces and can affect the success of a mission. In
particular, the department's nonintegrated and duplicative systems
impair its ability to combat fraud, waste, and abuse.[Footnote 13] As
such, DOD's business systems modernization is one of the high-risk
areas and is an essential enabler in addressing many of the
department's other high-risk areas. For example, modernized business
systems are integral to the department's efforts to address its
financial, supply chain, and information security management high-risk
areas.
Enterprise Architecture and IT Investment Management Controls Are
Critical to Achieving Successful Systems Modernization:
Effective use of a well-defined enterprise architecture is a hallmark
of successful organizations and a basic tenet of organizational
transformation and systems modernization. Since the early 1990s, we
have promoted federal department and agency enterprise architecture
adoption as an essential means to achieving a desired end: having
operational and technology environments that maximize institutional
mission performance and outcomes.[Footnote 14] Congress, the Office of
Management and Budget (OMB), and the federal Chief Information
Officers (CIO) Council have also recognized the importance of an
architecture-centric approach to modernization. The Clinger-Cohen Act
of 1996, among other things, requires the CIOs of federal departments
and agencies to develop, maintain, and facilitate architectures as a
means of integrating business processes and agency goals with IT.
[Footnote 15] Further, the E-Government Act of 2002 established the
OMB Office of Electronic Government and assigned it, among other
things, responsibility for overseeing the development of enterprise
architectures within and across agencies.[Footnote 16] In addition,
OMB, the CIO Council, and we have issued guidance that emphasizes the
need for system investments to be consistent with these
architectures.[Footnote 17] For example, in April 2003 and in August
2010, we published a framework[Footnote 18] that emphasizes the
importance of having an enterprise architecture as a critical frame of
reference for organizations when they are making IT investment
decisions. Also, in December 2008, OMB issued guidance[Footnote 19]
that addresses system investment compliance with agency architectures.
A corporate approach to IT investment management is another important
characteristic of successful public and private organizations.
Recognizing this, the Clinger-Cohen Act[Footnote 20] requires OMB to
establish processes to analyze, track, and evaluate the risks and
results of major capital investments in IT systems made by executive
agencies.[Footnote 21] In response to the Clinger-Cohen Act and other
statutes, OMB developed policy and issued guidance for planning,
budgeting, acquisition, and management of federal capital assets.
[Footnote 22] We have also issued guidance in this area that defines
institutional structures (such as investment boards), processes for
developing information on investments (such as cost/benefit), and
practices to inform management decisions (such as whether a given
investment is aligned with an enterprise architecture).[Footnote 23]
Enterprise Architecture: A Brief Description:
An enterprise architecture provides a clear and comprehensive picture
of an entity, whether it is an organization (e.g., a federal
department or agency) or a functional or mission area that cuts across
more than one organization (e.g., financial management). An
architecture describes the enterprise in logical terms (such as
interrelated business processes and business rules, information needs
and flows, and work locations and users) as well as in technical terms
(such as hardware, software, data, communications, security
attributes, and performance standards). It provides these perspectives
both for the enterprise's current environment and for its target
environment, and it provides a transition plan for moving from the
current to the target environment. This transition plan provides a
temporal road map for moving between the two environments and
incorporates considerations such as technology opportunities,
marketplace trends, fiscal and budgetary constraints, institutional
system development and acquisition capabilities, legacy and new system
dependencies and life expectancies, and the projected value of
competing investments.
The suite of products adopted for a given entity's enterprise
architecture, including its structure and content, is largely governed
by the framework used to develop the architecture. Since the 1980s,
various architecture frameworks have been developed, such as John A.
Zachman's "A Framework for Information Systems Architecture,"[Footnote
24] and the DOD Architecture Framework.[Footnote 25]
The importance of developing, implementing, and maintaining an
enterprise architecture is a basic tenet of both organizational
transformation and systems modernization. Managed properly, an
enterprise architecture can clarify and help optimize the
interdependencies and relationships among an organization's business
operations and the underlying IT infrastructure and applications that
support these operations. Moreover, when an enterprise architecture is
employed in concert with other important management controls, such as
portfolio-based capital planning and investment control practices, the
architecture can greatly increase the chances that an organization's
operational and IT environments will be configured to optimize mission
performance. The alternative, as our work has shown, is the
perpetuation of the kinds of operational environments that burden many
agencies today, where a lack of integration among business operations
and the IT resources supporting them leads to systems that are
duplicative, poorly integrated, and unnecessarily costly to maintain
and interface.[Footnote 26]
In February 2002 and April 2003, we issued versions 1.0 and 1.1 of our
Enterprise Architecture Management Maturity Framework; in August 2010,
we issued a major revision (version 2.0).[Footnote 27] The framework
provides a standard yet flexible benchmark against which to determine
where the enterprise stands in its progress toward the ultimate goal:
having a continuously improving enterprise architecture program that
can serve as a featured decision support tool when considering and
planning large-scale organizational restructuring or transformation
initiatives. In addition, it also provides a basis for developing
architecture management improvement plans, as well as for measuring,
reporting, and overseeing progress in implementing these plans.
Several approaches to structuring enterprise architecture exist and
can be applied to the extent that they are relevant and appropriate
for a given enterprise. In general, these approaches provide for
decomposing an enterprise into its logical parts and architecting each
of the parts in relation to enterprisewide needs and the inherent
relationships and dependencies that exist among the parts. As such,
the approaches are fundamentally aligned and consistent with a number
of basic enterprise architecture principles, such as incremental
rather than monolithic architecture development and implementation,
optimization of the whole rather than optimization of the component
parts, and maximization of shared data and services across the
component parts rather than duplication. Moreover, these approaches
are not mutually exclusive and, in fact, can all be applied to some
degree for a given enterprise, depending on the characteristics and
circumstances of that enterprise. The approaches, which are briefly
described here, are federated, segmented, and service-oriented.
Federated:
Under a federated approach, the architecture consists of a family of
coherent but distinct member architectures that conform to an
overarching corporate or parent architecture. This approach recognizes
that each federation member has unique goals and needs as well as
common roles and responsibilities with the members above and below it.
As such, member architectures (e.g., component, subordinate, or
subsidiary architectures) are substantially autonomous, but they also
inherit certain rules, policies, procedures, and services from the
parent architectures. A federated architecture enables component
organization autonomy while ensuring corporate or enterprisewide
linkages and alignment where appropriate.
Segmented:
A segmented approach to enterprise architecture development and use,
like a federated approach, employs a "divide and conquer" methodology
in which architecture segments are identified, prioritized, developed,
and implemented. In general, segments can be viewed as logical
aspects, or "slivers," of the enterprise that can be architected and
pursued as separate initiatives under the overall corporate
architecture. As such, the segments serve as a bridge between the
corporate frame of reference captured in the enterprise architecture
and individual programs within portfolios of system investments. OMB
has issued guidance related to segment architectures.[Footnote 28] As
part of its guidance, agencies are to group segments into three
categories: core mission areas (e.g., air transportation), business
services (e.g., financial management), and enterprise services (e.g.,
records management).
Service-Oriented:
A service-oriented approach to enterprise architecture is intended to
identify and promote the shared use of common business capabilities
across the enterprise. Under this approach, functions and applications
are defined and designed as discrete and reusable capabilities or
services that may be under the control of different organizational
entities. As such, the capabilities or services need to be, among
other things, (1) self-contained, meaning that they do not depend on
any other functions or applications to execute a discrete unit of
work; (2) published and exposed as self-describing business
capabilities that can be accessed and used; and (3) subscribed to via
well-defined and standardized interfaces. This approach is intended to
reduce redundancy and increase integration, as well as provide the
flexibility needed to support a quicker response to changing and
evolving business requirements and emerging conditions.
IT Investment Management: A Brief Description:
IT investment management is a process for linking investment decisions
to an organization's strategic objectives and business plans that
focuses on selecting, controlling, and evaluating investments in a
manner that minimizes risks while maximizing the return of investment.
[Footnote 29]
* During the selection phase, the organization (1) identifies and
analyzes each project's risks and returns before committing
significant funds to any project and (2) selects those IT projects
that will best support its mission needs.
* During the control phase, the organization ensures that, as projects
develop and investment expenditures continue, the projects meet
mission needs at the expected levels of cost and risk. If the project
is not meeting expectations, or if problems arise, steps are quickly
taken to address the deficiencies.
* During the evaluation phase, actual versus expected results are
compared once a project has been fully implemented. This is done to
(1) assess the project's impact on mission performance, (2) identify
any changes or modifications to the project that may be needed, and
(3) revise the investment management process based on lessons learned.
Consistent with this guidance, our IT Investment Management (ITIM)
framework consists of five progressive stages of maturity for any
given agency relative to selecting, controlling, and evaluating its
investment management capabilities.[Footnote 30] (See figure 2 for the
five ITIM stages of maturity.) The overriding purpose of the framework
is to encourage investment selection and control and to evaluate
processes that promote business value and mission performance, reduce
risk, and increase accountability and transparency. We have used the
framework in many of our evaluations, and a number of agencies have
adopted it.[Footnote 31]
In our ITIM framework, with the exception of the first stage, each
maturity stage is composed of "critical processes" that must be
implemented and institutionalized in order for the organization to
achieve that stage. Each ITIM critical process consists of "key
practices" (organizational structures, policies, and procedures) that
must be executed to implement the critical process. Our research shows
that agency efforts to improve investment management capabilities
should focus on implementing all lower-stage practices before
addressing the higher-stage practices.
Figure 2: The Five ITIM Stages of Maturity with Critical Processes:
[Refer to PDF for image: illustration]
Maturity stage: Stage 1: Creating investment awareness;
Critical processes:
* IT spending without disciplined investment processes.
Maturity stage: Stage 2: Building the investment foundation;
Critical processes:
* Instituting the investment board;
* Meeting business needs;
* Selecting an investment;
* Providing investment oversight;
* Capturing investment information.
Maturity stage: Stage 3: Developing a complete investment portfolio;
Critical processes:
* Defining the portfolio criteria;
* Creating the portfolio;
* Evaluating the portfolio;
* Conducting postimplementation reviews.
Maturity stage: Stage 4: Improving the investment process;
Critical processes:
* Improving the portfolio's performance;
* Managing the succession of information systems.
Maturity stage: Stage 5: Leveraging IT for strategic outcomes;
Critical processes:
* Optimizing the investment process;
* Using IT to drive strategic business change.
Source: GAO.
[End of figure]
Stage 2 critical processes lay the foundation by establishing
successful, predictable, and repeatable investment control processes
at the project level. Stage 3 is where the agency moves from project-
centric processes to portfolio-based processes and evaluates potential
investments according to how well they support the agency's missions,
strategies, and goals. Organizations implementing these Stage 2 and 3
practices have in place selection, control, and evaluation processes
that are consistent with the Clinger-Cohen Act.[Footnote 32] Stages 4
and 5 require the use of evaluation techniques to continuously improve
both investment processes and portfolios in order to better achieve
strategic outcomes.
Our research shows that agency efforts to improve investment
management capabilities should focus on implementing all lower-stage
practices before addressing the higher-stage practices and therefore
our reviews tend to focus on Stage 2 and Stage 3 critical processes.
Specifically, within Stage 2, there are five critical processes and
nine associated key practices (known as organizational commitments)
that call for policies and procedures associated with effective
project-level management. These are shown in table 1.
Table 1: Stage 2 Critical Processes and Associated Key Practices:
Critical process: Instituting the investment board;
Purpose: To define and establish an appropriate IT investment
management structure and the processes for selecting, controlling, and
evaluating IT investments;
Associated key practices:
1. An enterprisewide IT investment board composed of senior executives
from IT and business units is responsible for defining and
implementing the organization's IT investment governance process;
2. The organization has a documented IT investment process directing
each investment board's operations.
Critical process: Meeting business needs;
Purpose: To ensure that IT projects and systems support the
organization's business needs and meet users' needs;
Associated key practices:
3. The organization has documented policies and procedures for
identifying IT projects or systems that support the organization's
ongoing and future business needs.
Critical process: Selecting an investment;
Purpose: To ensure that a well-defined and disciplined process is used
to select new IT proposals and reselect ongoing investments;
Associated key practices:
4. The organization has documented policies and procedures for
selecting new IT proposals;
5. The organization has documented policies and procedures for
reselecting ongoing IT investments;
6. The organization has policies and procedures for integrating
funding with the process of selecting an investment.
Critical process: Providing investment oversight;
Purpose: To review the progress of IT projects and systems, using
predefined criteria and checkpoints in meeting cost, schedule, risk,
and benefit expectations and to take corrective action when these
expectations are not being met;
Associated key practices:
7. The organization has documented policies and procedures for
management oversight of IT projects and systems.
Critical process: Capturing investment information;
Purpose: To make available to decision makers information to evaluate
the impacts and opportunities created by proposed (or continuing) IT
investments;
Associated key practices:
8. The organization has documented policies and procedures for
identifying and collecting information about IT projects and systems
to support the investment management process;
9. An official is assigned responsibility for ensuring that the
information collected during project and systems identification meets
the needs of the investment management process.
Source: GAO.
[End of table]
Within Stage 3, there are four critical processes and five associated
key practices (known as organizational commitments) that call for
policies and procedures associated with effective portfolio-based
investment management. These are shown in table 2.
Table 2: Stage 3 Critical Processes and Associated Key Practices:
Critical process: Defining the portfolio criteria;
Purpose: To ensure that the organization develops and maintains IT
portfolio selection criteria that support its mission, organizational
strategies, and business priorities;
Associated key practices:
1. The organization has documented policies and procedures for
creating and modifying IT portfolio selection criteria;
2. Responsibility is assigned to an individual or group for managing
the development and modification of the IT portfolio selection
criteria.
Critical process: Creating the portfolio;
Purpose: To ensure that IT investments are analyzed according to the
organization's portfolio selection criteria and to ensure that an
optimal IT investment portfolio with manageable risks and returns is
selected and funded;
Associated key practices:
3. The organization has documented policies and procedures for
analyzing, selecting, and maintaining the investment portfolio.
Critical process: Evaluating the portfolio;
Purpose: To review the performance of the organization's investment
portfolios at agreed-upon intervals and to adjust the allocation of
resources among investments as necessary;
Associated key practices:
4. The organization has documented policies and procedures for
reviewing, evaluating, and improving the performance of its portfolios.
Critical process: Conducting post-implementation reviews;
Purpose: To compare the results of recently-implemented investments
with the expectations that were set for them and to develop a set of
lessons learned from these reviews;
Associated key practices:
5. The organization has documented policies and procedures for
conducting post-implementation reviews.
Source: GAO.
[End of table]
DOD's Institutional Approach to Business Systems Modernization:
The NDAA for Fiscal Year 2008 designated the Deputy Secretary of
Defense position as the Chief Management Officer for DOD and created a
deputy position to assist the Chief Management Officer.[Footnote 33]
The Chief Management Officer's responsibilities include developing and
maintaining a departmentwide strategic plan for business reform and
establishing performance goals and measures for improving and
evaluating overall economy, efficiency, and effectiveness, and
monitoring and measuring the progress of the department. The Deputy
Chief Management Officer's responsibilities include recommending to
the Chief Management Officer methodologies and measurement criteria to
better synchronize, integrate, and coordinate the business operations
to ensure alignment in support of the warfighting mission. The
Business Transformation Agency supports the Deputy Chief Management
Officer in leading and coordinating business transformation efforts
across the department. This includes maintaining and updating the
department's enterprise architecture for its business mission area.
[Footnote 34]
The Chief Management Officer and Deputy Chief Management Officer are
to interact with several entities to guide the direction, oversight,
and execution of DOD's business transformation efforts, which include
business systems modernization. These entities include the Defense
Business Systems Management Committee, which serves as the highest-
ranking investment review and decisionmaking body for business systems
modernization activities and is chaired by the Deputy Secretary of
Defense; the principal staff assistants, who serve as the
certification[Footnote 35] authorities for business system
modernizations in their respective core business missions; the
investment review boards (IRB),[Footnote 36] which are chaired by the
certifying authorities and form the review and decision-making bodies
for business system investments in their respective areas of
responsibility; and the Business Transformation Agency, which is
responsible for supporting the IRBs and for leading and coordinating
business transformation efforts across the department. In August 2010
and in January 2011, the Secretary of Defense announced the plans to
disestablish the Business Transformation Agency and the Office of the
Assistant Secretary of Defense for Networks and Information
Integration/Department of Defense CIO (ASD(NII)/DOD CIO) (who is a
member of the Defense Business Systems Management Committee),
respectively. According to DOD officials, the mission of the Office of
the Deputy Chief Management Officer duplicates many of the Business
Transformation Agency functions. They added that rather than lead in
the development of better business practices, the agency's prime focus
has changed to the management of several relatively small business
systems and providing direct support to the Deputy Chief Management
Officer on various policy issues. As a result, according to these
officials, the narrower function does not justify continuing the
Business Transformation Agency as a stand-alone defense agency.
The Secretary of Defense also directed that the remaining functions of
the Business Transformation Agency be reviewed and transferred to
other organizations in DOD, as appropriate and that the
disestablishment of the agency should be no later than June 30, 2011.
However, as of June 2011, these implementation decisions had yet to be
made and both the Business Transformation Agency and the Office of the
ASD (NII)/DOD CIO organizations were still in operation.
Table 3 lists governance entities and provides greater detail on their
roles, responsibilities, and composition.
Table 3: DOD Business Systems Modernization Governance Entities'
Roles, Responsibilities, and Composition:
Entity: Defense Business Systems Management Committee;
Roles and responsibilities:
* Provides strategic direction and plans for the business mission area
in coordination with the warfighting and enterprise information
environment mission areas[A];
* Recommends policies and procedures required to integrate DOD
business transformation and attain cross-department, end-to-end
interoperability of business systems and processes;
* Serves as approving authority for business system modernizations
greater than $1 million;
* Establishes policies and approves the business mission area
strategic plan, the enterprise transition plan for implementation of
business systems modernization, the transformation program baseline,
and the business enterprise architecture;
Composition: Chaired by the Deputy Secretary of Defense/Chief
Management Officer; the Vice Chair is the Deputy Chief Management
Officer. Includes senior leadership in the Office of the Secretary of
Defense, such as the Assistant Secretary of Defense for Network and
Information Integration/Department of Defense Chief Information
Officer (ASD(NII)/DOD CIO). Also includes the Military Department
Chief Management Officers, the heads of select defense agencies, and
other senior participation by the Joint Chiefs of Staff and the U.S.
Transportation Command.
Entity: Principal Staff Assistants/Certification Authorities;
Roles and responsibilities:
* Support the Defense Business Systems Management Committee's
management of enterprise business IT investments;
* Serve as the certification authorities accountable for the
obligation of funds for respective business system modernizations
within designated core business missions.[B];
* Review, approve, and oversee the planning, design, acquisition,
deployment, operation, maintenance, and modernization of the defense
business systems assigned;
* Provide the Defense Business Systems Management Committee with
recommendations for system investment approval;
* Provide input into enterprise-level architecture products and
transition plans that support their core business mission;
Composition: Under Secretaries of Defense for Acquisition, Technology,
and Logistics; Comptroller; and Personnel and Readiness; ASD(NII)/DOD
CIO; and the Deputy Secretary of Defense.
Entity: Investment Review Boards;
Roles and responsibilities:
* Serve as the oversight and investment decision making bodies for
those business capabilities that support activities under their
designated areas of responsibility;
* Review and recommend certification for all business systems
modernization investments costing more than $1 million that are
integrated and compliant with the business enterprise architecture;
Composition: Includes the principal staff assistants, Joint Staff,
ASD(NII)/DOD CIO, core business mission area representatives, military
departments, defense agencies, and combatant commands.
Entity: Component Precertification Authority[C];
Roles and responsibilities:
* Ensures component-level investment review processes integrate with
the investment management system;
* Identifies those component systems that require investment review
board certification and prepare, review, approve, validate, and
transfer investment documentation as required;
* Assesses and pre-certifies business process re-engineering efforts
and architecture compliance of component systems submitted for
certification and annual review;
Composition: Includes the Chief Management Officer from Air Force, the
Army, the Navy, and the DOD Deputy Chief Management Officer
representing the defense agencies or a business system supported by
more than one military department or defense agency.
Entity: Business Transformation Agency;
Roles and responsibilities:
* Operates under the authority of the Deputy Chief Management Officer;
* Maintains and updates the department's business enterprise
architecture and enterprise transition plan;
* Ensures that functional priorities and requirements of various
defense components, such as the Army and the Defense Logistics Agency,
are reflected in the architecture;
* Ensures adoption of DOD-wide information and process standards as
defined in the architecture;
* Serves as the day-to-day management entity of the business
transformation effort at the DOD enterprise level;
* Provides support to the investment review boards;
Composition: Composed of eight directorates (Chief of Staff, Defense
Business Systems Acquisition Executive, Enterprise Integration,
Enterprise Planning and Investment, Transformation Priorities and
Requirements Financial Management, Transformation Priorities and
Requirements Human Resource Management, Transformation Priorities and
Requirements Supply Chain Management, and Warfighter Requirements).
Source: GAO based on DOD documentation.
[A] According to DOD, the business mission area is responsible for
ensuring that capabilities, resources, and materiel are reliably
delivered to the warfighter. Specifically, the business mission area
addresses areas such as real property and human resources management.
[B] DOD has five core business missions: Human Resources Management,
Weapon Systems Lifecycle Management, Materiel Supply and Service
Management, Real Property and Installations Lifecycle Management, and
Financial Management.
[C] In the military departments, the Chief Management Officer is the
precertification authority. For the defense agencies, precertification
activities are performed by the component, and the DOD Deputy Chief
Management Officer is the precertification authority. These
precertification activities result in a Chief Management Officer
Determination Memorandum.
[End of table]
Overview of DOD's Tiered Accountability for Business Systems
Modernization:
Since 2005, DOD has employed a "tiered accountability" approach to
business systems modernization. Under this approach, responsibility
and accountability for business architectures and systems investment
management are assigned to different levels in the organization. For
example, the Business Transformation Agency is responsible for
developing the corporate business enterprise architecture (i.e., the
thin layer of DOD-wide policies, capabilities, standards, and rules)
and the associated enterprise transition plan. Each component is
responsible for defining a component-level architecture and transition
plan associated with its own tiers of responsibility and for doing so
in a manner that is aligned with (i.e., does not violate) the
corporate business enterprise architecture. Similarly, program
managers are responsible for developing program-level architectures
and plans and for ensuring alignment with the architectures and
transition plans above them. This concept is to allow for autonomy
while also ensuring linkages and alignment from the program level
through the component level to the corporate level. Table 4 describes
the four investment tiers and identifies the associated reviewing and
approving entities.
Table 4: DOD Investment Tiers:
Tier 1;
Tier description: Major automated information system[A] or major
defense acquisition program[B];
Reviewing/approving entities: Investment Review Board and Defense
Business Systems Management Committee.
Tier 2;
Tier description: Exceeding $10 million in total development/
modernization costs, but not designated as a major automated
information system or major defense acquisition program;
Reviewing/approving entities: Investment Review Board and Defense
Business Systems Management Committee.
Tier 3;
Tier description: Exceeding $1 million and up to $10 million in total
development/modernization costs;
Reviewing/approving entities: Investment Review Board and Defense
Business Systems Management Committee.
Tier 4;
Tier description: Investment funding required up to $1 million;
Reviewing/approving entities: Component-level review only (unless the
system or line of business it supports is designated as an interest
program by the Investment Review Board chair).
Source: GAO based on DOD documentation.
[A] A major automated information system is a program or initiative
that is so designated by the ASD(NII)/DOD CIO or that is estimated to
require program costs in any single year in excess of $32 million,
total program costs in excess of $126 million, or total life cycle
costs in excess of $378 million in fiscal year 2000 constant dollars.
[B] A major defense acquisition program is an acquisition program that
is so designated or estimated by the Under Secretary of Defense for
Acquisition, Technology, and Logistics to require an eventual total
expenditure for research, development, and test and evaluation of more
than $365 million or, for procurement, of more than $2.190 billion in
fiscal year 2000 constant dollars.
[End of table]
Consistent with the tiered accountability approach, the Fiscal Year
2008 NDAA required the secretaries of the military departments to
designate the department under secretaries as chief management
officers with primary responsibility for business operations.[Footnote
37] Moreover, the Fiscal Year 2009 Duncan Hunter NDAA required the
military departments to establish business transformation offices to
assist their chief management officers in the development of
comprehensive business transformation plans.[Footnote 38] In response,
all of the military departments have designated their respective Under
Secretaries as the chief management officers.
We reported in January 2011 that DOD and the military departments had
made limited progress in developing business transformation plans that
are supported by a strategic planning process and would enable them to
align goals and planning efforts and to measure progress.[Footnote 39]
Specifically, we reported that DOD had not set up internal mechanisms,
such as procedures and milestones, by which it can reach consensus
with the military departments and others on priorities, synchronize
the development of plans with each other and the budget process, and
guide efforts to monitor progress and take corrective action.
Therefore, while the military departments were in varying stages of
developing business transformation plans, it was unclear to what
extent the business transformation priorities for the military
departments will be aligned with the priorities identified in DOD's
Strategic Management Plan or how these business transformation
priorities will influence the department's budget requests.
Accordingly, we made recommendations to enhance DOD's ability to set
strategic direction for its business transformation efforts, and
better align and institutionalize its efforts to develop and implement
plans and measure progress against established goals. DOD partially
agreed with the recommendations.
DOD's Approach to Certifying Business System Investments:
DOD has a two-stage process to select investments, which it refers to
as certification and is a key step in its IT investment process that
DOD has aimed to model after GAO's ITIM framework. The first stage
involves selection of systems using the Joint Capabilities Integration
and Development System, the Defense Acquisition System, and the
Planning, Programming, Budgeting, and Execution management systems. At
this level, proposals and alternatives are viewed and prioritized for
system selection. The second stage of selection involves (1)
certifying and approving Tiers 1 through 3 investments and (2)
elevating certain component investments to an enterprisewide status.
More recently, DOD developed Business Capability Lifecycle guidance,
dated November 2010, intended to streamline business system
acquisition, and investment management processes to better guide and
constrain departmentwide systems modernizations.
For certification, the IRBs rely on documentation submitted by
appropriate chief management officers for the military departments and
precertification authorities for the defense agencies. This
documentation includes but is not limited to, a memorandum asserting
that an investment is compliant with the business enterprise
architecture; an economic viability analysis, which addresses the
investment's cost and benefits or cost effectiveness; a business
process reengineering determination, which identifies the investment's
business process weaknesses, gaps, and opportunities for process
improvement; and a certification dashboard, which includes cost and
schedule status information. DOD guidance also gives the boards broad
authority in their certification reviews and actions, thus allowing
each board to review and consider whatever investment-related
information that it deems appropriate. Moreover, Business
Transformation Agency and IRB officials told us that a board is not
limited in the conditions it can place on a program. After an IRB
review, the Defense Business Systems Management Committee will be
notified of all certification decisions and may elect to approve or
disapprove a certification. If a certification is approved, the
committee will sign an approval memo that warrants the ability to
obligate the related funding for the investment.
Under DOD's approach, there are four types of certification actions:
* Certify: An IRB certifies the modernization as fully meeting
criteria defined in the act and IRB investment review guidance, such
as compliance with the business enterprise architecture and the extent
to which the investment is consistent with component and department IT
investment portfolios, which are asserted by the component Chief
Management Officer.
Certify with conditions: An IRB certifies the modernization with the
understanding that it will address specific investment review board-
imposed conditions. For example, the Army's General Fund Enterprise
Business System was certified with a condition to develop a plan for
complying with the data standards of DOD's Item Unique Identifier
Registry.[Footnote 40]
* Recertify: An IRB certifies the obligation of additional
modernization funds for a previously-certified modernization
investment. For example, the Air Force's Defense Enterprise Accounting
and Management System was recertified in September 2010 for $26.7
million to be spent across fiscal years 2010 and 2011. This
recertification was in addition to the approximately $22.7 million
previously certified in December 2009. In addition, a program must
request IRB recertification if the program plans to redistribute
previously approved modernization funds among multiple fiscal years
and this redistribution will result in the funding for any given
fiscal year exceeding the previously approved amount by 10 percent or
more.
* Decertify: An IRB may decertify or reduce the amount of
modernization funds available to an investment when (1) a component
reduces funding for a modernization by more than 10 percent of the
originally certified amount, (2) the period of certification for a
modernization is shortened, or (3) the entire amount of funding is not
to be obligated as previously certified. For example, the Special
Operations Command's Special Operations Resource Business Information
System had about $4.59 million decertified because funding was reduced
by more than 10 percent of the originally certified amount. An
investment review board may also decertify a modernization after
development has been terminated. For example, DOD reported that
approximately $2.77 million in research, development, test and
evaluation funding for the Army's Wounded Warrior Accountability
System was decertified because it was determined that the system
provided a capability that could be better utilized under another
system.
Summary of Fiscal Year 2005 NDAA Requirements:
Congress included provisions in the Fiscal Year 2005 NDAA that are
aimed at ensuring DOD's development of a well-defined business
enterprise architecture and associated enterprise transition plan, as
well as the establishment and implementation of effective investment
management structures and processes.[Footnote 41] According to the
act, DOD is required to:
* develop a business enterprise architecture and an enterprise
transition plan for implementing the architecture,
* identify each business system proposed for funding in DOD's fiscal
year budget submissions,
* delegate the responsibility for business systems to designated
approval authorities within the Office of the Secretary of Defense,
* require each approval authority to establish investment review
structures and processes, and:
* effective October 1, 2005, not obligate appropriated funds for a
defense business system modernization with a total cost of more than
$1 million unless the approval authority certifies that the business
system modernization meets several conditions.[Footnote 42]
The Fiscal Year 2005 NDAA also requires that the Secretary of Defense
annually submit to the congressional defense committees a report on
the department's compliance with the above provisions.
Recent GAO Reviews of DOD's Business Systems Modernization:
Between 2005 and 2008, we reported that DOD had taken steps to comply
with key requirements of the Fiscal Year 2005 NDAA relative to
architecture development, transition plan development, budgetary
disclosure, and investment review, and to satisfy relevant systems
modernization management guidance; however, each report also concluded
that much remained to be accomplished relative to the act's
requirements and relevant guidance.[Footnote 43] We also reported that
DOD had fully satisfied the requirement concerning designated approval
authorities and continued to certify and approve modernizations
costing more than $1 million. We concluded that the department had
made progress in defining and beginning to implement institutional
management controls (i.e., processes, structures, and tools). We
reiterated existing recommendations to address each of the areas.
In May 2008, we reported that progress in establishing corporate
management controls needed to be replicated within the military
departments.[Footnote 44] For example, we reported that the military
departments did not yet have mature enterprise architecture programs.
We also reported that they had yet to fully establish key investment
review structures and had yet to define related policies and
procedures for effectively performing both project-level and portfolio-
based investment management. Because we had previously made
recommendations to DOD aimed at putting in place the management
controls needed to fully comply with the act and related federal
guidance, we did not make additional recommendations.
In May 2009, we reported that the pace of DOD's efforts in defining
and implementing key institutional modernization management controls
had slowed compared with progress made in each of the last four years,
leaving much to be accomplished to fully implement the act's
requirements and related guidance.[Footnote 45] For example:
* The corporate business enterprise architecture had yet to be
extended (i.e., federated) to the entire family of business mission
area architectures, including using an independent verification and
validation (IV&V) agent to assess the components' subsidiary
architectures and federation efforts.
* The fiscal year 2009 budget submission included some, but omitted
other key information about business system investments, in part
because of the lack of a reliable, comprehensive inventory of all
defense business systems.
* The business system information used to support the development of
the transition plan and DOD's budget requests, as well as
certification and annual reviews, was of questionable reliability.
* DOD and the military departments had yet to fully define key
practices (i.e., policies and procedures) related to effectively
performing both project-level (Stage 2) and portfolio-based (Stage 3)
investment management as called for in the ITIM. For example, of the
nine Stage 2 key practices and five Stage 3 key practices, DOD had
defined four and one, respectively, while Air Force had defined three
and one, respectively. Subsequent to our reports, Army reported that
it had (as of May 2009) efforts planned and under way to develop
effective investment management processes, but the efforts did not
fully satisfy any key practices at the time.
* Business system investments costing more than $1 million continue to
be certified and approved, but these decisions were not always based
on complete information.
Accordingly, we reiterated existing recommendations to address each of
these areas and further recommended that DOD, among other things,
improve the quality of investment-related information. DOD partially
agreed with our recommendations and described actions being planned or
under way to address them. DOD is currently in the process of
addressing these recommendations.
In May 2010,[Footnote 46] we reported that the scope and completeness
of key information provided in the report were limited. Specifically,
the report omitted information on the number of business system
investment certification actions taken during fiscal year 2009 and did
not include performance measures, such as measures of progress against
program cost, capability, and benefit commitments. Further, we
concluded that certification and approval decisions may not be
sufficiently justified because investments were certified and approved
without conditions even though our prior reports had identified
program weaknesses that were unresolved at the time of certification
and approval. Accordingly, we recommended that DOD ensure that the
scope and content of future annual reports to Congress on compliance
with section 332 of the NDAA for fiscal year 2005 be expanded to
include cost, capability, and benefits performance measures for each
business system modernization investment and actual performance
against these measures as well as all certification actions on its
business system modernization investments that were taken in the
previous year by the department. In addition, we recommended that DOD
guidance be revised to include provisions that require investment
review board certification submissions to disclose program weaknesses
raised by us and the status of actions to address our recommendations
to correct the weaknesses to ensure that investment review board
certification actions are better informed and justified. DOD agreed
with our recommendations.
DOD Continues to Strengthen Management of Its Business Systems
Modernization, but Long-standing Challenges Remain:
DOD continues to take steps to comply with the provisions of the
Fiscal Year 2005 NDAA and to satisfy relevant system modernization
management guidance. In particular, DOD released its fiscal year 2011
enterprise transition plan in December 2010, followed by an update to
its business enterprise architecture (version 8.0) in March 2011, and
its annual report to Congress in May 2011 describing steps that have
been taken and are planned relative to the act's requirements.
Collectively, these steps address several statutory provisions and
best practices concerning the business enterprise architecture,
transition plan, budgetary disclosure, and investment review of
systems costing in excess of $1 million. However, challenges that we
identified in prior years still need to be addressed in order for the
department to be in full compliance with guidance and NDAA
requirements. Most notably, the department has yet to extend and
evolve its business enterprise architecture to the military
departments' architectures and to fully define IT investment
management policies and procedures at the DOD enterprise and component
levels. DOD officials agreed that additional steps are needed. They
said that the lack of progress is due in part to the uncertainty and
pending decisions surrounding the roles and responsibilities of key
organizations and senior leadership positions, such as the Business
Transformation Agency and ASD(NII)/DOD CIO. However, until DOD fully
implements these long-standing institutional modernization management
controls required by the act, addressed in GAO recommendations, and
otherwise embodied in relevant guidance, its business systems
modernization will likely remain a high-risk program. As a result, it
is important that the department act quickly to resolve pending
decisions about roles and responsibilities.
Adopting a Federated Approach to Architecture Continues to Be a
Challenge with Much Remaining to Be Accomplished at the Component
Level:
Among other things, the act requires DOD to develop a business
enterprise architecture to cover all defense business systems and
their related functions and activities. According to the act, the
architecture should extend to all defense organizational components.
In 2006, the department adopted an incremental and federated approach
to developing such an architecture. Under this approach, the
department releases new architecture versions every year that include
a corporate business enterprise architecture that is to be augmented
by a coherent family of component architectures. As we have previously
reported, such an approach is consistent with best practices and
appropriate given DOD's scope and size.[Footnote 47]
On March 18, 2011, DOD released its business enterprise architecture
version 8.0, which focuses on improving the department's ability to
manage business operations from an end-to-end perspective. This
version continues to represent the thin layer of corporate
architectural policies, capabilities, rules, and standards that apply
DOD-wide (i.e., to all DOD federation members). This means that
version 8.0 appropriately focuses on addressing a limited set of
enterprise-level (DOD-wide) priorities and provides the overarching
and common architectural context that the distinct and autonomous
member (i.e., component) architectures inherit. Nevertheless, this
also means that version 8.0 does not provide the total federated
family of DOD parent and subsidiary architectures for the business
mission area. Having such an architecture is dependent on each
military department (Air Force, Army, and Navy) having the capability
to manage its enterprise architecture and develop the necessary
content.
To assist DOD in its architecture federation efforts, we have
previously made a number of recommendations. Specifically, in May
2007, we recommended that the department include in its annual report,
required under the act, the results of its IV&V contractor's
assessment of the completeness, consistency, understandability, and
usability of the federated family of architectures.[Footnote 48] While
DOD agreed with the recommendation, none of its annual reports since
2007, including its latest annual report (issued in May 2011), have
included this information. According to Business Transformation Agency
officials, IV&V activities have focused on the corporate business
enterprise architecture and not on the entire federated family of
architectures. Business Transformation Agency officials provided us
with a report dated March 25, 2011, which summarizes selected IV&V
contractor observations and recommendations relative to version 8.0's
ability to provide a foundation for business enterprise architecture
federation. Overall, the summary confirms our findings by stating
that, while there has been previous work to develop a business
enterprise architecture federation plan, the execution of the
federation has yet to be completed. According to Business
Transformation Agency officials, the current requirement under the
IV&V contract is to provide analysis of the business enterprise
architecture. There are no plans for reviewing the military
departments' enterprise architectures.
The challenges that the department faces in federating its
architecture and the importance of disclosing to congressional defense
committees the state of its federation efforts are amplified by our
prior report on the state of the military departments' enterprise
architecture programs. Specifically, we reported in May 2008 that none
of the three military departments could demonstrate through verifiable
documentation that it had established all of the core foundational
commitments and capabilities needed to effectively manage the
development, maintenance, and implementation of an architecture,
[Footnote 49] as outlined in our Enterprise Architecture Management
Maturity Framework version 1.1.[Footnote 50]
Since then, each of the military departments has adopted a federated
approach to developing its respective architecture program. However,
the extent to which each of their architecture programs has improved
since 2008 varies. For example, while all three have or are in the
process of establishing an executive committee with responsibility and
accountability for the department's enterprise architecture, none have
fully developed an enterprise architecture methodology or have a well-
defined business enterprise architecture and transition plan to guide
and constrain business transformation initiatives. Table 5 provides a
description of the military departments' progress relative to those
elements that we previously reported as not satisfied by one or more
of the military departments. (These elements were common to both
versions 1.1 and 2.0 of our Enterprise Architecture Maturity
Management Framework.)[Footnote 51]
Table 5: Description of Progress for Enterprise Architecture Core
Elements Previously Reported as Not Fully Satisfied by One or More
Military Departments:
Adequate resources exist: [Empty].
Core element[A]: Adequate resources exist;
2011 evaluation:
Air Force: We previously reported this element as being fully
satisfied[B];
Army: According to the Army, not all of their enterprise architecture
budgetary needs are met. Officials stated that the majority of Army
enterprise architecture efforts are funded in an ad hoc or
decentralized process;
Navy: We previously reported this element as being fully satisfied[B].
Core element[A]: Committee or group representing the enterprise is
responsible for directing, overseeing, and approving the enterprise
architecture;
2011 evaluation:
Air Force: According to Air Force officials, three committees with
representation across the enterprise have been established to direct,
oversee, and approve the architecture. The roles and responsibilities
for approving the architecture have been documented in a charter.
However the Air Force has yet to document the roles and
responsibilities for directing and overseeing the architecture in a
charter;
Army: Various enterprisewide executive committees exist that address
architecture-related issues. For example, officials stated that the
LandWarNet/Battle Command group approves individual architectures and
will continue to do so until the Army Architecture Governance
Committee is established. Officials also stated that a reason for this
delay is that the regulation has not yet been approved. They further
reported that the regulation is expected to be published in August
2011 and calls for the Army Architecture Governance Committee to be
responsible and accountable for directing, overseeing, and approving
the enterprise architecture;
Navy: An executive-level committee with representation from across the
enterprise to direct and oversee the architecture has been
established. The roles and responsibilities for directing and
overseeing the architecture have been documented in an approved
charter. As of May 2011, this committee became the approval authority
for the Navy enterprise architecture and plans to approve version 3.0,
scheduled for release by the end of July 2011. However, the charter
has yet to be updated to reflect the committee's approval
responsibility.
Core element[A]: Program office responsible for enterprise
architecture development and maintenance exists;
2011 evaluation:
Air Force: We previously reported this element as being fully
satisfied[B];
Army: Army has yet to establish a program office with responsibility
for the department's enterprise architecture development and
maintenance. According to Army officials, the Office of Business
Transformation oversees the development of the business systems
architecture. However, they stated that decisions are yet to be made
about how to best approach the establishment of an Army-wide program
office. According to officials, a regulation, expected to be published
in August 2011, will assign the Office of the Chief Architect with the
responsibilities of managing enterprise architecture functions;
Navy: Navy has yet to explicitly assign a program office with
responsibility for the department's enterprise architecture
development and maintenance; instead, the department has identified a
small team of people who conduct activities that are typically
associated with such an office. The department has no plans for
establishing an enterprise architecture program office. According to
Navy officials, it is difficult to justify the creation of a large
enterprise architecture program office in a fiscally constrained
environment.
Core element[A]: Enterprise architecture being developed using a
framework, methodology, and automated tool;
2011 evaluation:
Air Force: We previously reported that the Air Force enterprise
architecture was being developed using a framework and automated tool.
However, a documented methodology that includes defined steps, tasks,
standards, tools, techniques, and measures that govern how the
architecture is to be developed, maintained, and validated has yet to
be developed. Officials stated that the development of a methodology
was pushed back due to budget constraints. No time frames have been
established for developing such a methodology;
Army: We previously reported that the enterprise architecture was
being developed using a framework and automated tool. However, a
documented methodology that includes defined steps, tasks, standards,
tools, techniques, and measures that govern how the architecture is to
be developed, maintained, and validated has yet to be developed;
Navy: We previously reported that the enterprise architecture was
being developed using a framework and automated tool. However, a
documented methodology that includes defined steps, tasks, standards,
tools, techniques, and measures that govern how the architecture is to
be developed, maintained, and validated has yet to be developed.
Officials indicated that there are no specific time frames for when a
Navy enterprise architecture methodology will be developed. According
to Navy officials, this is due to the department's current focus on
other resource-intensive commitments, such as applying the current
enterprise architecture content.
Core element[A]: Written and approved policy exists for enterprise
architecture development;
2011 evaluation:
Air Force: We previously reported this element as being fully
satisfied[B];
Army: Army has developed a draft policy that according to officials
calls for enterprise architecture development, maintenance, and use;
identifies key players, such as the Deputy Under Secretary of the
Army, the Army CIO, and Deputy Chief of Staff;
and describes the roles, responsibilities, and relationships for these
key players. However, the policy has yet to be approved. According to
Army officials, the draft policy is expected to be approved in August
2011;
Navy: Navy has developed and approved a policy for enterprise
architecture development.
Core element[A]: Progress against enterprise architecture plans is
measured and reported;
2011 evaluation:
Air Force: Enterprise architecture status is reported to the Chief
Architect on a quarterly basis. However, this progress is not measured
and reported relative to an enterprise architecture plan;
Army: Army-wide architecture development progress is not measured and
reported. Specifically, the execution and completion of corporate
architecture tasks are not fully defined in an enterprise architecture
program plan, work breakdown structure, and schedule;
Navy: We previously reported this element as being fully satisfied.[B].
Core element[A]: Written and approved policy exists for enterprise
architecture maintenance;
2011 evaluation:
Air Force: We previously reported this element as being fully
satisfied[B];
Army: Army has developed a draft policy that according to officials
calls for enterprise architecture development, maintenance, and use;
identifies key players, such as the Deputy Under Secretary of the
Army, the Army CIO, and Deputy Chief of Staff;
and describes the roles, responsibilities, and relationships for these
key players. However, the policy has yet to be approved;
Navy: Navy has developed a policy that calls for the department's CIO
to oversee the maintenance of the department's enterprise architecture.
Core element[A]: Enterprise architecture products and management
processes undergo IV&V;
2011 evaluation:
Air Force: Enterprise architecture products and management processes
have not undergone IV&V;
Army: Enterprise architecture products and management processes have
not undergone IV&V;
Navy: While enterprise architecture products have undergone
verification and validation assessments, they were not conducted by an
independent body. Moreover, enterprise architecture management
processes have not been subject to IV&V.
Core element[A]: Enterprise architecture products describe the current
and target environments and a transition plan;
2011 evaluation:
Air Force: The department continues to develop its corporate and
subordinate enterprise architectures that describe aspects of its
business environment. However, it has not developed a business
enterprise architecture that provides a clear and comprehensive
picture of its current and target business environments. In addition,
while the department has identified and described business
transformation initiatives, it has not developed a business transition
plan;
Army: The department has made progress in developing the first version
of its business systems architecture and transition plan. However,
important enterprise architecture products describing its current and
target business environments have not been developed. In addition, its
transition plan does not include important elements, such as gap
analyses at the enterprise level and for each business function;
Navy: The department continues to develop its corporate enterprise
architecture that describes aspects of its business environment. It
also has developed a business transition plan. However, it has not
developed a business enterprise architecture that provides a clear and
comprehensive picture of its current and target business environments.
In addition, the business transition plan is missing key elements,
such as gap analyses.
Core element[A]: Committee or group representing the enterprise or the
investment review board has approved current version of enterprise
architecture;
2011 evaluation:
Air Force: The June 2011 charter for the executive committee includes
a requirement for approval of the enterprise architecture. The current
version of the enterprise architecture was approved by the CIO, as it
was completed prior to the ratification of this charter. According to
Air Force officials, executive committee approval will begin with the
next version of the architecture;
Army: The executive committee has yet to approve a version of an
enterprisewide architecture;
Navy: In May 2011, the executive committee became the approval
authority for the Navy enterprise architecture and plans to approve
version 3.0, scheduled for release by the end of July 2011. However,
the charter has yet to be updated to reflect the committee's approval
responsibility. Instead, all enterprise architectures to date were
approved by another board. However, this board did not include
representatives from across the entire organization.
Core element[A]: Quality of enterprise architecture products is
measured and reported;
2011 evaluation:
Air Force: We previously reported this element as being fully
satisfied[B];
Army: Army does not measure or report the quality of enterprisewide
architecture products;
Navy: The Navy enterprise architecture IV&V working group assesses the
quality of enterprise architecture products against a set of criteria.
This assessment is submitted to the Navy enterprise architecture
approval board for approval and the results are posted to Navy's
internal enterprise architecture communication tool.
Core element[A]: Written and approved organization policy exists for
IT investment compliance with enterprise architecture;
2011 evaluation:
Air Force: We previously reported this element as being fully
satisfied[B];
Army: Army has yet to document the requirement for IT investment
compliance with enterprise architecture. Officials stated that, in the
future, investments will be required to demonstrate alignment to the
business systems architecture, but did not indicate future plans for
assessing compliance against the Army's enterprisewide architecture,
which is yet to be developed;
Navy: Navy has developed a policy that requires all investments
registered in its inventory to be assessed annually for compliance
with the Navy enterprise architecture.
Core element[A]: Enterprise architecture is integral component of IT
investment management process;
2011 evaluation:
Air Force: We previously reported this element as being fully
satisfied[B];
Army: The Army did not provide evidence that enterprise architecture
is an integral component of its IT investment management process;
Navy: Enterprise architecture is a component of Navy's IT investment
management process, as investments must be assessed for enterprise
architecture compliance during the IT investment management review
process.
Core element[A]: Enterprise architecture products are periodically
updated;
2011 evaluation:
Air Force: We previously reported this element as being fully
satisfied[B];
Army: The Army has yet to develop an enterprisewide architecture, to
include architecture products representing its current and target
environments. In lieu, the department continues to develop segment
architecture products, such as the business systems architecture;
Navy: We previously reported this element as being fully satisfied.[B].
Core element[A]: IT investments comply with enterprise architecture;
2011 evaluation:
Air Force: While the department has documented compliance criteria, it
has yet to provide evidence demonstrating that IT investments have
undergone enterprise architecture compliance assessments;
Army: As stated, the Army has yet to develop an enterprisewide
architecture against which compliance assessments can be conducted. In
addition, no evidence was provided that investments are assessed for
compliance with any of the segment architectures. According to Army
officials, investments will be required to demonstrate compliance with
the Army's business systems architecture;
Navy: Navy has documented specific enterprise architecture compliance
requirements. Further, Navy provided evidence that compliance
assessments were being conducted.
Core element[A]: Organization head has approved current version of
enterprise architecture;
2011 evaluation:
Air Force: The Air Force CIO, delegated as head of enterprise
architecture by the Secretary of the Air Force, has approved the
latest version of its enterprise architecture;
Army: As stated, the Army has yet to develop an enterprisewide
architecture that would be approved by the Secretary of the Army or a
delegated official;
Navy: The Navy CIO, who was delegated the responsibility for
overseeing the development and maintenance of the Navy enterprise
architecture by the Secretary of the Navy, has approved the current
version of its enterprise architecture.
Core element[A]: Return on enterprise architecture investment is
measured and reported;
2011 evaluation:
Air Force: The Air Force did not provide evidence demonstrating that
return on enterprise architecture investment is measured and reported.
Officials stated that a lack of industry-recognized metrics has
inhibited the department's ability to develop useful metrics for
measuring return on enterprise architecture investment;
Army: As stated, the Army has yet to develop an enterprisewide
architecture. Therefore, enterprise architecture return on investment
is not measured and reported. In addition, there was no evidence
provided that enterprise architecture return on investment for each of
its segments is measured and reported;
Navy: Navy did not provide evidence demonstrating that return on
enterprise architecture investment is measured and reported. Officials
stated that a lack of best practices for measuring the value of
enterprise architecture has inhibited the department's ability to
demonstrate return on investment to corporate-level executives.
Core element[A]: Compliance with enterprise architecture is measured
and reported;
2011 evaluation:
Air Force: The Air Force has documented enterprise architecture
compliance criteria. Officials stated that the Air Force architecting
division reports the results of its compliance assessments to the
investment review board, which is responsible for final approval of
the investment, in the form of a score of either "pass" or "fail"
Officials also stated that without a score of "pass" on the
architecture component, funding for an investment can be withheld.
However, the department has yet to provide evidence demonstrating that
IT investments have undergone enterprise architecture compliance
assessments;
Army: As stated, Army has yet to develop an enterprisewide
architecture against which compliance assessments can be conducted. In
addition, there was no evidence provided that showed compliance with
any of the segment architectures is measured and reported;
Navy: Navy has provided evidence showing that compliance assessments
with the Navy enterprise architecture are measured against criteria
laid out in the IT investment management process. Final approval
decisions are made by the Navy CIO or Deputy CIOs and recorded in the
system inventory.
Source: GAO analysis of military departments' documentation.
[A] These core elements represent those that have remained unchanged
from version 1.1 to version 2.0 in our Enterprise Architecture
Maturity Management Framework or those that have been slightly
modified. They also represent only those elements that we previously
reported as not being satisfied or being only partially satisfied by
one or more military departments.
[B] We did not evaluate the current status of elements that we
previously reported as having been satisfied by a military department.
[End of table]
As described in the table, the military departments have made progress
in managing their respective enterprise architecture programs since we
last reported in 2008. However, each has yet to address key elements,
including developing the architecture content, in order to advance to
a level that can be considered fully mature. According to DOD
officials, the lack of progress in addressing key management elements
has been due to the uncertainty and pending decisions surrounding the
roles and responsibilities of key organizations and senior leadership
positions. Air Force officials attributed the state of its enterprise
architecture program in part to the lack of resources. Army officials
also stated that a major challenge has been the lack of resources
(i.e., people and funding) due to the shift of resources to higher
priority initiatives. Navy officials stated that there is not one
overarching reason for the state of the Navy enterprise architecture
program but rather a number of reasons, such as limited resources.
Navy officials agreed that work remains to be done and stated that
they will continue to address the missing elements as they move
forward. Although all the military departments reported resources to
be a challenge, as noted earlier, DOD requested about $17.3 billion
for its business systems environment. Given the department's
prioritization of about $17 billion, the military department
architecture programs have not received resources that they deemed
sufficient to meet their needs. What this means is that DOD, as a
whole, is not as well positioned as it should be to realize the
significant benefits that a well-managed federation of architectures
could afford its business systems modernization efforts. We have
ongoing work looking at the status of each of the military
departments' enterprise architecture programs relative to all the
applicable elements in our Enterprise Architecture Management Maturity
Framework version 2.0.[Footnote 52]
Military Departments Continue to Develop Architecture Content, but
Have yet to Develop Well-Defined Business Enterprise Architectures and
Transition Plans:
The Fiscal Year 2009 NDAA[Footnote 53] requires that each military
department develop a well-defined enterprisewide business architecture
and transition plan encompassing end-to-end business processes and
that is capable of providing accurate and timely information in
support of business decisions of the military department. However,
while each of the departments has taken steps to develop architecture
content, none has a well-defined business enterprise architecture and
associated transition plan to guide and constrain its business
transformation initiatives. Individual examples of progress made and
challenges still facing each department are discussed next.
Department of the Air Force:
The Department of the Air Force is developing a corporate enterprise
architecture and 12 subordinate "sub-enterprise" architectures, each
of which is to be supported by subordinate domain architectures (i.e.,
acquisition), as appropriate. According to Air Force officials, work
is still under way to identify the respective domains and determining
which domains will support each sub-enterprise. According to these
officials, the Air Force business enterprise architecture was until
now being developed as a separate initiative but work is under way to
integrate it with the Agile Combat Support sub-enterprise
architecture.[Footnote 54]
In 2008, we reported that, as part of its Agile Combat Support sub-
enterprise architecture, the department had developed enterprise
architecture products that described some, but not all, elements of
its current and target environments as well as a transition plan for
its business area. Specifically, we reported that the Air Force had
not defined the architecture products that described its logistics
enterprise architecture for its current environment and its
acquisition enterprise architecture and health services enterprise
architectures for its target environments. We also reported that,
while it had developed a sequencing plan of systems, it had not
defined a gap analysis describing how the department would transition
from the current to the target environment. Since then, the department
has developed architectural artifacts that capture some of the missing
business-related elements. For example, the Air Force has identified
acquisition transformation goals, health services operational
activities, and logistics systems and services, such as the Commodity
Management Service.[Footnote 55] In addition, the department has
identified and described business transformation initiatives (e.g.,
streamline civilian hiring process to meet DOD's goal of 80 days by
2012). However, not all important business enterprise architecture
contents have been described, including current functional
capabilities and data objects necessary for current business
operations.
Furthermore, although the department has identified some business-
related elements, it does not have a well-defined business enterprise
architecture and a transition plan to guide and constrain business
transformation initiatives. In particular, it has yet to develop
architectural products under its Air Force business enterprise
architecture that would describe its current and target business
environments in terms of business, information/data, application/
service, technology, performance, and security. According to the Air
Force, it is focused instead on capturing and using existing
architecture artifacts that describe current and target architectures
associated with priority areas for improvement. In addition, although
the Air Force has outlined the business improvement priorities, it has
yet to develop an enterprise transition plan for the business
environment, including descriptions of gaps in terms of functional
capabilities, performance shortfalls of business processes, and
potential duplications of system functions.
The department also has yet to determine what, if any, of the business
architectural artifacts under the Agile Combat Support sub-enterprise
is to be leveraged for the development of the Air Force business
enterprise architecture. According to Air Force officials, the Air
Force business enterprise architecture and the Agile Combat Support
sub-enterprise will use one common set of artifacts, and the
department has identified the business architectural artifacts from
the Agile Combat Support sub-enterprise to be merged with the Air
Force business enterprise architecture. However, we have yet to be
provided with a listing of these artifacts. In addition, according to
these officials, discussion is still under way on how the department
plans to federate the sub-enterprise architectures, including the
Agile Combat Support, to the corporate enterprise architecture.
According to Air Force officials, in the future, the Air Force
business enterprise architecture will focus on end-to-end processes
with defined outputs and it will include an alignment of business
systems to these end-to-end business processes. Also, according to the
Air Force, it plans to use an architecture-based approach to align
current Air Force capabilities and systems against specific business
processes to identify duplication or gaps in process-based
capabilities. According to the department, an implementation plan will
be developed by the end of fiscal year 2011.
Without architectural descriptions of current and target business
environments and an associated transition plan, as well as a clear
picture of how to leverage prior content in development of the Air
Force business enterprise architecture, the department is not well
positioned to realize the significant benefits that a well-defined
enterprisewide business architecture and transition plan can provide,
including objective decision making regarding capability enhancements
across end-to-end business processes and resources allocation across
business system investments.
Department of the Army:
The Department of the Army has yet to develop or establish plans for
developing a corporate enterprise architecture that identifies and
describes rules, policies, procedures, and services to be federated
across the Army. Instead, the department's approach consists of
developing three separate and distinct segment architectures (i.e.,
battle command, networks, and generating force), each of which is
being developed by separate department organizational units and is
supported by individual segment and solution architectures. According
to department officials, the department's current approach to
developing its business enterprise architecture calls for this content
to be developed as part of the generating force enterprise
architecture effort.
In 2008, we reported that Army had not defined its current and target
business environments and developed a transition plan. To its credit,
the department has since made progress in developing its first version
of the Army's business enterprise architecture. Specifically, it has
defined the scope of this architecture, which comprises traditional
business functions such as acquisition, logistics, financial
management, human capital management, and installation and
environment. The scope of the Army's business enterprise architecture
is also extended to include training and sub-segments of the
LandWarNet/Battle Command[Footnote 56] functions. This provides
logical groupings of the key business activities the department
performs, and can be used to identify subbordinate architectures that
support the department's generating force mission area. In addition,
the architecture includes the 15 end-to-end business processes listed
in the DOD business enterprise architecture and Army plans to use
these processes to guide and constrain business process modeling
efforts and identify business systems that are to be integrated. This
is consistent with DOD's approach of using end-to-end business
processes to optimize business processes and the systems that support
them. Further, the department has outlined its business transformation
initiatives (e.g., civilian hiring reform) to accelerate business
process improvement and cost savings.
However, Army does not have a well-defined business enterprise
architecture and a transition plan to guide and constrain business
transformation initiatives. In particular, although the first version
provides systems mappings to several end-to-end business processes
(e.g., Procure-to-Pay),[Footnote 57] it has yet to provide evidence of
mappings of system functions to all the end-to-end business processes,
such as Service Request-to-Resolution.[Footnote 58] According to
officials, the mapping of system functions to process steps will take
place when the department performs detailed business process mapping.
Further, the first version does not describe how end-to-end business
processes will be implemented, including principles and guidance for
implementing technologies that would support planned business systems
investments. Without this, there is an increased risk of incompatible
implementation and/or not being able to leverage the most benefits
from the technologies.
The Army has also made progress in developing aspects of a business
transition plan. For example, it identifies business transformation
initiatives, such as civilian hiring reform, and describes results or
changes to business operations to be achieved by the business
transformation initiatives. It also includes diagrams that depict the
migration from legacy business systems to commercial off-the-shelf
enterprise resource planning solutions. Such diagrams provide a life
cycle view of enterprise systems resources, including describing how
each system evolves over time. However, the plan still does not
include important content such as gap analyses at the enterprise level
and for each business function (e.g. acquisition, financial
management) and timelines for addressing the gaps. A gap analysis is
an assessment of the differences between the current and target
architectures. For example, a performance gap analysis identifies
performance measures (e.g., effectiveness) of a business process,
highlights which performance measures are not being met in the current
environment, and describes performance expectations for these measures
in the target environment, thereby describing expected performance
improvements of the business process. This performance gap analysis
should also identify the business process activities or steps that
need to be changed to achieve the future performance expectations. As
such, these gap analyses are important to help identify changes or
adjustments that are necessary at the enterprise level and within each
business function to achieve desired business performance results and
mission outcomes.
Department of the Navy:
The Department of the Navy's enterprise architecture is comprised of
corporate architecture products, which include applicable laws,
regulations, and policies as well as enterprisewide reference models
and architecture content and subordinate architectures, which comprise
nine segment architectures, with each addressing a distinct functional
area such as logistics and command and control. These subordinate
architectures are to be developed by communities of practice groups
that have yet to be formally established. According to Navy officials,
the current approach calls for the Navy business enterprise
architecture content to be incorporated into the corporate enterprise
architecture and applicable segment architectures (e.g., corporate
management and support, force support, and logistics segment
architectures), as appropriate.
In 2008, our analysis showed that Navy had developed enterprise
architecture products that describe some, but not all, elements of its
current and target environments as well as a transition plan for its
business area. Specifically, we found that the Navy had not defined
its current core business processes or provided a comprehensive
picture of its current business problems that the department needs to
address. We also found that it had yet to develop a well-defined
target architecture, including the purpose and scope of all the
functional areas (e.g., joint logistics and planning). In addition, we
also found that, while the department had developed a transition plan
as part of its architecture, the plan was not based on a gap analysis
between the current and target environments. This is important to lay
out a road map for optimizing mission performance and transforming
business operations by systematically implementing changes to
technologies and processes.
Since 2008, Navy has identified its business segments such as
logistics and force support, representing some of its core business
processes. In addition, according to Navy officials, it plans to
establish communities of practice to oversee the development of
segment reference architectures and a road map for developing these
architectures. It has also added additional business-related
architecture content. For example, the architecture includes the
Navy's Common Operational Activity List that specifies some business
activities related to identifying and resolving accounting records
discrepancies. Such operational activities provide a basis from which
the department identifies the business activities or functions to be
reused (and optimized) across the department and those activities or
functions that remain unique to each business segment or domain.
Nonetheless, Navy does not have a well-defined enterprisewide business
architecture and transition plan to guide and constrain business
transformation initiatives. In particular, the department has not
developed a business enterprise architecture that provides a clear and
comprehensive picture of its current and target business environments.
For example, the enterprise architecture does not describe current and
target business capabilities, systems that are to be integrated to
support DOD end-to-end business processes, and information exchange
requirements among business activities. According to Navy officials,
rather than capturing the department's current and target business
environments, the focus of Navy's enterprise architecture is to
develop artifacts that are "actionable" such as enterprise rules for
assessing compliance of business systems. While such actionable
artifacts provide value, they do not provide a comprehensive and
systematic approach for transforming business operations. Further,
there is no evidence that the department has documented current
problems or defined the scope and purpose for all of the business-
related segment reference architectures. Documenting current problems
will enable the Navy to identify opportunities (e.g., new
applications, new processes, or new management approaches) for
improvement and to assess whether transformation efforts address these
problems.
Moreover, although Navy has developed an enterprise transition plan,
the plan is not well-defined and does not include an enterprise gap
analysis that identifies the differences between the current and
target business architectures, particularly the critical differences
or shortfalls that affect the successful accomplishment of the
department's mission. According to officials, gaps and shortfalls in
the current department programs can be identified through enterprise
architecture compliance assessments, and enterprise architecture
waivers are granted with specific expiration dates and conditions that
are to be met to address the gaps and shortfalls. Nevertheless, the
focus of these compliance assessments is on gaps and shortfalls of
individual programs (e.g., limited system availability) and not on
gaps and shortfalls from the perspective of end-to-end business
processes (e.g., process gaps) that are needed to achieve the target
environment. Moreover, a well-defined business enterprise architecture
and transition plan is essential for establishing an implementation
road map to address key gaps and shortfalls that can significantly
impact business operations across the department and for evolving and
developing business systems that optimize their mission value.
Fiscal Year 2012 Budget Submission Did Not Include Key Information on
All Business Systems:
Among other things, the Fiscal Year 2005 NDAA requires DOD's annual IT
budget submission to include key information on each business system
for which funding is being requested, such as the system's designated
approval authority and the appropriation type and amount of funds
associated with modernization (i.e., development) and current services
(i.e., operations and maintenance).
The department's fiscal year 2012 budget submission includes a range
of information for 1,637[Footnote 59] business system investments.
[Footnote 60] Of these, 272 involve development and modernization. For
each of the 272, the information provided includes the system's name,
approval authority, and appropriation type. The submission also
identifies the amount of the fiscal year 2012 request that is for
development and modernization versus operations and maintenance. For
systems in excess of $1 million in modernization funding, the
submission also cites its certification status (e.g., approved,
approved with conditions, approved decertification close-out,
[Footnote 61] and withdrawn[Footnote 62]) and the Defense Business
Systems Management Committee approval date, where applicable.
However, similar to prior budget submissions, the fiscal year 2012
budget submission still does not reflect all business system
investments. To prepare the submission, DOD relied on business system
investment information (e.g., funds requested, mission area, and
system description) that is entered by the components into DOD's
Select and Native Programming Data Input System--Information
Technology (SNAP-IT). In accordance with DOD guidance and according to
ASD(NII)/DOD CIO officials, the business systems listed in SNAP-IT
should match the systems listed in the Defense Information Technology
Portfolio Repository (DITPR)--the department's authoritative business
systems inventory. However, the DITPR data provided by DOD in March
2011 included 2,258 business systems. Therefore, SNAP-IT does not
reflect about 620 business systems that are identified in DITPR.
We previously reported that the information between SNAP-IT and DITPR
were not consistent and accordingly made a recommendation for DOD to
develop and implement plans for reconciling and validating the
completeness and reliability of information in its DITPR and SNAP-IT
system data repositories, and to include information on the status of
these efforts in the department's fiscal year 2010 report in response
to the act.[Footnote 63] DOD agreed with the need to reconcile
information between the two repositories and stated that it had begun
to take actions to address this. However, according to the Office of
the ASD(NII)/DOD CIO, efforts to provide automated SNAP-IT and DITPR
integration work were delayed due to increased SNAP-IT requirements in
supporting the fiscal year 2012 budget submission and ongoing
reorganization efforts within DOD. The department plans to restart the
process of integrating the two systems beginning in the third quarter
of fiscal year 2011. Until DOD has a reliable, comprehensive inventory
of all defense business systems, it will not be able to ensure the
completeness and reliability of the department's IT budget
submissions. Moreover, the lack of current and accurate information
increases the risk of oversight decisions that are not prudent and
justified.
DOD Has Continued to Establish Investment Management Processes but Has
Yet to Fully Define and Implement Key Practices:
Since our 2009 report, DOD has continued to establish investment
management processes but has not fully defined all key practices.
Further, with regard to certifying and overseeing investments--two key
DOD IT management processes for selecting, managing, and monitoring
investments--the department largely followed these processes for four
department investments we reviewed,[Footnote 64] but key steps
integral to these processes were not performed. Until DOD fully
defines these key practices and performs integral key steps, it is
unlikely that the department's reported 2,258 business system
investments--totaling $17.4 billion in fiscal year 2011--will be
managed in an effective manner that maximizes mission performance
while minimizing or eliminating system overlap and duplication.
DOD Has Continued to Establish Effective Investment Management
Processes, but Has Yet to Fully Define Many Key Processes and
Associated Policies and Procedures:
Since we reported in 2009, DOD has continued to make progress in
establishing the kind of investment management processes and
associated key practices (i.e., policies and procedures) called for in
the act and our ITIM framework[Footnote 65] as being integral to
effective IT investment management. Specifically, since 2009, Air
Force, Navy, and Army have implemented additional key practices
associated with effectively managing investments as individual
business system programs (Stage 2) and as portfolios of programs
(Stage 3), while the DOD-level organizations (herein referred to as
DOD enterprise) responsible for DOD-level processes have not. With
regard to Stage 2 practices, Navy and Army implemented two key
practices, and Air Force implemented one such practice. For Stage 3,
Navy implemented one key practice. For those key practices that have
yet to be fully defined, DOD enterprise and the military departments
have in large part partially defined these practices. Nonetheless,
even with this progress, DOD enterprise and the military departments
have yet to fully define a majority of the Stage 2 and Stage 3
practices.
Table 6 provides a summary (by DOD enterprise and each military
department) of the key Stage 2 practices implemented since 2009 along
with those practices we reported in 2009 as having been implemented.
The table also includes those practices yet to be implemented. As
such, table 6 provides an overall snapshot of where DOD enterprise and
the military departments stand with regard to building their
investment management foundation, including the ability to manage
investments as individual business system programs.
Table 6: Summary of Key Practices for Stage 2 Critical Processes-
Building the Investment Foundation:
Critical process: Instituting the investment board;
Key practice: An enterprisewide IT investment board composed of senior
executives from IT and business units is responsible for defining and
implementing the organization's IT investment governance process;
DOD enterprise: Key practice was implemented in or before May 2009;
Air Force: Key practice was implemented in or before May 2009;
Navy:Key practice was implemented since May 2009;
Army: Key practice is not implemented.
Critical process: Instituting the investment board;
Key practice: The organization has a documented IT investment process
directing each investment board's operations;
DOD enterprise: Key practice is not implemented;
Air Force: Key practice was implemented since May 2009;
Navy: Key practice is not implemented;
Army: Key practice is not implemented.
Critical process: Meeting business needs;
Key practice: The organization has documented policies and procedures
for identifying IT projects or systems that support the organization's
ongoing and future business needs;
DOD enterprise: Key practice was implemented in or before May 2009;
Air Force: Key practice was implemented since May 2009;
Navy: Key practice was implemented since May 2009;
Army: Key practice is not implemented.
Critical process: Selecting an investment;
Key practice: The organization has documented policies and procedures
for selecting a new investment;
DOD enterprise: Key practice is not implemented;
Air Force: Key practice is not implemented;
Navy: Key practice is not implemented;
Army: Key practice is not implemented.
Critical process: Selecting an investment;
Key practice: The organization has documented policies and procedures
for reselecting ongoing investments;
DOD enterprise: Key practice is not implemented;
Air Force: Key practice is not implemented;
Navy: Key practice is not implemented;
Army: Key practice is not implemented.
Critical process: Selecting an investment;
Key practice: The organization has documented policies and procedures
for integrating investment funding with investment selection;
DOD enterprise: Key practice is not implemented;
Air Force: Key practice is not implemented;
Navy: Key practice is not implemented;
Army: Key practice is not implemented.
Critical process: Providing investment oversight;
Key practice: The organization has documented policies and procedures
for management oversight of IT projects and systems;
DOD enterprise: Key practice is not implemented;
Air Force: Key practice is not implemented;
Navy: Key practice is not implemented;
Army: Key practice is not implemented.
Critical process: Capturing investment information;
Key practice: The organization has documented policies and procedures
for identifying and collecting information about IT projects and
systems to support the investment management process;
DOD enterprise: Key practice was implemented in or before May 2009;
Air Force: Key practice was implemented in or before May 2009;
Navy: Key practice was implemented since May 2009;
Army: Key practice was implemented since May 2009.
Key practice: An official is assigned responsibility for ensuring that
the information collected during project and systems identification
meets the needs of the investment management process;
DOD enterprise: Key practice was implemented in or before May 2009;
Air Force: Key practice was implemented in or before May 2009;
Navy: Key practice was implemented since May 2009;
Army: Key practice was implemented since May 2009.
Source: GAO.
[End of table]
As shown in the table, since 2009:
* DOD enterprise has not implemented any additional key practices.
* Air Force has implemented one key practice--documenting policies for
meeting business needs. Specifically, in its IT Investment Review
Guide, Air Force defines a process for ensuring that IT business
system investments support the department's ongoing and future
business needs. This process includes having an IRB--consisting of
senior executives from IT and functional business units, including the
Office of the Air Force CIO--to regularly review all business systems,
including those in operations and maintenance, to assess their
alignment with business needs using factors such as how well
investments support the Air Force's mission and their strategic value
and risk.
* Navy has implemented two additional key practices--(1) instituting
an enterprisewide IT investment board and (2) documenting policies for
meeting business needs. Specifically, in March 2011, Navy established
an Information Enterprise Governance Board--consisting of senior
executives from IT and functional business units, including the Navy
CIO--to serve as a business systems IRB. Among other things, the board
is responsible for business system investment governance, including
approving and annually reviewing business system investments. In
addition, for meeting business needs, Navy's Investment Review Guide
dated October 2009 defines a process for conducting annual reviews of
ongoing IT investments to ensure they support ongoing and future
business needs. The process calls for the annual review of all
business systems, including those in operations and maintenance, to
demonstrate that they support ongoing and future business needs by,
among other things, complying with applicable strategic business
guidance such as DOD's business enterprise architecture (BEA).
* Army has implemented two key practices associated with capturing
investment information. First, it has established policies and
procedures for collecting information about the department's
investments. Specifically, Army's investment review guide dated March
2010 defines procedures directing Army's system owners to submit,
update, and maintain IT projects and system information in a
departmental data repository called the Army Portfolio Management
Solution. Second, Army has assigned responsibility for investment
information collection and accuracy. Specifically, Army's investment
review guide states that system owners are responsible for the
accuracy of their data in the repository.
With regard to Stage 3 key practices, the following table provides a
summary (by DOD enterprise and each military department) of the key
practices implemented since 2009 and those practices that have yet to
be implemented. Table 7 provides an overall snapshot of where DOD
enterprise and the military departments stand in having the capability
to build a complete investment portfolio.
Table 7: Summary of Key Practices for Stage 3 Critical Processes-
Developing a Complete Investment Portfolio:
Critical process: Defining the portfolio criteria;
Key practice: The organization has documented policies and procedures
for creating and modifying IT portfolio selection criteria;
DOD enterprise: Key practice is not implemented;
Air Force: Key practice is not implemented;
Navy: Key practice is not implemented;
Army: Key practice is not implemented.
Critical process: Defining the portfolio criteria;
Key practice: Responsibility is assigned to an individual or group to
manage the development and modification of the IT portfolio selection
criteria;
DOD enterprise: Key practice was implemented in or before May 2009;
Air Force: Key practice was implemented in or before May 2009;
Navy: Key practice was implemented since May 2009;
Army: Key practice is not implemented.
Critical process: Creating the portfolio;
Key practice: The organization has documented policies and procedures
for analyzing, selecting, and maintaining the investment portfolios;
DOD enterprise: Key practice is not implemented;
Air Force: Key practice is not implemented;
Navy: Key practice is not implemented;
Army: Key practice is not implemented.
Critical process: Evaluating the portfolio;
Key practice: The organization has documented policies and procedures
for reviewing, evaluating, and improving the performance of its
portfolio(s);
DOD enterprise: Key practice is not implemented;
Air Force: Key practice is not implemented;
Navy: Key practice is not implemented;
Army: Key practice is not implemented.
Critical process: Conducting post-implementation reviews;
Key practice: The organization has documented policies and procedures
for conducting post-implementation reviews;
DOD enterprise: Key practice is not implemented;
Air Force: Key practice is not implemented;
Navy: Key practice is not implemented;
Army: Key practice is not implemented.
Source: GAO.
[End of table]
As shown in the table, although DOD enterprise, Air Force, and Army
have not implemented additional key practices, Navy has implemented
one key practice associated with defining portfolio criteria--
assigning responsibility to an individual or group to manage the
development and modification of IT portfolio selection criteria.
Specifically, Navy developed guidance that assigns Mission Area Leads
[Footnote 66] and Functional Area Managers[Footnote 67] with
responsibility for portfolio selection criteria.
With regard to the Stage 2 and Stage 3 key practices that have yet to
be fully implemented, it is important to note that DOD and the
military departments have partially defined these practices. For
example, for selection, DOD established a process that calls for
investments involving more than $1 million in obligations to be
certified and approved before funds are to be expended. Specifically,
the process calls for investments to be, among other things, compliant
with DOD's BEA and be economically justified. However, the process
does not fully address all aspects of the selection key practice. For
example, the process does not specify how the IRBs are to use the full
range of cost, schedule, and benefit data in making selection (i.e.,
certification) decisions, as called for in our ITIM framework.
In addition, for oversight, DOD has established an oversight process
that calls for, among other things, investments to be reviewed
annually to assess how each is performing. As part of the process, the
IRBs assess program performance relative to cost, schedule, and
capability commitments. However, DOD's oversight process does not
provide sufficient visibility into the military department's
investment management activities, including its reviews of systems in
operations and maintenance and smaller investments, commonly referred
to as Tier 4 investments.
Nonetheless, DOD enterprise and the military departments have still
not fully defined these Stage 2 and 3 key practices. Specifically,
with regard to the nine Stage 2 practices, DOD enterprise, Air Force,
and Navy, as shown in table 6, have yet to fully define five key
practices (or 56 percent of the practices), and Army has yet to do so
for seven (or 78 percent) of the practices.
With regard to the five Stage 3 practices, DOD enterprise, Air Force,
and Navy, as shown in table 7, have yet to fully define four key
practices (or 80 percent of the practices), and Army has yet to do so
for any (100 percent) of the practices.
Officials from DOD enterprise and the military departments attributed
the incomplete state of their IT investment management processes to
the following:
* DOD enterprise officials said the condition of its processes,
including the lack of progress since 2009, was due in part to current
ongoing DOD efforts to reorganize the business systems governance
organizations (e.g., the Business Transformation Agency) that are
responsible for implementing IT investment management at the DOD
enterprise level. As noted earlier, in August 2010, the Secretary of
Defense announced the plans to disestablish the Business
Transformation Agency and that the functions of the Business
Transformation Agency, such as its IT investment management function,
be reviewed and transferred to other organizations in DOD, as
appropriate. The DOD officials stated that these implementation plans
have yet to be finalized and have resulted in their investment
management implementation efforts being delayed. These officials added
that they are aware of the absence of documented project-level and
portfolio-level management practices; they also said they are
currently working on developing policies and procedures to address the
missing processes and practices but were not able to provide us with
milestones and a plan with defined steps for when the policies and
procedures were to be completed.
* Air Force and Navy officials said their investment management
implementation efforts were taking longer than originally planned
given their workload and other priorities assigned to them since
initiating investment management efforts. They also acknowledged that
their processes were missing certain documented project-level and
portfolio-level management policies and procedures and said they were
in the process of developing policies and procedures to address these
missing processes and practices. In particular, Air Force officials
stated that they planned to have their policies and procedures
approved and finalized by October 2011.
* Army officials said the state of their IT investment management
process is due to the fact that the department focused on first
establishing selected institutional capabilities--such as defining
roles and responsibilities and establishing a department-level IRB--
rather than attempting to do everything at once. The officials added
that once its initial steps are completed, Army intends to then focus
on implementing remaining Stage 2 and three key processes and
practices. Specifically, these officials said that they are aware that
Army lacks a military department-level IRB and added that until now
they have been relying on functional area experts to review
investments before they are sent to the DOD IRBs. They also
acknowledged that Army is missing certain documented project-level and
portfolio-level management policies and procedures. They further
stated that the department is currently working on guidance to address
these missing items with the goal of having it approved and finalized
by August 2011.
As discussed in our ITIM framework and previous reports on DOD's
investment management of its business systems,[Footnote 68] adequately
documenting both policies and associated procedures that govern how an
organization manages its IT projects and investment portfolios is
important because doing so provides the basis for rigor, discipline,
and repeatability in how investments are selected and controlled
across the entire organization. Until DOD fully defines missing
policies and procedures, it is unlikely that the department's reported
2,258 business systems will be managed in a consistent, repeatable,
and effective manner that, among other things, maximizes mission
performance while minimizing or eliminating system overlap and
duplication. To this point, there is evidence showing that DOD is not
managing its systems in this manner. For example, DOD reported that of
its 79 major business and other IT investments, roughly a third are
encountering cost, schedule, and performance shortfalls requiring
immediate and sustained management attention. In addition, we have
consistently reported[Footnote 69] for some time that DOD's business
system environment has been characterized by (1) little
standardization, (2) multiple systems performing the same tasks, (3)
the same data stored in multiple systems, and (4) manual data entry
into multiple systems. Because DOD spends over $10 billion each year
on its business systems and related IT infrastructure, the potential
for identifying and avoiding the costs associated with duplicative
functionality across its business system investments is significant.
DOD Has Largely Followed Its Certification and Oversight Processes for
the Investments Under Review, but Validation and Other Key Steps Were
Not Performed:
As discussed, certification and oversight are key DOD processes for
selecting, and overseeing IT investments. DOD guidance[Footnote 70]
calls for investments to be certified and approved before funds are to
be expended on modernization activities. More specifically, the
guidance states that investments involving more than $1 million in
obligations are to be certified by designated approval authorities and
as part of that certification, authority officials are to attest that
each investment:
* is compliant with DOD's BEA, including all relevant architecture
products, such as products that specify the technical standards needed
to promote interoperability among related systems or examine overlaps
with other business systems;
* is economically justified, based on an economic viability analysis
developed using disciplined and rigorous cost estimating practices;
and:
* has undergone sufficient business process reengineering analysis,
including identifying and developing approaches to streamlining and
improving involved processes.
As part of each of these three requirements, we have said it is
important for designated approval authorities to validate the results
of BEA and other assessments to ensure investment decision making is
based on accurate and reliable information.[Footnote 71] More
specifically, we previously reported that DOD had not been performing
this step and made recommendations that they do so.[Footnote 72] DOD
agreed with our findings and recommendations, and stated that it
planned to assign validation responsibilities and issue guidance that
describes the methodology for performing validation activities but
were not able to provide a date for when this would be completed.
Once investments have been certified, DOD guidance calls for
investments to be effectively overseen. This includes reviewing
annually the progress of investments using predefined criteria and
checkpoints, in meeting cost, schedule, risk, and benefit expectations.
Consistent with this, DOD guidance calls for IRBs to annually review
certified investments and in doing so, to focus on program performance
against cost, schedule, and performance baselines, and progress in
meeting the certification conditions discussed. Our ITIM research and
experience with federal agencies also shows that it is important for
oversight and other decisionmaking authorities to validate performance
information used to make decisions so that investment decision making
is based on accurate and reliable information.[Footnote 73]
Certification:
DOD largely followed the certification process for each of the four
investments we reviewed, but did not perform validation and other key
aspects of the process. Specifically:
BEA compliance. DOD enterprise and the military departments took steps
to assess BEA compliance of their respective systems. This included
following DOD guidance (the appropriate version of DOD's BEA
Compliance Guidance) and using an automated tool (called Architecture
Compliance and Requirements Traceability)[Footnote 74] to determine
and report on the extent of each system's architectural compliance. In
addition, in each case, once the BEA assessment had been completed,
the appropriate DOD enterprise and military department
precertification authorities asserted (via memorandum of
certification) that each system was compliant with DOD's BEA.
For example, on the Project Management Resource Tool (PMRT) project,
Air Force followed the BEA compliance guidance and used the
Architecture Compliance and Requirements Traceability tool to develop
a compliance report that mapped BEA activities to PMRT's capabilities.
In August 2010, Air Force's Director of Business Transformation and
Deputy Chief Management Officer stated in a supporting
precertification memorandum that this information was used to assert
that PMRT was compliant with DOD's BEA.
In another example, on the Defense Travel System, DOD also assessed
BEA compliance by using the BEA Compliance Guidance and the
Architecture Compliance and Requirements Traceability tool to develop
a report showing extent of compliance. In an October 2010,
precertification memo, DOD's CIO said this information was used to
assert that the system was compliant with DOD's BEA.
Although DOD took these steps to certify BEA compliance, it did not
take other key steps. For example, DOD and component designated
approval authorities did not validate the assessments and assertions.
Specifically, the BEA compliance assessments performed on the
investments under review were not validated by DOD certification and
approval entities.[Footnote 75] Although this was not done, the
systems were nevertheless certified as compliant. We reported[Footnote
76] on this weakness in 2008 and made recommendations to DOD to, among
other things, explicitly assign responsibility for validating program
BEA compliance assessments and issue guidance that describes the
methodology for performing such validation activities. To date, DOD
has yet to implement these recommendations. However, DOD officials
told us the department has actions planned or underway to address
these recommendations, although they were not able to provide
milestones for when this would be accomplished.
In addition, our 2008 report showed that DOD BEA assessments did not
include all relevant architecture products, such as products that
specify the technical standards needed to promote interoperability
among related systems or examine overlaps with other business systems.
[Footnote 77] Despite the limited assessments, DOD nonetheless
certified the investments as BEA compliant even though they did not
adequately demonstrate such compliance. Accordingly, we have made
recommendations to DOD to revise its BEA compliance guidance, among
other things, to address these shortfalls. To date, DOD has yet to
implement the recommendations. However, DOD officials said the
department has actions planned or underway to address the
recommendations but they were not able to provide a date for when this
would be completed.
Economic viability analysis. For the investments under review, DOD
enterprise and the military departments used DOD's IT Investment
Review Process Guidance (dated January 2009) that specifies how
investment economic viability is to be analyzed and assessed. They
also used a related automated tool designed to support the development
of such analyses. Once the analyses had been performed, the
precertification authorities for each of the systems asserted that the
efforts had been reviewed, and showed the investments were
economically justified to proceed with obligating funds.
For example, on the Logistics Modernization Program, Army used DOD's
guidance to conduct its economic viability analysis. The Army also
used the economic viability tool to complete the analysis. In
addition, in December 2010 memorandum, the Army's Chief Management
Officer asserted that the economic viability analysis had been
completed, and showed the investment was justified to proceed with
obligating funds.
In another example, on the Navy Enterprise Resource Planning system,
Navy used the January 2009 DOD guidance to comply with the economic
viability analysis requirement and used the economic viability tool to
complete the analysis. Further, in a July 2010 memo, the Navy's Chief
Management Officer asserted that the investment's economic viability
analysis had been completed, and showed the investment was justified
to obligate funds.
Although DOD enterprise and the military departments took these steps
to justify the investments, they did not perform other key steps.
Specifically, DOD enterprise and the military departments did not use
important cost estimating practices critical to developing such
analyses. For example, in developing its economic justification for
its ERP system, Navy did not implement key aspects of earned value
management or develop risk mitigation strategies to address this risk.
We have previously reported[Footnote 78] on these weaknesses and made
recommendations to address them. Although the recommendations are
still open, DOD enterprise and military department officials have said
they have actions planned and under way to address the
recommendations. However, they were not able to provide a timetable
for when the actions are to be completed.
Business process reengineering assessment. For the investments under
review, DOD enterprise and the military departments used DOD's
Business Process Reengineering Guidance (dated April 2011) to assess
whether the investments complied with the business process
reengineering requirement. Consistent with the guidance, DOD
enterprise and the military departments completed questionnaires
(contained in the guidance) that aim to help DOD enterprise and the
military departments identify and develop approaches to streamlining
and improving existing business processes. Once these assessments had
been completed, the DOD enterprise and military department
precertification authorities asserted that business process
reengineering assessments had been performed.
For example, on the PMRT project, Air Force used the DOD reengineering
guidance to assess whether there were ways to streamline and improve
existing business processes to be supported by the system investment.
Air Force completed the assessment in July 2010. Air Force reported
that as part of this assessment, it had representatives from the
offices of the CIO and the Deputy Chief Management Officer review the
completed assessment questionnaire and supporting documentation to
determine whether the project team had followed the reengineering
requirement. Subsequently, in August 2010, the Air Force's Director of
Business Transformation and the Deputy Chief Management Officer used
this information to assert that sufficient business process
reengineering had been conducted in order for the program to obligate
investment funding.
While DOD enterprise and military department precertification
authorities largely followed DOD's guidance, they did not perform the
key step of validating the results of these reengineering assessments
to ensure they, among other things, accurately assessed process
weaknesses and identified opportunities to streamline and improve
affected processes. We have ongoing work on actions the Air Force and
Army have taken to comply with statutory requirements regarding
business process reengineering.
The reason DOD did not follow key aspects of the certification
process--primarily not validating assessment results--is attributed in
part to unclear roles and responsibilities. According to military
department officials responsible for the investments we reviewed,
validation activities did not occur because DOD policy and guidance
does not explicitly require them to be performed and there is no
guidance that specifies how assessments should be validated. According
to DOD officials, the oversight and designated approval authorities
did not validate the DOD enterprise-level assessments and assertions
because DOD policy and guidance has not yet been revised to require
these authorities to do so. Consequently, until the policy and
guidance is updated and roles and responsibilities with regard to who
is to perform validation functions are clearly defined, there is an
increased risk that DOD will be making business system investment
decisions based on information that is inaccurate and unreliable.
Oversight:
For the investments under review, DOD largely followed its oversight
process but did not perform an important validation activity.
Specifically, DOD's oversight process (as specified in the January
2009 Investment Review Guide) calls for investments to be reviewed
annually to assess how each is performing. As part of this process,
the IRBs assess program performance relative to, among other things,
cost, schedule, and capability commitments. The IRBs do this using
updated information provided by the programs and screened by DOD
enterprise and military department precertification authorities for
completeness. These oversight reviews are important because an
investment board should have visibility into each project's
performance and progress toward predefined cost, schedule, and benefit
expectations as well as each project's exposure to risk. Without such
visibility and validated information, organizations risk making
investment decisions that are inconsistent and are not fully grounded
in reliable and accurate data.
Consistent with this direction, DOD conducted (or is planning to
conduct) annual reviews for the investments we reviewed.[Footnote 79]
For example, DOD conducted annual reviews for the Defense Travel
System in December 2010 and the Navy Enterprise Resource Planning
system in July 2010. In doing these reviews, DOD assessed investment
performance using the cost, schedule, and performance information
provided by the programs and screened by precertification authorities.
Although DOD largely followed its oversight process, it did not
validate the cost, schedule, and performance information used for
decision making. This finding is consistent with our previous
report[Footnote 80] noting that DOD's oversight process does not
provide for sufficient visibility into the military department's
investment management activities, including its reviews of systems in
operations and maintenance and smaller investments, commonly referred
to as Tier 4 investments. Such visibility is important because DOD
reports that only 100 of approximately 2,258 total business systems
are annually reviewed by the IRBs. This means that the vast majority
of business systems are overseen only within the military departments.
Accordingly, we have made recommendations to address this area. DOD
officials said they plan to address the recommendations but were
unable to provide a schedule for when this work is to be completed.
In explaining why the information used by the IRBs is not validated,
DOD officials cited the same reasons--outdated policy and guidance and
unclear roles and responsibilities--as those provided for the lack of
validation in the investment certification process. Consequently,
until such roles and responsibilities are clarified, DOD faces
increased risk that it will not effectively be able to oversee its
extensive business systems investments.
DOD's Annual Report Describes Certification Actions for Its Business
System Investments:
Among other things, the act[Footnote 81] requires DOD to submit an
annual report to congressional committees on DOD's compliance with
requirements of the act, including a description of specific actions
the department has taken on each business system modernization
investment submitted for certification. The act further requires that
such investments involving more than $1 million in obligations must be
certified by a designated approval authority[Footnote 82] as meeting
specific criteria, such as demonstrating compliance with DOD's
business enterprise architecture.[Footnote 83] Further, the act
requires the Defense Business Systems Management Committee to approve
each of these certifications.
In May 2010,[Footnote 84] we reported that the department's annual
report did not discuss certification actions for all systems on which
certification actions had been taken, primarily excluding business
system recertifications. We recommended the Deputy Secretary of
Defense expand future DOD annual reports to Congress to include all
certification actions that had been taken in the previous year by the
department on its business system modernization investments. DOD
agreed with our recommendation and stated that it would include
recertifications in future reports. Since then, the department has
addressed this recommendation by including all types of certification
actions in its 2011 annual report, including recertification actions.
As it has since 2005, DOD continues to certify and approve business
system modernization investments in excess of $1 million.
DOD's annual report identifies IRB certification actions associated
with 137 business system investments that underwent the IRB
certification and Defense Business Systems Management Committee
approval process for fiscal year 2010 and cost approximately $1.3
billion. Specifically, the annual report accurately states that during
fiscal year 2010, 52 unique business system modernizations were
certified--35 with and 17 without conditions. For the 35 systems, 32
conditions were reported. Examples of conditions cited in the report
are the need for business enterprise architecture compliance to
improve interoperability and integration of cross-functional processes
and improved program management functions. The report also identifies
93 recertifications and 28 decertifications. For example, the Navy's
Enterprise Resource Planning system had about $7.6 million recertified
in early August and another $96.6 million recertified later in the
month.
While DOD has made progress by reporting its certification actions,
the basis for these certification actions and subsequent approvals is
limited as discussed in the previous section.
Conclusions:
A well-defined federated architecture and accompanying transition
plans for the business mission area, along with well-defined
investment management policies and procedures across all levels of the
department, are critical to effectively addressing DOD's business
systems modernization high-risk area. Relatedly, it is important for
the department to obtain independent assessments of the completeness,
consistency, understandability, and usability of the federated family
of business mission area architectures, including associated
transition plans. Equally important is for the department to actually
implement its architecture and investment management controls in the
years ahead on each and every business system investment, and in doing
so ensure that it has reliable information on each investment on which
to base executive decision making.
DOD has continued to take steps in defining and implementing these key
institutional modernization management controls, but challenges that
we identified in prior years still need to be addressed. Specifically,
while DOD continues to release updates to its corporate enterprise
architecture, the architecture has yet to be federated through
development of aligned subordinate architectures for each of the
military departments. In this regard, each of the military departments
has made progress in managing its respective architecture program, but
there are still limitations in the scope and completeness, as well as
the immaturity of the military department architecture programs,
including the completeness of their own transition plans. In addition,
while DOD continues to establish investment management processes, the
DOD enterprise and the military departments' approaches to business
systems investment management still lacks the defined policies and
procedures to be considered effective investment selection, control,
and evaluation mechanisms. Finally, information used to support the
development of DOD's budget requests, as well as to inform
certification decisions, is still of questionable reliability.
Collectively, these long-standing limitations in the department's
institutional modernization management controls continue to put the
billions of dollars spent annually on thousands of business system
investments at risk. Our previous recommendations to the department
have been aimed at accomplishing these and other important activities
related to its business systems modernization. To the department's
credit, it has agreed with these recommendations and is committed to
implementing them.
However, the state of progress of DOD and military department business
system modernization efforts is due, in part, to uncertainty and
pending decisions surrounding the roles and responsibilities of key
organizations and senior leadership positions. Accordingly, it is
essential that DOD resolve these matters expeditiously, as doing so is
on the department's critical path for fully establishing the full
range of institutional management controls needed to address its
business systems modernization high-risk area.
Recommendation for Executive Action:
Because we have existing recommendations that address the
institutional management control weaknesses discussed in this report,
we are making no further recommendations in these areas.
To address the uncertainty and pending decisions surrounding the roles
and responsibilities of key organizations, we recommend that the
Secretary of Defense expeditiously complete the implementation of the
announced transfer of functions of the Business Transformation Agency
and the Office of the Assistant Secretary of Defense for Networks and
Information Integration/Department of Defense CIO and provide
specificity as to when and where these functions will be transferred.
Agency Comments and Our Evaluation:
In written comments on a draft of this report, signed by the Deputy
Chief Management Officer and reprinted in appendix II, the department
agreed with our recommendation and stated that it expects to announce
the implementation details concerning the transfer of functions of the
Business Transformation Agency and the Office of the Assistant
Secretary of Defense for Networks and Information Integration/
Department of Defense CIO prior to June 30, 2011.
We support the department's efforts to address our recommendation and
reiterate the importance of following through in implementing the
recommendation within the stated time frame.
We are sending copies of this report to interested congressional
committees; the Director, Office of Management and Budget; and the
Secretary of Defense. This report will also be available at no charge
on our Web site at [hyperlink, http://www.gao.gov].
If you or your staffs have any questions on matters discussed in this
report, please contact me at (202) 512-6304 or melvinv@gao.gov.
Contact points for our Offices of Congressional Relations and Public
Affairs may be found on the last page of this report. GAO staff who
made major contributions to this report are listed in appendix III.
Signed by:
Valerie C. Melvin:
Director:
Information Management and Human Capital Issues:
List of Committees:
The Honorable Carl Levin:
Chairman:
The Honorable John McCain:
Ranking Member:
Committee on Armed Services:
United States Senate:
The Honorable Daniel Inouye:
Chairman:
The Honorable Thad Cochran:
Ranking Member:
Subcommittee on Defense:
Committee on Appropriations:
United States Senate:
The Honorable Howard P. McKeon:
Chairman:
The Honorable Adam Smith:
Ranking Member:
Committee on Armed Services:
House of Representatives:
The Honorable C.W. Bill Young:
Chairman:
The Honorable Norman Dicks:
Ranking Member:
Subcommittee on Defense:
Committee on Appropriations:
House of Representatives:
[End of section]
Appendix I: Objective, Scope, and Methodology:
As agreed with congressional defense committees, our objective was to
assess the actions by the Department of Defense (DOD) to comply with
provisions of section 332 of the Ronald W. Reagan National Defense
Authorization Act for Fiscal Year 2005.[Footnote 85] This included (1)
developing a business enterprise architecture and a transition plan
for implementing the architecture, (2) identifying systems information
in its annual budget submission, (3) establishing a system investment
approval and accountability structure along with an investment review
process, and (4) certifying and approving any system modernizations
costing in excess of $1 million. (See the background section of this
report for additional information on the act's requirements.) Our
methodology relative to each of these four provisions is as follows:
* To address the architecture and transition plan provision, we
focused on the progress the departments of the Air Force, Army, and
Navy have made in developing their respective parts of the federated
DOD business enterprise architecture. In doing so, we compared the
baseline enterprise architecture program status information as
presented in our 2008 report,[Footnote 86] with information on the
current status of each military department's enterprise architecture
program. In doing so, we focused on those select elements that were
either similar or slightly modified across versions 1.1 and 2.0 of our
Enterprise Architecture Management Maturity Framework[Footnote 87] and
that were either partially or not satisfied by one or more of the
military departments. Specifically, we reviewed written responses and
supporting documentation on steps completed, under way, or planned
from each military department to identify examples of progress made in
addressing those elements that we had previously identified as being
not satisfied or partially satisfied. We also reviewed business
architectural artifacts to determine the progress each department had
made in developing their respective business architectural content
since we last reported in 2008.[Footnote 88] We interviewed cognizant
DOD officials to validate the responses and identify any
discrepancies. Further, we reviewed the independent verification and
validation contractor's statement of work and other work products to
determine whether they addressed the department's federated family of
corporate and subordinate architectures.
* To determine whether DOD's fiscal year 2012 IT budget submission was
prepared in accordance with the criteria set forth in the act, we
reviewed and analyzed the Report on Defense Business System
Modernization FY 2005 National Defense Authorization Act, Section 332,
dated March 2011 and compared it to the specific requirements in the
act. We also compared information contained in the department's system
that is used to prepare its budget submission (SNAP-IT) with
information in DOD's Defense Information Technology Portfolio
Repository (DITPR) system to determine if DOD's fiscal year 2012
budget request included all business systems. We interviewed Assistant
Secretary of Defense for Networks and Information Integration/
Department of Defense Chief Information Officer officials to discuss
the accuracy and comprehensiveness of information contained in the
SNAP-IT system, the discrepancies in the information contained in the
DITPR and SNAP-IT systems, and efforts under way or planned to address
these discrepancies. We did not independently validate the reliability
of the cost and budget figures provided by DOD because the specific
amounts were not relevant to our findings.
* To assess the establishment of DOD enterprise and component
investment management structures and processes, we analyzed whether
DOD and its military departments' information technology investment
management processes were compliant with federal guidance and the
extent to which DOD and the military departments were following their
investment management processes, including those at the DOD enterprise-
level for approving and certifying investments. To perform the first
task, we compared the status of DOD enterprise and military department
(Air Force, Army, and Navy) investment management processes--as noted
in our May 2009 report[Footnote 89] and other sources--with the
current status of these organization's processes. As part of this
analysis, we focused on the definition of project-level (Stage 2) and
portfolio-level (Stage 3) policies and procedures contained in our
Information Technology Investment Management (ITIM) Framework[Footnote
90] that were identified in our previous work as not being
established. Specifically, we analyzed written department responses
and supporting documentation on steps completed, under way, or planned
against ITIM key practices to identify where progress had been made in
addressing such previously identified practices. Where there were
variances (i.e., support did not show the department was meeting a key
practice), we reviewed relevant documentation and interviewed
appropriate DOD enterprise and military department officials to
identify the causes and impacts.
With regard to our second task, we selected four DOD enterprise-level
and military department-level investments that met the following
criteria: the investment was (1) either a (Tier 1 or 2) major
automated information system from key DOD functional areas (i.e.,
Weapon Systems Lifecycle Management; Materiel Supply and Services
Management; and Human Resources Management) and (2) was at a life
cycle phase--such as production and deployment and operations and
maintenance--where there were extensive opportunities for system
investment officials to demonstrate the organization was following
ITIM key practices.[Footnote 91] In reviewing these investments, we
focused on DOD enterprise and military department activities related
to certification and oversight, which are a key part of selecting,
managing, and overseeing IT investments as called for in our ITIM
framework and DOD guidance. For certification, we reviewed DOD
Investment Review Board guidance to understand the types of actions
related to the certification of business system modernizations and, in
doing so, focused on three certification requirements (e.g., ensuring
that designated approval authorities assert that each investment is
compliant with the business enterprise architecture). For each
requirement, we reviewed supporting documentation from DOD enterprise
and the military departments to determine whether there was a
documented process for how the requirement was to be certified and to
ascertain whether artifacts prepared as part of the process
demonstrated that the certification process was being followed. We did
the same for the oversight process. When there were variances between
the criteria and what DOD enterprise and the military departments had
done, we interviewed cognizant DOD enterprise-level and military
department-level officials on the causes and impacts.
* To determine whether the department was certifying and approving
business system investments with annual obligations exceeding $1
million, we reviewed and analyzed all Defense Business Systems
Management Committee certification approval memoranda as well as IRB
certification memoranda issued prior to the Defense Business Systems
Management Committee's final approval decisions for fiscal year 2010
and compared the results to those certification actions described in
the annual report to identify differences. We also reviewed DOD IRB
guidance to understand the types of actions related to certification
of business system modernizations. We interviewed officials from the
Business Transformation Agency and IRBs to discuss any discrepancies.
We conducted this performance audit at DOD and military department
offices in Arlington, Virginia, from January 2011 to June 2011, in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objective.
[End of section]
Appendix II: Comments from the Department of Defense:
Deputy Chief Management Officer:
9010 Defense Pentagon:
Washington, DC 20301-9010:
June 17 2011:
Ms. Valerie Melvin:
Director, Defense Capabilities and Management:
U.S. Government Accountability Office:
441 G Street, N.W.
Washington, DC 20548:
Dear Ms. Melvin:
This is the Department of Defense response to the U.S. Government
Accountability Office (GAO) Draft Report, GAO-II-684, 'Department of
Defense: Further Actions Needed to Institutionalize Key Business
System Modernization Management Controls,' dated June 10, 2011 (GAO
Code 310961). The Department concurs with the recommendation contained
in the draft report.
The Department appreciates the opportunity to respond to your draft
report. Should you have any questions, please contact Mr. Bryan
Kitchens, Bryan.Kitchens@bta.mil, 703-602-4743.
Sincerely,
Signed by:
Elizabeth A. McGrath:
Attachment: As stated.
[End of letter]
GAO Draft Report Dated June 10, 2011:
GA0-11-684 (GAO Code 310961):
"Department Of Defense: Further Actions Needed To Institutionalize Key
Business System Modernization Management Controls"
Department Of Defense Comments To The GAO Recommendation:
Recommendation 1: The GAO recommends that the Secretary of Defense
expeditiously complete the implementation of the announced transfer of
functions of the Business Transformation Agency (BTA) and the Office
of the Assistant Secretary of Defense for Networks and Information
Integration (ASD(NII))/Department of Defense CIO and provide
specificity as to when and where these function will be transferred.
DoD Response: Concur. Announcement of implementation details
concerning BTA and NII function transfers is expected to occur prior
to June 30, 2011.
[End of section]
Appendix III GAO Contact and Staff Acknowledgments:
GAO Contact:
Valerie C. Melvin (202) 512-6304 or melvinv@gao.gov:
Staff Acknowledgments:
In addition to the contact named above, key contributors to this
report were Gerard Aflague, Mathew Bader, Carl Barden, Shaun Byrnes,
Debra Conner, Elena Epps, Rebecca Eyler, Nancy Glover, Neelaxi
Lakhmani (Assistant Director), Anh Le, Lori Martinez, Gary Mountjoy,
Freda Paintsil, David Powner (Director), Christine San, Sylvia Shanks,
Donald Sebers, Teresa Smith, Jennifer Stavros-Turner, and Adam
Vodraska.
[End of section]
Footnotes:
[1] Business systems support DOD's business operations, such as
civilian personnel, finance, health, logistics, military personnel,
procurement, and transportation.
[2] GAO, High-Risk Series: An Update, [hyperlink,
http://www.gao.gov/products/GAO-11-278] (Washington, D.C.: February
2011).
[3] An enterprise architecture, or modernization blueprint, provides a
clear and comprehensive picture of an entity, whether it is an
organization (e.g., federal department or agency) or a functional or
mission area that cuts across more than one organization (e.g.,
financial management). This picture consists of snapshots of the
enterprise's current or "as is" operational and technological
environment and its target or "to be" environment, and contains a
capital investment road map for transitioning from the current to the
target environment. These snapshots consist of "views," which are
basically one or more architecture products that provide conceptual or
logical representations of the enterprise.
[4] GAO, Information Technology: Architecture Needed to Guide
Modernization of DOD's Financial Operations, [hyperlink,
http://www.gao.gov/products/GAO-01-525] (Washington, D.C.: May 17,
2001).
[5] GAO, DOD Business Systems Modernization: Long-standing Weaknesses
in Enterprise Architecture Development Need to Be Addressed,
[hyperlink, http://www.gao.gov/products/GAO-05-702] (Washington, D.C.:
July 22, 2005); DOD Business Systems Modernization: Billions Being
Invested without Adequate Oversight, [hyperlink,
http://www.gao.gov/products/GAO-05-381] (Washington, D.C.: Apr. 29,
2005); DOD Business Systems Modernization: Limited Progress in
Development of Business Enterprise Architecture and Oversight of
Information Technology Investments, [hyperlink,
http://www.gao.gov/products/GAO-04-731R] (Washington, D.C.: May 17,
2004); DOD Business Systems Modernization: Important Progress Made to
Develop Business Enterprise Architecture, but Much Work Remains,
[hyperlink, http://www.gao.gov/products/GAO-03-1018] (Washington,
D.C.: Sept. 19, 2003); Business Systems Modernization: Summary of
GAO's Assessment of the Department of Defense's Initial Business
Enterprise Architecture, [hyperlink,
http://www.gao.gov/products/GAO-03-877R] (Washington, D.C.: July 7,
2003); Information Technology: Observations on Department of Defense's
Draft Enterprise Architecture, [hyperlink,
http://www.gao.gov/products/GAO-03-571R] (Washington, D.C.: Mar. 28,
2003); DOD Business Systems Modernization: Improvements to Enterprise
Architecture Development and Implementation Efforts Needed,
[hyperlink, http://www.gao.gov/products/GAO-03-458] (Washington, D.C.:
Feb. 28, 2003); and [hyperlink,
http://www.gao.gov/products/GAO-01-525].
[6] Pub. L. No. 108-375, § 332, 118 Stat. 1811, 1851-1856 (Oct. 28,
2004) (codified in part at 10 U.S.C. § 2222).
[7] 10 U.S.C. § 2222(i), as amended.
[8] GAO, DOD Business Systems Modernization: Recent Slowdown in
Institutionalizing Key Management Controls Needs to Be Addressed,
[hyperlink, http://www.gao.gov/products/GAO-09-586] (Washington, D.C.:
May 18, 2009); DOD Business Systems Modernization: Military
Departments Need to Strengthen Management of Enterprise Architecture
Programs, [hyperlink, http://www.gao.gov/products/GAO-08-519]
(Washington D.C.: May 12, 2008); Business Systems Modernization:
Department of the Navy Needs to Establish Management Structure and
Fully Define Policies and Procedures for Institutionally Managing
Investments, [hyperlink, http://www.gao.gov/products/GAO-08-53]
(Washington, D.C.: Oct. 31, 2007); Business Systems Modernization: Air
Force Needs to Fully Define Policies and Procedures for
Institutionally Managing Investments, [hyperlink,
http://www.gao.gov/products/GAO-08-52] (Washington, D.C.: Oct. 31,
2007).
[9] GAO, Business Systems Modernization: DOD Continues to Improve
Institutional Approach, but Further Steps Needed, [hyperlink,
http://www.gao.gov/products/GAO-06-658] (Washington, D.C.: May 15,
2006).
[10] GAO, High-Risk Series: An Update, [hyperlink,
http://www.gao.gov/products/GAO-11-278] (Washington, D.C.: February
2011).
[11] These seven high-risk areas include DOD's overall approach to
business transformation, business systems modernization, contract
management, financial management, supply chain management, support
infrastructure management, and weapon systems acquisition.
[12] The seven governmentwide high-risk areas include disability
programs, ensuring the effective protection of technologies critical
to U.S. national security interests, interagency contracting,
information systems and critical infrastructure, information sharing
for homeland security, human capital, and real property.
[13] GAO, DOD Business Systems Modernization: Planned Investment in
Navy Program to Create Cashless Shipboard Environment Needs to Be
Justified and Better Managed, [hyperlink,
http://www.gao.gov/products/GAO-08-922] (Washington, D.C.: Sept. 8,
2008); DOD Travel Cards: Control Weaknesses Resulted in Millions of
Dollars of Improper Payments, [hyperlink,
http://www.gao.gov/products/GAO-04-576] (Washington, D.C.: June 9,
2004); Military Pay: Army National Guard Personnel Mobilized to Active
Duty Experienced Significant Pay Problems, [hyperlink,
http://www.gao.gov/products/GAO-04-89] (Washington, D.C.: Nov. 13,
2003); and Defense Inventory: Opportunities Exist to Improve Spare
Parts Support Aboard Deployed Navy Ships, [hyperlink,
http://www.gao.gov/products/GAO-03-887] (Washington, D.C.: Aug. 29,
2003).
[14] GAO, Strategic Information Planning: Framework for Designing and
Developing System Architectures, [hyperlink,
http://www.gao.gov/products/GAO/IMTEC-92-51] (Washington, D.C.: June
1992).
[15] 40 U.S.C. § 11315(b)(2).
[16] 44 U.S.C. § 3602(f)(14).
[17] GAO, Organizational Transformation: A Framework for Assessing and
Improving Enterprise Architecture Management (Version 2.0),
[hyperlink, http://www.gao.gov/products/GAO-10-846G] (Washington, D.C.
: August 2010); Information Technology Investment Management: A
Framework for Assessing and Improving Process Maturity, [hyperlink,
http://www.gao.gov/products/GAO-04-394G] (Washington, D.C.: March
2004); Information Technology: A Framework for Assessing and Improving
Enterprise Architecture Management, Version 1.1, [hyperlink,
http://www.gao.gov/products/GAO-03-584G] (Washington, D.C.: April
2003); OMB Capital Programming Guide, Version 1.0 (July 1997); and CIO
Council, A Practical Guide to Federal Enterprise Architecture, Version
1.0 (February 2001).
[18] [hyperlink, http://www.gao.gov/products/GAO-10-846G] and
[hyperlink, http://www.gao.gov/products/GAO-03-584G].
[19] OMB, Improving Agency Performance Using Information and
Information Technology (Enterprise Architecture Assessment Framework v
3.0) (December 2008).
[20] 40 U.S.C. § 11302(c)(1). The Clinger-Cohen Act expanded the
responsibilities of OMB and the agencies that had been established
under the Paperwork Reduction Act of 1995 with regard to IT
management. See 44 U.S.C. 3504(a)(1)(B)(vi) (OMB); 44 U.S.C.
3506(h)(5) (agencies).
[21] We have made recommendations to improve OMB's process for
monitoring high-risk IT investments; see GAO, Information Technology:
OMB Can Make More Effective Use of Its Investment Reviews, [hyperlink,
http://www.gao.gov/products/GAO-05-276] (Washington, D.C.: Apr. 15,
2005).
[22] This policy is set forth and guidance is provided in OMB Circular
No. A-11, Part 7, Sec. 300, et seq. (July 2010), and in the Supplement
to Part 7, Capital Programming Guide (June 2006), which directs
agencies to develop, implement, and use a capital programming process
to build their capital asset portfolios.
[23] GAO, Cost Estimating and Assessment Guide: Best Practices for
Developing and Managing Capital Program Costs, [hyperlink,
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March
2009); GAO-04-394G; [hyperlink, http://www.gao.gov/products/GAO-03-
584G]; and Assessing Risks and Returns: A Guide for Evaluating Federal
Agencies' IT Investment Decision-making, [hyperlink,
http://www.gao.gov/products/GAO/AIMD-10.1.13] (Washington, D.C.:
February 1997).
[24] J. A. Zachman, "A Framework for Information Systems
Architecture," IBM Systems Journal 26, no. 3 (1987).
[25] DOD, Department of Defense Architecture Framework, version 2.0,
Volumes I-III (May 2009).
[26] GAO, Federal Aviation Administration: Stronger Architecture
Program Needed to Guide Systems Modernization Efforts, [hyperlink,
http://www.gao.gov/products/GAO-05-266] (Washington, D.C.: Apr. 29,
2005); Homeland Security: Efforts Under Way to Develop Enterprise
Architecture, but Much Work Remains, [hyperlink,
http://www.gao.gov/products/GAO-04-777] (Washington, D.C.: Aug. 6,
2004); [hyperlink, http://www.gao.gov/products/GAO-04-731R];
Information Technology: Architecture Needed to Guide NASA's Financial
Management Modernization, [hyperlink,
http://www.gao.gov/products/GAO-04-43] (Washington, D.C.: Nov. 21,
2003); [hyperlink, http://www.gao.gov/products/GAO-03-1018];
[hyperlink, http://www.gao.gov/products/GAO-03-877R]; Information
Technology: DLA Should Strengthen Business Systems Modernization
Architecture and Investment Activities, [hyperlink,
http://www.gao.gov/products/GAO-01-631] (Washington, D.C.: June 29,
2001); and Information Technology: INS Needs to Better Manage the
Development of Its Enterprise Architecture, [hyperlink,
http://www.gao.gov/products/GAO/AIMD-00-212] (Washington, D.C.: Aug.
1, 2000).
[27] [hyperlink, http://www.gao.gov/products/GAO-10-846G].
[28] OMB, Improving Agency Performance Using Information and
Information Technology (Enterprise Architecture Assessment Framework
v3.1), (Washington, D.C.: June 2009); Federal Segment Architecture
Working Group and OMB, Federal Segment Architecture Methodology,
version 1.0 (Washington, D.C.: December 2008); and OMB, Federal
Enterprise Architecture Practice Guidance (Washington, D.C.: November
2007).
[29] [hyperlink, http://www.gao.gov/products/GAO-04-394G]; [hyperlink,
http://www.gao.gov/products/GAO/AIMD-10.1.13]; GAO, Executive Guide:
Improving Mission Performance Through Strategic Information Management
and Technology, GAO/AIMD-94-115 (Washington, D.C.: May 1994); and OMB,
Evaluating Information Technology Investments, A Practical Guide
(Washington, D.C.: November 1995).
[30] [hyperlink, http://www.gao.gov/products/GAO-04-394G].
[31] GAO, Information Technology: HUD Needs to Better Define
Commitments and Disclose Risks for Modernization Projects in Future
Expenditure Plans, [hyperlink, http://www.gao.gov/products/GAO-11-72]
(Washington, D.C.: Nov. 23, 2010); Information Technology: HUD Needs
to Strengthen Its Capacity to Manage and Modernize Its Environment,
[hyperlink, http://www.gao.gov/products/GAO-09-675] (Washington, D.C.:
July 31, 2009); Information Technology: FDA Needs to Establish Key
Plans and Processes for Guiding Systems Modernization Efforts,
[hyperlink, http://www.gao.gov/products/GAO-09-523] (Washington D.C.:
June 2, 2009); Information Technology: SSA Has Taken Key Steps for
Managing Its Investments, but Needs to Strengthen Oversight and Fully
Define Policies and Procedures, [hyperlink,
http://www.gao.gov/products/GAO-08-1020] (Washington, D.C.: Sept. 12,
2008); Information Technology: Treasury Needs to Strengthen Its
Investment Board Operations and Oversight, [hyperlink,
http://www.gao.gov/products/GAO-07-865] (Washington, D.C.: July 23,
2007); Information Technology: DHS Needs to Fully Define and Implement
Policies and Procedures for Effectively Managing Investments,
[hyperlink, http://www.gao.gov/products/GAO-07-424] (Washington, D.C.:
Apr. 27, 2007); Information Technology: Centers for Medicare &
Medicaid Services Needs to Establish Critical Investment Management
Capabilities, [hyperlink, http://www.gao.gov/products/GAO-06-12]
(Washington, D.C.: Oct. 28, 2005); Information Technology: HHS Has
Several Investment Management Capabilities in Place, but Needs to
Address Key Weaknesses, [hyperlink,
http://www.gao.gov/products/GAO-06-11] (Washington, D.C.: Oct. 28,
2005).
[32] 40 U.S.C. §§ 11311-11313.
[33] Pub. L. No. 110-181, § 904, 122 Stat. 3, 273 (Jan. 28, 2008).
[34] According to DOD, the business mission area is responsible for
ensuring that capabilities, resources, and materiel are reliably
delivered to the warfighter. Specifically, the business mission area
addresses areas, such as real property and human resources management.
[35] The act (10 U.S.C. § 2222(a), as amended), requires designated
approval authorities to certify that a defense business system
modernization (1) has been determined by the appropriate Chief
Management Officer to (a) be in compliance with the enterprise
architecture and (b) have undertaken appropriate business process re-
engineering efforts; (2) is necessary to achieve a critical national
security capability or address a critical requirement in an area such
as safety or security; or (3) is necessary to prevent a significant
adverse effect on a project that is needed to achieve an essential
capability, taking into consideration the alternative solutions for
preventing such an adverse effect.
[36] These investment review boards are for Financial Management,
Weapon Systems Lifecycle Management and Materiel Supply and Services
Management, Real Property and Installations Lifecycle Management, and
Human Resources Management. In August 2009, DOD's Enterprise Guidance
Board was chartered as the DOD CIO's Investment Review Board.
[37] Pub. L. No. 110-181, § 904(b), 122 Stat. 3, 274.
[38] Pub. L. No. 110-417, § 908, 122 Stat. 4356, 4569 (Oct. 14, 2008).
[39] GAO, Defense Business Transformation: DOD Needs to Take
Additional Actions to Further Define Key Management Roles, Develop
Measurable Goals, and Align Planning Efforts, [hyperlink,
http://www.gao.gov/products/GAO-11-181R] (Washington, D.C.: Jan. 26,
2011).
[40] The Item Unique Identification Registry is a relational database
that is intended to store acquisition and logistics information to
track, catalog, and inventory items, such as equipment and spare
parts, via machine-readable item identifiers.
[41] Pub. L. No. 108-375, § 332 (10 U.S.C. § 2222).
[42] The act (10 U.S.C. § 2222(a), as amended), requires designated
approval authorities to certify that a defense business system
modernization (1) has been determined by the appropriate Chief
Management Officer to (a) be in compliance with the enterprise
architecture and (b) have undertaken appropriate business process re-
engineering efforts; (2) is necessary to achieve a critical national
security capability or address a critical requirement in an area such
as safety or security; or (3) is necessary to prevent a significant
adverse effect on a project that is needed to achieve an essential
capability, taking into consideration the alternative solutions for
preventing such an adverse effect.
[43] GAO, DOD Business Systems Modernization: Progress in Establishing
Corporate Management Controls Needs to Be Replicated Within Military
Departments, [hyperlink, http://www.gao.gov/products/GAO-08-705]
(Washington, D.C.: May 15, 2008); DOD Business Systems Modernization:
Progress Continues to Be Made in Establishing Corporate Management
Controls, but Further Steps Are Needed, [hyperlink,
http://www.gao.gov/products/GAO-07-733] (Washington, D.C.: May 14,
2007); Business Systems Modernization: DOD Continues to Improve
Institutional Approach, but Further Steps Needed, [hyperlink,
http://www.gao.gov/products/GAO-06-658] (Washington, D.C.: May 15,
2006); and DOD Business Systems Modernization: Important Progress Made
in Establishing Foundational Architecture Products and Investment
Management Practices, but Much Work Remains, [hyperlink,
http://www.gao.gov/products/GAO-06-219] (Washington, D.C.: Nov. 23,
2005).
[44] [hyperlink, http://www.gao.gov/products/GAO-08-705].
[45] [hyperlink, http://www.gao.gov/products/GAO-09-586].
[46] GAO, Business Systems Modernization: Scope and Content of DOD's
Congressional Report and Executive Oversight of Investments Need to
Improve, [hyperlink, http://www.gao.gov/products/GAO-10-663]
(Washington, D.C.; May 24, 2010).
[47] GAO, DOD Business Systems Modernization: Progress in Establishing
Corporate Management Controls Needs to Be Replicated Within Military
Departments, [hyperlink, http://www.gao.gov/products/GAO-08-705]
(Washington, D.C.: May 15, 2008).
[48] [hyperlink, http://www.gao.gov/products/GAO-07-733].
[49] GAO, DOD Business Systems Modernization: Military Departments
Need to Strengthen Management of Enterprise Architecture Programs,
[hyperlink, http://www.gao.gov/products/GAO-08-519] (Washington, D.C.:
May 12, 2008).
[50] GAO, Information Technology: A Framework for Assessing and
Improving Enterprise Architecture Management (Version 1.1),
[hyperlink, http://www.gao.gov/products/GAO-03-584G] (Washington,
D.C.: April 2003).
[51] [hyperlink, http://www.gao.gov/products/GAO-03-584G] and
[hyperlink, http://www.gao.gov/products/GAO-10-846G].
[52] [hyperlink, http://www.gao.gov/products/GAO-10-846G].
[53] Pub. L. No. 110-417, § 908(b)(2), 122 Stat. 4356, 4569 (Oct. 14,
2008).
[54] The Agile Combat Support Architecture is planned to include all
combat services/support activities and generating force activities of
the Air Force, as well as the Air Force business enterprise
architecture.
[55] The Commodity Management Service provides functions for creating,
configuring, and calculating the data required for watch boards for
fuel and ammunition data and empowers the Global Combat Support System-
Joint Logistics Management application system.
[56] The LandWarNet/Battle Command Directorate is the primary Army
Staff organization chartered to validate, prioritize, and synchronize
Army network requirements across the Army. It is responsible for
monitoring the activities and outputs of the various Army agencies
that support the development and delivery of the network in order to
ensure network and battle command requirements meet Army operational
objectives and priorities.
[57] Procure-to-Pay encompasses all business functions necessary to
obtain goods and services.
[58] Service Request-to-Resolution is the process of performing
maintenance on materiel/assets requiring repair or rebuild including
the manufacture of parts, modifications, testing, and reclamation.
[59] Of the approximately 2,386 unique investments in DOD's Select and
Native Programming Data Input System--Information Technology, 749 are
categorized as either national security systems (i.e., intelligence
systems, cryptologic activities related to national security, military
command and control systems, and equipment that is an integral part of
a weapon or weapons system or is critical to the direct fulfillment of
military or intelligence missions or systems that store, process, or
communicate classified information) or are not within the business
mission area (e.g., warfighting mission area).
[60] DOD's budget submission includes funding totals for past,
current, and future years. Of the 1,637 business system investments
included in the fiscal year 2012 budget submission, 1,440 have
requested funding for fiscal year 2012. Further, 272 systems have
requested funding for development modernization. The remaining systems
have requested funding for current services. A given system could have
funding for current services as well as development modernization.
[61] The certification status "decertification close-out" identifies a
modernization effort that is complete and that does not require any
additional funds.
[62] The certification status "withdrawn" identifies a modernization
that is no longer requesting funding for a previously acted-on
modernization.
[63] [hyperlink, http://www.gao.gov/products/GAO-09-586].
[64] The investments we reviewed included DOD enterprise-level and
military department-level systems: specifically, DOD's Defense Travel
System, Air Force's Project Management Resource Tool, Army's Logistics
Modernization Program, and Navy's Enterprise Resource Planning system.
Details on our methodology for selecting these investments are
described in appendix I.
[65] GAO, Information Technology Investment Management: A Framework
for Assessing and Improving Process Maturity, [hyperlink,
http://www.gao.gov/products/GAO-04-394G] (Washington, D.C.: March
2004).
[66] Mission Area Leads are responsible for managing the IT portfolio
for their respective mission area (i.e., Business Mission Area,
Intelligence Mission Area, Enterprise Information Environment Mission
Area), including developing portfolio guidance, outcome measures, and
overseeing the mission area IT portfolio of investments.
[67] Functional Area Managers are to develop and manage functional
area (i.e., Human Resource Management, Real Property and Installation
Lifecycle Management, Financial Management) IT portfolios. These
managers are also responsible for, among other things, recommending,
reviewing, and overseeing functional area IT investments.
[68] GAO, Business Systems Modernization: DOD Needs to Fully Define
Policies and Procedures for Institutionally Managing Investments,
[hyperlink, http://www.gao.gov/products/GAO-07-538] (Washington, D.C.:
May 11, 2007); Business Systems Modernization: Air Force Needs to
Fully Define Policies and Procedures for Institutionally Managing
Investments, [hyperlink, http://www.gao.gov/products/GAO-08-52]
(Washington D.C.: Oct. 31, 2007); Business Systems Modernization:
Department of the Navy Needs to Establish Management Structure and
Fully Define Policies and Procedures for Institutionally Managing
Investments, [hyperlink, http://www.gao.gov/products/GAO-08-53]
(Washington, D.C.: Oct. 31, 2007).
[69] GAO, Opportunities to Reduce Potential Duplication in Government
Programs, Save Tax Dollars, and Enhance Revenue, [hyperlink,
http://www.gao.gov/products/GAO-11-318SP] (Washington, D.C.: Mar. 1,
2011).
[70] DOD, IT Defense Business Systems Investment Review Process
Guidance (January 2009).
[71] GAO, DOD Business Systems Modernization: Key Navy Programs'
Compliance with DOD's Federated Business Enterprise Architecture Needs
to Be Adequately Demonstrated, [hyperlink,
http://www.gao.gov/products/GAO-08-972] (Washington, D.C.: Aug. 7,
2008).
[72] [hyperlink, http://www.gao.gov/products/GAO-08-972].
[73] [hyperlink, http://www.gao.gov/products/GAO-08-972].
[74] This tool is used to filter BEA segments in an organized manner
to facilitate system compliance.
[75] [hyperlink, http://www.gao.gov/products/GAO-08-972].
[76] [hyperlink, http://www.gao.gov/products/GAO-08-972].
[77] [hyperlink, http://www.gao.gov/products/GAO-08-972].
[78] GAO, DOD Business Systems Modernization: Important Management
Controls Being Implemented on Major Navy Program, but Improvements
Needed in Key Areas, [hyperlink,
http://www.gao.gov/products/GAO-08-896] (Washington, D.C.: Sept. 8,
2008).
[79] Air Force officials reported that DOD plans to perform an annual
review on PMRT in September 2011.
[80] GAO, Business Systems Modernization: DOD Needs to Fully Define
Policies and Procedures for Institutionally Managing Investments,
[hyperlink, http://www.gao.gov/products/GAO-07-538] (Washington, D.C.:
May 11, 2007).
[81] Section 332 of the Fiscal Year 2005 NDAA (10 U.S.C. § 2222(a), as
amended).
[82] The approval authorities, as discussed earlier in this report,
include the Under Secretary of Defense for Acquisition, Technology,
and Logistics; the Under Secretary of Defense (Comptroller); the Under
Secretary of Defense for Personnel and Readiness; the ASD(NII)/DOD
CIO; and the Deputy Secretary of Defense. They are responsible for the
review, approval, and oversight of business systems and must establish
investment review processes for systems under their cognizance.
[83] The act (10 U.S.C. § 2222(a), as amended), requires designated
approval authorities to certify that a defense business system
modernization (1) has been determined by the appropriate Chief
Management Officer to (a) be in compliance with the enterprise
architecture and (b) have undertaken appropriate business process re-
engineering efforts; (2) is necessary to achieve a critical national
security capability or address a critical requirement in an area such
as safety or security; or (3) is necessary to prevent a significant
adverse effect on a project that is needed to achieve an essential
capability, taking into consideration the alternative solutions for
preventing such an adverse effect.
[84] [hyperlink, http://www.gao.gov/products/GAO-10-663].
[85] Ronald W. Reagan National Defense Authorization Act for Fiscal
Year 2005, Pub. L. No. 108-375, § 332, 118 Stat. 1811, 1851-1856 (Oct.
28, 2004), as amended (codified in part at 10 U.S.C. § 2222).
[86] GAO, DOD Business Systems Modernization: Military Departments
Need to Strengthen Management of Enterprise Architectures, [hyperlink,
http://www.gao.gov/products/GAO-08-519] (Washington, D.C.: May 12,
2008).
[87] GAO, Information Technology: A Framework for Assessing and
Improving Enterprise Architecture Management (Version 1.1),
[hyperlink, http://www.gao.gov/products/GAO-03-584G] (Washington,
D.C.: April 2003); and Organizational Transformation: A Framework for
Assessing and Improving Enterprise Architecture Management (Version
2.0), [hyperlink, http://www.gao.gov/products/GAO-10-846G]
(Washington, D.C.: August 2010).
[88] [hyperlink, http://www.gao.gov/products/GAO-08-519].
[89] GAO, DOD Business Systems Modernization: Recent Slowdown in
Institutionalizing Key Management Controls Needs to Be Addressed,
[hyperlink, http://www.gao.gov/products/GAO-09-586] (Washington, D.C.:
May 18, 2009).
[90] GAO, Information Technology Investment Management: A Framework
for Assessing and Improving Process Maturity, [hyperlink,
http://www.gao.gov/products/GAO-04-394G] (Washington, D.C.: March
2004).
[91] The DOD-level investment selected was the Defense Travel System.
The military-level investments were: Project Management Resource Tool
(Air Force), Logistics Modernization Program (Army), and Navy
Enterprise Resource Planning system (Navy).
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
