Business Systems Modernization
Scope and Content of DOD's Congressional Report and Executive Oversight of Investments Need to Improve Gao ID: GAO-10-663 May 24, 2010Since 1995, GAO has designated the Department of Defense's (DOD) multibillion dollar business systems modernization program as high risk, and it continues to do so today. To assist in addressing DOD's modernization challenges, the Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005 (the act) requires the department to, among other things, report specific information about business system investments, including (1) milestones and actual performance against specified measures and any revisions and (2) actions taken to certify that a modernization investment involving more than $1 million meets defined conditions before obligating funds. The act also directs GAO to review each report. As agreed, GAO focused on the fiscal year 2010 report's compliance with, among other things, these provisions of the act. To do so, GAO compared DOD's report to the act's reporting requirements, interviewed DOD officials, analyzed relevant documentation, and leveraged prior GAO reports.
DOD's fiscal year 2010 report to Congress on its business systems modernization program complies with key provisions in the act, but its scope and content are nevertheless limited. Specifically, (1) The report includes milestones, performance against milestones, and milestone revisions for specific investments. However, other important performance measures, such as measures of progress against program cost, capability, and benefit commitments are not included in the report. DOD officials attributed the missing performance-related information to various factors, including that most of the investments addressed in the report have not progressed far enough in their life cycles to measure cost, capability, and benefit performance. However, the report also cites a number of investments that have produced business improvements and cost savings, and thus it follows that performance-related information about investment costs incurred, capabilities delivered, and benefits realized is available and can be reported relative to program expectations. Moreover, programs that have not yet delivered capabilities or realized benefits have nevertheless incurred costs, which DOD can report relative to expected costs. (2) The report identifies certification actions associated with 116 business system modernization investments. However, the report omits certification actions for 40 other investments. According to DOD officials, the omitted actions are not new certifications, but rather are recertifications that were intentionally excluded from the report. However, certification memoranda show this is not the case for four of the actions and DOD guidance defines a recertification as a type of certification action. Further, the underlying bases for a number of reported actions are limited because program weaknesses that GAO's prior work has raised, such as the reliability of the systems' economic analyses and the sufficiency of business enterprise architecture compliance determinations, are not reflected in the reported certification actions. DOD's guidance does not require that certification submissions disclose program weaknesses that GAO has raised, and DOD officials stated that reviews are limited to the information that is submitted. As a result, DOD's annual report does not provide the full range of information that is needed to permit meaningful and informed congressional oversight of the department's business systems modernization efforts. Moreover, the bases for some certification actions exclude relevant information about known investment weaknesses, and thus these actions may not be sufficiently justified.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Randolph C. Hite Team: Government Accountability Office: Information Technology Phone: (202) 512-6256GAO-10-663, Business Systems Modernization: Scope and Content of DOD's Congressional Report and Executive Oversight of Investments Need to Improve
This is the accessible text file for GAO report number GAO-10-663
entitled 'Business Systems Modernization: Scope and Content of DOD's
Congressional Report and Executive Oversight of Investments Need to
Improve' which was released on May 24, 2010.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
United States Government Accountability Office:
GAO:
May 2010:
Business Systems Modernization:
Scope and Content of DOD's Congressional Report and Executive
Oversight of Investments Need to Improve:
GAO-10-663:
GAO Highlights:
Highlights of GAO-10-663, a report to congressional committees.
Why GAO Did This Study:
Since 1995, GAO has designated the Department of Defense‘s (DOD)
multibillion dollar business systems modernization program as high
risk, and it continues to do so today. To assist in addressing DOD‘s
modernization challenges, the Ronald W. Reagan National Defense
Authorization Act for Fiscal Year 2005 (the act) requires the
department to, among other things, report specific information about
business system investments, including (1) milestones and actual
performance against specified measures and any revisions and (2)
actions taken to certify that a modernization investment involving
more than $1 million meets defined conditions before obligating funds.
The act also directs GAO to review each report. As agreed, GAO focused
on the fiscal year 2010 report‘s compliance with, among other things,
these provisions of the act. To do so, GAO compared DOD‘s report to
the act‘s reporting requirements, interviewed DOD officials, analyzed
relevant documentation, and leveraged prior GAO reports.
What GAO Found:
DOD‘s fiscal year 2010 report to Congress on its business systems
modernization program complies with key provisions in the act, but its
scope and content are nevertheless limited. Specifically,
* The report includes milestones, performance against milestones, and
milestone revisions for specific investments. However, other important
performance measures, such as measures of progress against program
cost, capability, and benefit commitments are not included in the
report. DOD officials attributed the missing performance-related
information to various factors, including that most of the investments
addressed in the report have not progressed far enough in their life
cycles to measure cost, capability, and benefit performance. However,
the report also cites a number of investments that have produced
business improvements and cost savings, and thus it follows that
performance-related information about investment costs incurred,
capabilities delivered, and benefits realized is available and can be
reported relative to program expectations. Moreover, programs that
have not yet delivered capabilities or realized benefits have
nevertheless incurred costs, which DOD can report relative to expected
costs.
* The report identifies certification actions associated with 116
business system modernization investments. However, the report omits
certification actions for 40 other investments. According to DOD
officials, the omitted actions are not new certifications, but rather
are recertifications that were intentionally excluded from the report.
However, certification memoranda show this is not the case for four of
the actions and DOD guidance defines a recertification as a type of
certification action. Further, the underlying bases for a number of
reported actions are limited because program weaknesses that GAO‘s
prior work has raised, such as the reliability of the systems‘
economic analyses and the sufficiency of business enterprise
architecture compliance determinations, are not reflected in the
reported certification actions. DOD‘s guidance does not require that
certification submissions disclose program weaknesses that GAO has
raised, and DOD officials stated that reviews are limited to the
information that is submitted.
As a result, DOD‘s annual report does not provide the full range of
information that is needed to permit meaningful and informed
congressional oversight of the department‘s business systems
modernization efforts. Moreover, the bases for some certification
actions exclude relevant information about known investment
weaknesses, and thus these actions may not be sufficiently justified.
What GAO Recommends:
GAO is recommending that future annual reports include additional
information about investment performance measures and certification
actions and that DOD guidance be revised to ensure that certification
submissions disclose unresolved GAO findings and recommendations. DOD
agreed with GAO‘s recommendations.
View [hyperlink, http://www.gao.gov/products/GAO-10-663] or key
components. For more information, contact Randolph C. Hite at (202)
512-3439 or hiter@gao.gov.
[End of section]
Contents:
Letter:
Background:
DOD's Annual Report Discusses Programs' Performance Against
Milestones, but Does Not Address Performance Against Other Program
Commitments:
DOD's Report Discusses Business Systems Modernization Programs'
Improvements in Business Operations and Cost Savings:
DOD's Annual Report Does Not Describe Certification Actions for All
Systems, and Justification of Reported Certification Actions Is
Limited:
DOD's Report States That Certification Waivers Were Not Granted:
Conclusions:
Recommendations for Executive Action:
Agency Comments:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Comments from the Department of Defense:
Appendix III: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: DOD Business Systems Modernization Governance Entities'
Roles, Responsibilities, and Composition:
Table 2: DOD Investment Tiers:
Table 3: Summary of Systems Certified, Recertified, and Decertified:
Figures:
Figure 1: Simplified View of DOD Organizational Structure:
Figure 2: Key Milestone Status:
Abbreviations:
ASD(NII)/DOD CIO: Assistant Secretary of Defense (Networks and
Information Integration)/Department of Defense Chief Information
Officer:
BEA: business enterprise architecture:
BTA: Business Transformation Agency:
CIO: chief information officer:
CMO: chief management officer:
DBSMC: Defense Business Systems Management Committee:
DCMO: Deputy Chief Management Officer:
DITPR: Defense Information Technology Portfolio Repository:
DOD: Department of Defense:
DRRS: Defense Readiness Reporting System:
ERP: enterprise resource planning:
ETP: enterprise transition plan:
GCSS-MC: Global Combat Support System-Marine Corps:
IRB: Investment Review Board:
IT: information technology:
NDAA: National Defense Authorization Act:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
May 24, 2010:
Congressional Committees:
For decades, the Department of Defense (DOD) has been challenged in
modernizing its timeworn business systems.[Footnote 1] In 1995, we
designated DOD's Business Systems Modernization Program as high risk,
and we continue to designate it as such today.[Footnote 2] In light of
the department's business systems modernization management challenges,
Congress included provisions in the Ronald W. Reagan National Defense
Authorization Act for Fiscal Year 2005[Footnote 3] (the act) for the
department to report annually on its modernization progress. More
specifically, the act requires the department's annual report to,
among other things, (1) describe milestones and actual performance
against specified measures and any revisions, (2) discuss specific
improvements in business operations and cost savings resulting from
successful business system modernization efforts, (3) describe
specific actions on each business system modernization submitted for
certification,[Footnote 4] and (4) identify any business system
modernization with an obligation in excess of $1 million that was not
certified during the preceding fiscal year and reasons for the waiver.
Additionally, the act directs us to submit to the cognizant
congressional committees--within 60 days of DOD's report submission--
an assessment of the department's actions to comply with these
requirements.
As agreed with your offices, the objective of our review was to assess
the actions taken by DOD to comply with key aspects of section 332 of
the act.[Footnote 5] To accomplish this objective, we focused on the
extent to which DOD's annual report addressed the four requirements
described above. In doing so, we compared the nature and scope of the
information contained in the department's report to Congress, which
was submitted on March 25, 2010, as well as the supporting information
that DOD used in preparing the report, to four requirements in the
act. In addition, we leveraged our recent reports related to DOD's
business system investments to evaluate the content of key aspects of
the report.[Footnote 6] We also met with cognizant DOD officials to
discuss actions taken or planned.
We conducted this performance audit at DOD offices in Arlington,
Virginia, from January to May 2010, in accordance with generally
accepted government auditing standards. Those standards require that
we plan and perform the audit to obtain sufficient, appropriate
evidence to provide a reasonable basis for our findings and
conclusions based on our audit objectives. We believe that the
evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives. Details on our objective,
scope, and methodology are contained in appendix I.
Background:
DOD is a massive and complex organization entrusted with more taxpayer
dollars than any other federal department or agency.[Footnote 7]
Organizationally, the department includes the Office of the Secretary
of Defense, the Joint Chiefs of Staff, the military departments,
numerous defense agencies and field activities, and various unified
combatant commands that are responsible for either specific geographic
regions or specific functions. (See figure 1 for a simplified
depiction of DOD's organizational structure.)
Figure 1: Simplified View of DOD Organizational Structure:
[Refer PDF for image: organizational structure]
Top level:
Secretary of Defense;
* Deputy Secretary of Defense[A].
Second level, reporting to Secretary of Defense:
* Department of the Army;
* Department of the Navy;
* Department of the Air Force;
* Office of the Secretary of Defense;
- DOD field activities;
- Defense agencies;
* Inspector General;
* Joint Chiefs of Staff
- Combatant commands[B].
Source: GAO based on DOD documentation.
[A] The Deputy Secretary of Defense serves as the Chief Management
Officer, who is intended to provide focused and sustained leadership
over DOD's business transformation efforts.
[B] The Chairman of the Joint Chiefs of Staff serves as the spokesman
for the commanders of the combatant commands, especially on the
administrative requirements of the commands.
[End of figure]
In support of its military operations, the department performs an
assortment of interrelated and interdependent business functions,
including logistics management, procurement, health care management,
and financial management. As we have previously reported, the DOD
systems environment that supports these business functions is overly
complex and error prone, and is characterized by (1) little
standardization across the department, (2) multiple systems performing
the same tasks, (3) the same data stored in multiple systems, and (4)
the need for data to be entered manually into multiple systems.
[Footnote 8] Moreover, the department's nonintegrated and duplicative
systems impair its ability to combat fraud, waste, and abuse.[Footnote
9] The department recently reported that this systems environment is
composed of approximately 2,480 separate business systems. For fiscal
year 2010, DOD requested about $15.5 billion in funds to operate,
maintain, and modernize its business systems and associated
information technology (IT) infrastructure.
DOD currently bears responsibility, in whole or in part, for 15 of the
31 programs across the federal government that we have designated as
high risk because they are highly susceptible to fraud, waste, abuse,
and mismanagement.[Footnote 10] Eight of these areas are specific to
the department,[Footnote 11] and 7 other high-risk areas are shared
with other federal agencies.[Footnote 12] Collectively, these high-
risk areas relate to DOD's major business operations that are
inextricably linked to the department's ability to perform its overall
mission, directly affect the readiness and capabilities of U.S.
military forces, and can affect the success of a mission. DOD's
business systems modernization is one of the high-risk areas, and it
is an essential enabler to addressing many of the department's other
high-risk areas. For example, modernized business systems are integral
to the department's efforts to address its financial, supply chain,
and information security management high-risk areas.
DOD's Institutional Approach to Business Systems Modernization:
The National Defense Authorization Act (NDAA) for Fiscal Year 2008
designates the Deputy Secretary of Defense as the Chief Management
Officer (CMO) for DOD and created a Deputy CMO position.[Footnote 13]
The CMO's responsibilities include developing and maintaining a
departmentwide strategic plan for business reform and establishing
performance goals and measures for improving and evaluating overall
economy, efficiency, and effectiveness and monitoring and measuring
the progress of the department. The Deputy CMO's responsibilities
include recommending to the CMO methodologies and measurement criteria
to better synchronize, integrate, and coordinate the business
operations to ensure alignment in support of the warfighting mission.
The Business Transformation Agency (BTA) supports the Deputy CMO in
leading and coordinating business transformation efforts across the
department. This includes maintaining and updating the department's
enterprise architecture[Footnote 14] for its business mission area.
[Footnote 15]
The CMO and Deputy CMO are to interact with several entities to guide
the direction, oversight, and execution of DOD's business
transformation efforts, which include business systems modernization.
These entities include the Defense Business Systems Management
Committee (DBSMC), which serves as the highest-ranking investment
review and decision-making body for business systems modernization
activities and is chaired by the Deputy Secretary of Defense; the
principal staff assistants, who serve as the certification[Footnote
16] authorities for business system modernizations in their respective
core business missions; the investment review boards (IRB),[Footnote
17] which are chaired by the certifying authorities and form the
review and decision-making bodies for business system investments in
their respective areas of responsibility; and the BTA, which supports
IRBs and leads and coordinates business transformation efforts across
the department. (Table 1 lists these entities and provides greater
detail on their roles, responsibilities, and composition.)
Table 1: DOD Business Systems Modernization Governance Entities'
Roles, Responsibilities, and Composition:
Entity: DBSMC;
Roles and responsibilities:
* Provides strategic direction and plans for the business mission area
in coordination with the warfighting and enterprise information
environment mission areas;
* Recommends policies and procedures required to integrate DOD
business transformation and attain cross-department, end-to-end
interoperability of business systems and processes;
* Serves as approving authority for business system modernizations
greater than $1 million;
* Establishes policies and approves the business mission area
strategic plan, the enterprise transition plan for implementation of
business systems modernization, the transformation program baseline,
and the business enterprise architecture (BEA);
Composition:
Chaired by the Deputy Secretary of Defense/CMO; the Vice Chair is the
Deputy Chief Management Officer (DCMO). Includes senior leadership in
the Office of the Secretary of Defense such as the Under Secretaries
of Defense for Acquisition, Technology, and Logistics; Comptroller;
Personnel and Readiness; and Assistant Secretary of Defense (Networks
and Information Integration)/Department of Defense Chief Information
Officer (ASD(NII)/DOD CIO). Also includes the Military Department
Chief Management Officers, the heads of select Defense Agencies, and
other senior participation by the Joint Chiefs of Staff and the U.S.
Transportation Command.
Entity: Principal Staff Assistants/Certification Authorities;
Roles and responsibilities:
* Support the DBSMC's management of enterprise business IT investments;
* Serve as the certification authorities accountable for the
obligation of funds for respective business system modernizations
within designated core business missions[A];
* Provide the DBSMC with recommendations for system investment
approval;
Composition:
Under Secretaries of Defense for Acquisition, Technology, and
Logistics; Comptroller; and Personnel and Readiness; ASD(NII)/DOD CIO;
and the Deputy Secretary of Defense.
Entity: IRBs;
Roles and responsibilities:
* Serve as the oversight and investment decision-making bodies for
those business capabilities that support activities under their
designated areas of responsibility;
* Recommend certification for all business systems modernization
investments costing more than $1 million that are integrated and
compliant with the BEA;
Composition:
Includes the principal staff assistants; Joint Staff; ASD(NII)/DOD CIO;
core business mission area representatives; military departments;
defense agencies; and combatant commands.
Entity: Component Precertification Authority;
Roles and responsibilities:
* Ensures component-level investment review processes integrate with
the investment management system;
* Identifies those component systems that require IRB certification
and prepare, review, approve, validate, and transfer investment
documentation as required;
* Assesses and precertifies architecture compliance of component
systems submitted for certification and annual review;
* Acts as the component's principal point of contact for communication
with the IRBs;
Composition:
Includes the chief management officers from the Air Force, Army, and
Navy and other designated representatives from other defense agencies.
Entity: BTA;
Roles and responsibilities:
* Operates under the authority of the DCMO;
* Maintains and updates the department's BEA and enterprise transition
plan;
* Ensures that functional priorities and requirements of various
defense components, such as the Army and the Defense Logistics Agency,
are reflected in the architecture;
* Ensures adoption of DOD-wide information and process standards as
defined in the architecture;
* Serves as the day-to-day management entity of the business
transformation effort at the DOD enterprise level;
* Provides support to IRBs;
Composition:
Composed of eight directorates (Chief of Staff, Defense Business
Systems Acquisition Executive, Enterprise Integration, Enterprise
Planning and Investment, Transformation Priorities and Requirements
Financial Management, Transformation Priorities and Requirements Human
Resource Management, Transformation Priorities and Requirements Supply
Chain Management, and Warfighter Requirements).
Source: GAO based on DOD documentation.
[A] DOD has five core business missions: Human Resources Management,
Weapon System Lifecycle Management, Materiel Supply and Service
Management, Real Property and Installations Lifecycle Management, and
Financial Management.
[End of table]
Overview of DOD's Tiered Accountability for Business Systems
Modernization:
Since 2005, DOD has employed a "tiered accountability" approach to
business systems modernization. Under this approach, responsibility
and accountability for business architectures and systems investment
management are assigned to different levels in the organization. For
example, the BTA is responsible for developing the corporate BEA
(i.e., the thin layer of DOD-wide policies, capabilities, standards,
and rules) and the associated enterprise transition plan (ETP). Each
component is responsible for defining a component-level architecture
and transition plan associated with its own tiers of responsibility
and for doing so in a manner that is aligned with (i.e., does not
violate) the corporate BEA. Similarly, program managers are
responsible for developing program-level architectures and plans and
for ensuring alignment with the architectures and transition plans
above them. This concept is to allow for autonomy while also ensuring
linkages and alignment from the program level through the component
level to the corporate level. (Table 2 describes the four investment
tiers and identifies the associated reviewing and approving entities.)
Table 2: DOD Investment Tiers:
Tier 1;
Tier description: Major automated information system[A] or major
defense acquisition program[B];
Reviewing/approving entities: IRB and DBSMC.
Tier 2;
Tier description: Exceeding $10 million in total development/
modernization costs, but not designated as a Major Automated
Information System or Major Defense Acquisition Program;
Reviewing/approving entities: IRB and DBSMC.
Tier 3;
Tier description: Exceeding $1 million and up to $10 million in total
development/modernization costs;
Reviewing/approving entities: IRB and DBSMC.
Tier 4;
Tier description: Investment funding required up to $1 million;
Reviewing/approving entities: Component-level review only (unless the
system or line of business it supports is designated as an interest
program by the IRB chair).
Source: GAO based on DOD documentation.
[A] A Major Automated Information System is a program or initiative
that is so designated by the ASD(NII)/DOD CIO or that is estimated to
require program costs in any single year in excess of $32 million,
total program costs in excess of $126 million, or total life cycle
costs in excess of $378 million in fiscal year 2000 constant dollars.
[B] A Major Defense Acquisition Program is an acquisition program that
is so designated or estimated by the Under Secretary of Defense for
Acquisition, Technology, and Logistics to require an eventual total
expenditure for research, development, and test and evaluation of more
than $365 million or, for procurement, of more than $2.190 billion in
fiscal year 2000 constant dollars.
[End of table]
Consistent with the tiered accountability approach, the NDAA for
Fiscal Year 2008 required the secretaries of the military departments
to designate the department under secretaries as CMOs with primary
responsibility for business operations.[Footnote 18] Moreover, the
Duncan Hunter NDAA for Fiscal Year 2009 required the military
departments to establish business transformation offices to assist
their CMOs.[Footnote 19]
Summary of Fiscal Year 2005 NDAA Requirements:
Congress included provisions in the NDAA for Fiscal Year 2005 that are
aimed at ensuring DOD's establishment and implementation of effective
investment management structures and processes.[Footnote 20] According
to the act, DOD is required to develop a BEA; develop an ETP for
implementing the architecture; identify each business system proposed
for funding in DOD's fiscal year budget submissions; delegate the
responsibility for business systems to designated approval authorities
within the Office of the Secretary of Defense; require each approval
authority to establish investment review structures and processes;
and, effective October 1, 2005, not obligate appropriated funds for a
defense business system modernization with a total cost of more than
$1 million unless the approval authority certifies that the business
system modernization meets several conditions. [Footnote 21]
The NDAA for Fiscal Year 2005 also requires that the Secretary of
Defense submit to congressional defense committees an annual report on
the department's compliance with the act's provisions. This report is
to:
1. describe actions taken and planned for meeting the act's
requirements, including:
a) specific milestones and actual performance against specified
performance measures and any revision of such milestones and
performance measures and:
b) specific actions on the defense business system modernizations
submitted for certification under such subsection;
2. discuss specific improvements in business operations and cost
savings resulting from successful defense business systems
modernization efforts;
3. identify the number of defense business system modernizations
certified; and:
4. identify any defense business system modernization with an
obligation in excess of $1 million during the preceding fiscal year
that was not certified as required, and the reasons for the waiver.
Summary of Recent GAO Reviews of DOD's Business Systems Modernization
and Business Transformation Efforts:
Between 2005 and 2008, we reported that DOD had taken increasing steps
to comply with key requirements of the NDAA for Fiscal Year 2005
relative to architecture development, transition plan development,
budgetary disclosure, and investment review and to satisfy relevant
systems modernization management guidance, but that much remained to
be accomplished relative to the act's requirements and relevant
guidance.[Footnote 22] Nevertheless, we concluded that the department
had made important progress in defining and beginning to implement
institutional management controls (i.e., processes, structures, and
tools).
Notwithstanding this progress, in May 2009, we reported that the pace
of DOD's efforts in defining and implementing key institutional
modernization management controls had slowed compared with progress
made in each of the last 4 years, leaving much to be accomplished to
fully implement the act's requirements and related guidance.[Footnote
23] For example:
* The corporate BEA had yet to be extended (i.e., federated) to the
entire family of business mission area architectures, including using
an independent verification and validation agent to assess the
components' subsidiary architectures and federation efforts.
* The fiscal year 2009 budget submission included some, but omitted
other key information about business system investments, in part
because of the lack of a reliable, comprehensive inventory of all
defense business systems.
* IT investment management policies and procedures at the corporate
and component levels were not fully defined.
* The business system information used to support the development of
the transition plan and DOD's budget requests, as well as
certification and annual reviews, was of questionable reliability.
* Business system investments costing more than $1 million continue to
be certified and approved, but these decisions were not always based
on complete information.
Accordingly, we reiterated existing recommendations to address each of
these areas and further recommended that DOD, among other things,
improve the quality of investment-related information. DOD partially
agreed with our recommendations and described actions being planned or
under way to address them. DOD is currently in the process of
addressing these recommendations.
DOD's Annual Report Discusses Programs' Performance Against
Milestones, but Does Not Address Performance Against Other Program
Commitments:
The act requires that DOD's report describe milestones for business
system modernization programs and actual performance against
performance measures. In addition, the act requires that the report
specify any revisions to milestones and performance measures.
To its credit, DOD's annual report includes milestones, performance
against milestones, and milestone revisions for 76 programs. However,
other important performances measures, which typically include
measures associated with determining progress against program cost,
capability, and benefit commitments, are not included in the report.
BTA officials cited various reasons for the scope and content of the
information provided and not provided, but these reasons are at odds
with other aspects of its report. Without including information on
program performance against, and revisions to, such key measures as
cost, capability, and benefit commitments, DOD is not providing
Congress with the information needed to inform its oversight of
business system modernization programs.
DOD's Report Includes Program Milestones as Well as Revisions to and
Performance Against These Milestones:
Consistent with the act's requirement that DOD report on specific
milestones and revisions to the milestones, DOD's March 2010 report
includes a summary of the status of milestones that were to be
completed during fiscal 2009, the revisions associated with these
milestones (e.g. delayed or deleted), and the reason for the revision.
[Footnote 24] Specifically, the report lists three categories of
milestones:
* Standard acquisition milestones: key events and dates[Footnote 25]
that are provided for under DOD's system acquisition process.[Footnote
26]
* BEA compliance milestones: time frames for addressing specific IRB
certification conditions related to ensuring BEA compliance.
* Interim milestones: key events and dates to supplement DOD's system
acquisition process milestones (e.g., implementing specific system
capabilities or upgrading infrastructure by a given date).
DOD's report includes a total of 224 milestones[Footnote 27] that
collectively span 76 programs.[Footnote 28] Of these 224 milestones,
35 are standard acquisition milestones, 22 are BEA compliance
milestones, and 167 are interim milestones. The report also discusses
performance against these milestones. Specifically, of the 224
milestones, 56 percent are reported to have been met, while 21 and 23
percent are reported to have been deleted (i.e., determined to be
unnecessary) or not met, respectively. (See fig 2.) With respect to
acquisition and compliance milestones, the percentage of milestones
that were reported as not being met was 66 and 50 percent,
respectively.
Figure 2: Key Milestone Status:
[Refer to PDF for image: stacked vertical bar graph]
Milestone category: Acquisition;
Total number of milestones: Delayed: 14;
Total number of milestones: Deleted: 9;
Total number of milestones: Met: 12;
Total number of milestones: 35.
Milestone category: Compliance;
Total number of milestones: Delayed: 5;
Total number of milestones: Deleted: 6;
Total number of milestones: Met: 11;
Total number of milestones: 22.
Milestone category: Interim;
Total number of milestones: Delayed: 33;
Total number of milestones: Deleted: 31;
Total number of milestones: Met: 103;
Total number of milestones: 167.
Milestone category: Total;
Total number of milestones: Delayed: 52;
Total number of milestones: Deleted: 46;
Total number of milestones: Met: 126;
Total number of milestones: 224.
Source: GAO analysis of DOD annual report.
[End of figure]
DOD's Report Does Not Address Performance Against Other Program
Commitments:
Beyond milestones, the act requires DOD's annual report to address
actual performance against performance measures and any revision of
these performance measures. As we have previously reported, meaningful
information about program performance is typically measured in terms
of program cost, capability, and benefit commitments, in addition to
schedule commitments. Through such a range of performance measures,
valuable insight into the health and success of a business system
investment can be gained.[Footnote 29]
As we previously discussed in this report, DOD's annual report does
report schedule commitments (i.e., milestones) for its modernizing
programs. While DOD's annual report also includes examples of business
improvements and costs savings, this program performance information
is not reported against performance measure baselines. Further, the
report remains silent with respect to other important performance
measures such as progress made against cost and capability
commitments, which would allow congressional decision makers to
understand the extent to which programs are meeting cost, capability,
and benefit commitments.
BTA officials stated they focused the annual report on programs that
had planned milestones during fiscal year 2009. Further, they said
they focused on program milestones because most of the investments
covered by the report have not progressed far enough in their life
cycles to measure cost, capability, and benefit performance. In
addition, the annual report states that DOD does not include
performance measures in its annual report for any system that has
reached initial or full operational capability and is no longer
modernizing. Nevertheless, the report includes descriptions of a
number of programs that have progressed to the point where DOD reports
on actual operational efficiencies and dollar savings that have
accrued, which, in turn, means that these programs have progressed to
the point that DOD can report on progress against defined performance
commitments, such as the costs that have been incurred, the
capabilities that have been delivered, and benefits that have been
realized. Moreover, the programs that have not yet delivered
capabilities or realized benefits have incurred costs, which DOD can
report relative to expected costs.
Establishing performance measures and monitoring actual-versus-
expected performance using the full range of measures are essential to
understanding the health of any IT investment. By not including
information on each program's performance against defined cost,
capability, and benefit commitments, DOD is not providing Congress
with important information for informing its oversight of business
system modernization investments.
DOD's Report Discusses Business Systems Modernization Programs'
Improvements in Business Operations and Cost Savings:
The act requires that DOD's annual report discuss specific
improvements in business operations and cost savings resulting from
successful business system modernization programs. DOD met this
requirement by including 18 "case in point" examples in the report.
Among other things, each narrative generally describes the program and
provides high-level information on system capabilities delivered and
benefits achieved to date. Specific examples include:
* The Air Force Recruiting Information Support System is to be the
primary Web-based recruiting system for the Air Force. According to
the annual report, the Air Force's legacy recruiting information
system has been slow and, at times, unavailable to users.
Modernization of the system (e.g., new hardware and software upgrades)
is reported to have improved recruiters' productivity by reducing the
wait time for processing recruiter requests. Other reported
improvements include allowing recruiters to build applicant files off
line and upload them at a later time and reducing the recruiter's
dependence on Internet connectivity. Future plans include merging the
Air Force Recruiting Information Support System with the Air Force
Reserve Recruiting System to increase functionality and decrease
system response time.
* The Army Learning Management System[Footnote 30] is a Web-based
system for training, scheduling, and career planning for soldiers.
According to the annual report, in 2009 the system contributed to an
88 percent increase in the number of student accounts, 111 percent
increase in the number of courses offered, and 157 percent increase in
the number of course completions.
* Wide Area Work Flow is a Web-based system to centralize and automate
DOD's largely manual business payment process. The annual report
states that the system has thus far allowed the cost of processing a
payment to decrease from between $22 and $30 to between $6 and $12.
Other cited benefits include allowing suppliers to have a single point
of interface with DOD for payment invoicing, receipt, and acceptance.
* The Defense Agencies Initiative is an enterprise resource planning
(ERP) system[Footnote 31] to standardize and integrate enterprise data
to support financial decisions in real time. According to the annual
report, the system resulted in a reduction in the time it takes to
post financial obligations from 60 days to less than 2 days, and a
reduction in the time to close out monthly financial reports from 4
days to less than 1 day. Further, the report states that financial
information is now available to BTA on a real-time basis and thereby
enables proactive management of agency finances.
DOD's Annual Report Does Not Describe Certification Actions for All
Systems, and Justification of Reported Certification Actions Is
Limited:
Among other things, the act requires DOD's annual report to describe
specific actions the department has taken on each business system
modernization investment submitted for certification. More
specifically, the act states that such investments involving more than
$1 million in obligations must be certified by a designated approval
authority[Footnote 32] as meeting specific criteria, such as
demonstrating compliance with DOD's BEA.[Footnote 33] Further, the act
requires the DBSMC to approve each of these certifications.
To its credit, DOD's annual report identifies IRB certification
actions associated with 116 business system investments. However,
certification actions associated with 40 other investments are not
included. Further, the bases for several of the fiscal year 2009
system certification actions and subsequent approvals are limited
because program weaknesses and issues that our prior work has raised
about, for example, the systems' economic analyses and BEA compliance
determinations, are not reflected in the reported certification
actions.
According to BTA officials, only new certifications are included in
the report, even though DOD guidance states that recertifications are
also certification actions. Moreover, this guidance does not require
programs to disclose program weaknesses and issues raised by us or
others.
By not fully identifying in its annual report the certification
actions taken on all business system modernization investments, DOD is
not fully informing congressional oversight. Further, by not ensuring
that all certifications reflect known program weaknesses, business
system modernization program certification and approval decisions are
not being fully informed and thus may not be adequately justified.
DOD Has Established an Approach to Certifying Business System
Investments:
DOD has established what it describes as a "tiered accountability"
approach to meeting the act's requirements for certifying business
system investments. Under this approach, investment review begins
within the military departments and defense agencies and advances
through a hierarchy of review and decision-making authorities,
depending on the size, nature, and significance of the investment. For
those investments that meet the act's dollar threshold, namely those
with planned modernizations in excess of $1 million, this sequence of
review and decision making includes component precertification, IRB
certification, and DBMSC approval.[Footnote 34] For those investments
that do not meet the dollar threshold, investment decision-making
authority remains with the responsible component.
According to the department's approach, reviews for modernization
investments of more than $1 million focus on program alignment with
the BEA; alignment with the department's strategic mission, goals, and
objectives; and oversight commensurate with the program's cost, scope,
and complexity. The approach further requires that these reviews be
completed before a component can obligate modernization funds.
At the component level, program managers are responsible for the
information about their respective programs in a central repository
known as the Defense IT Portfolio Repository system (DITPR).[Footnote
35] The component precertification authority is responsible for
precertifying that a given system investment is compliant with the
BEA, reviewing the system funding requests, and ensuring that the IRB
responsible for the investment receives complete, current, accurate,
and timely information. The precertification authority is also
responsible for "asserting" the status and validity of the investment
information by submitting a component precertification letter to the
responsible IRB.[Footnote 36]
At the corporate level, an IRB reviews the precertification letter and
related material and makes a recommendation for a specific
certification action for each of its investments. After the IRB makes
its recommendation, it prepares a certification memorandum that
documents the IRB's decisions and any related conditions. The
memorandum is forwarded to the DBSMC, which either approves or
disapproves the IRB's decision and issues a memorandum containing its
decision. If the DBSMC disapproves a system investment's
certification, it is up to the component's precertification authority
to decide whether or not to resubmit the investment after it has
resolved the reasons for the disapproval.
Under DOD's approach, there are four types of certification actions:
* Certify: An IRB certifies the modernization as fully meeting
criteria defined in the act and IRB investment review guidance, such
as compliance with the BEA and the extent to which the investment is
consistent with component and department IT investment portfolios,
which are asserted by the component precertification authority.
* Certify with conditions: An IRB certifies the modernization with the
understanding that it will address specific IRB-imposed conditions.
For example, the Army's Real Estate Management Information System was
certified with a condition to provide a plan for how data elements
would comply with certain business rules in DOD's BEA.
* Recertify: An IRB certifies the obligation of additional
modernization funds for a previously certified modernization
investment. For example, the Air Force's Cargo Movement Operations
System was recertified in April 2009 for $6.3 million to be spent in
fiscal years 2009 through 2012. This recertification was in addition
to the $21.1 million previously certified in fiscal year 2007. In
addition, a program must request IRB recertification if the program
plans to redistribute previously approved modernization funds among
multiple fiscal years and this redistribution will result in the
funding for any given fiscal year exceeding the previously approved
amount by 10 percent or more.
* Decertify: An IRB reduces the amount of modernization funds
available to an investment when the entire amount of funding is not to
be obligated as previously certified. For example, the Defense
Financial Accounting Service's Standard Disbursing Initiative had
about $5.5 million decertified because the funds were no longer
needed. An IRB may also decertify a modernization after it has been
completed. For example, DOD reported that $213,000 for the Air
National Guard's Reserve Order Writing System was decertified at the
time the system was completed because the funds were no longer needed.
DOD's Report Excludes a Number of Certification Actions Taken on
Business System Modernizations:
The act requires that DOD's annual report describe specific actions
taken for each business system investment submitted for certification.
However, the department's annual report discusses fiscal year 2009
certification actions on only 116 of the 156 systems on which
certification actions were taken. More specifically, the report states
that during fiscal year 2009, 92 business system modernizations were
certified--32 with and 60 without conditions. For the 32 systems, 58
conditions were collectively reported. Examples of conditions cited in
the report are the need for a system to comply with the Standard
Financial Information Structure[Footnote 37] and the need to develop a
plan for complying with the data standards of DOD's Item Unique
Identifier Registry.[Footnote 38] The report also identifies 24
decertifications. For example, the Air Force's Enhanced Technical
Information Management System had about $13.9 million in funding
decertified (i.e., reduced), and the Defense Financial Accounting
Service's Standard Disbursing Initiative had about $5.5 million
decertified.
However, fiscal year 2009 IRB and DBSMC decision memoranda and meeting
minutes show that 40 systems had additional certification actions that
were not included in DOD's annual report.[Footnote 39] Of these 40
systems, 2 were certified without conditions, 2 were certified with
conditions, and 36 were recertified. Collectively, DOD's annual report
omits about 26 percent of its certification actions. (See table 3 for
a summary of actual, reported, and unreported certification actions.)
Table 3: Summary of Systems Certified, Recertified, and Decertified:
Action: Certify without conditions;
Actual: 62;
Reported: 60;
Not reported: 2.
Action: Certify with conditions;
Actual: 34;
Reported: 32;
Not reported: 2.
Action: Recertify;
Actual: 36;
Reported: 0;
Not reported: 36.
Action: Decertify;
Actual: 24;
Reported: 24;
Not reported: 0.
Action: Total;
Actual: 156;
Reported: 116;
Not reported: 40.
Source: GAO analysis of DOD documentation.
[End of table]
According to BTA officials, the excluded certification actions are all
recertifications, which they said are intentionally not reported
because they are not new certifications. They also told us that the
four new certifications were, in fact, recertifications. However,
DBSMC and IRB memoranda and meeting minutes identify these four
certification actions as new certifications.[Footnote 40] Moreover,
DOD guidance defines a recertification as a type of certification
action, thus making it subject to the act's reporting requirements.
Without complete reporting of its certification actions, DOD is not in
full compliance with its guidance, and DOD is limiting congressional
visibility into the full scope of its business systems modernization
efforts.
IRB Certification Actions Do Not Reflect and Disclose Known
Limitations in Programs' BEA Alignment, Economic Justification, and
Management:
According to DOD guidance, IRB certification addresses, among other
things, a program's alignment with the BEA and its management relative
to factors such as system cost, scope, and complexity. To make a
certification decision, IRBs rely on documentation submitted by the
component precertification authority, including a certification
dashboard, which includes cost and schedule status information; an
economic viability analysis, which addresses the investment's cost and
benefits or cost effectiveness; and regulatory and standards
compliance determinations. DOD guidance also gives IRBs broad
authority in their certification reviews and actions, thus allowing
each board to review and consider whatever investment-related
information that it deems appropriate. Moreover, BTA and IRB officials
told us that an IRB is not limited in the conditions it can place on a
program.
The IRB certification actions described in DOD's latest annual report
are limited because they do not reflect significant limitations in the
department's basis for determining an investment's alignment to the
BEA. Specifically, we recently reported that key DOD BEA compliance
assessments did not include all relevant architecture products, such
as products that specify the technical standards needed to promote
interoperability among related systems or examine overlaps with other
business systems. In addition, we reported that these compliance
assessments were not validated by DOD certification and approval
entities.[Footnote 41] Despite these limitations, business systems
modernization programs were certified as compliant with the BEA even
though they did not adequately demonstrate such compliance.
Accordingly, we recommended that DOD revise its BEA compliance
guidance, tool, and IRB verification requirements to address these
shortfalls. To date, DOD has yet to implement these recommendations,
and thus the compliance determination weaknesses remain. Despite this,
DOD's latest annual report does not disclose these limitations on any
of the 116 investment certification actions that it describes.
In addition, the fiscal year 2009 IRB certification actions described
in the latest annual report are further limited in that they do not
reflect weaknesses we have recently reported with the economic
justification for and management of certain programs.[Footnote 42] For
example,
* We reported in September 2009 that the Defense Readiness Reporting
System (DRRS) program[Footnote 43] was not being effectively managed
and made recommendations to address a number of acquisition management
weaknesses including the absence of a reliable integrated master
schedule and well defined and managed requirements, and adequate
testing.[Footnote 44] As stated in our report, we briefed the DRRS
program office on the results of our work prior to its DOD Human
Resources Management IRB certification review. However, these results
were not disclosed to the IRB. Rather, the certification package that
the precertification authority submitted to the IRB stated that DRRS
was on track for meeting its cost, schedule, and performance measures
and highlighted no program risks. Based on this submission, DRRS was
certified by the IRB and approved by the DBSMC to obligate $24.625
million in fiscal years 2009 and 2010. According to the chair of the
IRB, the board did not validate the information in the submissions it
received, and the results of our review were not disclosed to the IRB.
* We reported in July 2008 that the Global Combat Support System-
Marine Corps (GCSS-MC) program[Footnote 45] had not been economically
justified on the basis of reliable estimates of both benefits and
costs and that key program management controls, such as the use of
earned value management and risk management had not been properly
implemented.[Footnote 46] Accordingly, we made recommendations to
address these weaknesses. GCSS-MC was certified with conditions and
recertified during fiscal year 2009. Neither the weaknesses that we
previously reported nor the status of our recommendations to address
them were evident in the conditions accompanying the certification,
even though our recommendations had yet to be implemented at the time
of these certification actions.
* We reported in September 2008[Footnote 47] that the Navy ERP program
did not use important cost estimating practices when economically
justifying the program, did not implement key aspects of earned value
management, and did not have risk mitigation strategies in place to
address the risks described in our report, including risks associated
with these issues. Accordingly, we made recommendations to address
these weaknesses. However, when Navy ERP was recertified during fiscal
year 2009, conditions relative to any of these weaknesses did not
accompany the recertification, although our recommendations had yet to
be implemented at the time of recertification.
Officials representing each of the IRBs stated that the boards depend
on the component precertification authorities to provide them with
complete and reliable information about each system investment. Among
other things, IRB officials stated that such information should
include the results of reviews by us and others. However, DOD guidance
does not state that GAO-related information, such as open
recommendations or the focus and results of our ongoing reviews, is to
be included in the certification packages provided to the IRBs.
Further, the Special Assistant to the Deputy Chief Management Officer
told us it is each program's milestone decision authority[Footnote 48]
that is ultimately responsible for addressing known program management
issues, including those raised by GAO.
By not having and considering relevant information about the state of
each system modernization investment certified and approved, such as
the results of our reviews and the status of actions to implement our
recommendations that pertain to the investment, DOD's certification
and approval decisions are based on limited information, and thus may
not be justified.
DOD's Report States That Certification Waivers Were Not Granted:
The act requires DOD to identify in its annual report any defense
business system modernization with an obligation in excess of $1
million during the preceding fiscal year that was not certified and
approved according to the act's provisions, along with any reasons for
these requirements being waived. According to DOD's latest annual
report, system investments were certified according to the act's
requirements during fiscal year 2009, and no systems were granted a
certification waiver. Similarly, each of DOD's annual reports since
March of 2006 has stated that no systems were approved on the basis of
a certification waiver. According to officials representing each of
the IRBs, while program officials sometimes seek to be certified on
the basis of a waiver, their practice is to ensure that the program
office addresses any issues underlying a waiver request before the
investment is placed on an IRB's certification review agenda. As a
result, they stated that a system is unlikely to go before an IRB for
certification until it can be certified with conditions.
Conclusions:
DOD's latest annual report on its business systems modernization
program complies with statutory requirements pertaining to the
report's content, but the scope and completeness of key information
that is provided in the report is otherwise limited. In particular,
the report omits information on numerous business system investment
certification actions taken during fiscal year 2009. In addition,
while it includes schedule-focused performance measures and
performance against these measures for the modernization investments
discussed, as required by statute, it does not include similar
information for other performance measures, such as cost, capability,
and benefit commitments and performance against these commitments.
Collectively, this means that DOD's annual report does not provide
congressional committees with the full range of information necessary
to permit meaningful and informed oversight of DOD's business systems
modernization program.
Beyond the scope and content of DOD's annual report, the basis for the
IRB certifications have been limited because DOD guidance does not
provide for disclosure of our findings concerning investments being
considered. In particular, investments have been certified and
approved without conditions even though our prior reports have
identified program weaknesses that were unresolved at the time of
certification and approval. As a result, these certification and
approval decisions may not be sufficiently justified.
Recommendations for Executive Action:
To facilitate congressional oversight and promote departmental
accountability, we recommend that the Secretary of Defense direct the
Deputy Secretary of Defense, as the chair of the DBSMC, to ensure that
the scope and content of future DOD annual reports to Congress on
compliance with section 332 of the Ronald W. Reagan National Defense
Authorization Act for Fiscal Year 2005, as amended, be expanded to
include:
* Cost, capability, and benefit performance measures for each business
system modernization investment and actual performance against these
measures.
* All certification actions, as defined in DOD guidance, which were
taken in the previous year by the department on its business system
modernization investments.
To ensure that IRB certification actions are better informed and
justified, we further recommend that the Secretary direct the Deputy
Secretary to ensure that DOD guidance be revised to include provisions
that require IRB certification submissions disclose program weaknesses
raised by GAO and the status of actions to address our recommendations
to correct the weaknesses.
Agency Comments:
In written comments on a draft of this report, signed by the Assistant
Deputy Chief Management Officer and reprinted in appendix II, the
department agreed with our recommendations.
We are sending copies of this report to interested congressional
committees; the Director, Office of Management and Budget; and the
Secretary of Defense. This report will also be available at no charge
on our Web site at [hyperlink, http://www.gao.gov].
If you or your staffs have any questions on matters discussed in this
report, please contact me at (202) 512-3439 or hiter@gao.gov. Contact
points for our Offices of Congressional Relations and Public Affairs
may be found on the last page of this report. GAO staff who made major
contributions to this report are listed in appendix III.
Signed by:
Randolph C. Hite:
Director:
Information Technology Architecture and Systems Issues:
List of Committees:
The Honorable Carl Levin:
Chairman:
The Honorable John McCain:
Ranking Member:
Committee on Armed Services:
United States Senate:
The Honorable Daniel Inouye:
Chairman:
The Honorable Thad Cochran:
Ranking Member:
Subcommittee of Defense:
Committee on Appropriations:
United States Senate:
The Honorable Ike Skelton:
Chairman:
The Honorable Howard P. McKeon:
Ranking Member:
Committee on Armed Services:
House of Representatives:
The Honorable Norm Dicks:
Chairman:
The Honorable C.W. Bill Young:
Ranking Member:
Subcommittee on Defense:
Committee on Appropriations:
House of Representatives:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
As agreed with congressional defense committees, our objective was to
assess the actions by the Department of Defense (DOD) to comply with
the requirements of key aspects of section 332 of the Ronald W. Reagan
National Defense Authorization Act for Fiscal Year 2005 (the act).
[Footnote 49] To address this, we focused on the extent to which DOD's
annual report to Congress addressed the following provisions of the
act: (1) describe milestones and actual performance against specified
measures and any revisions, (2) discuss specific improvements in
business operations and cost savings resulting from successful
business system modernization efforts, (3) describe specific actions
on each business system investment submitted for certification, and
(4) identify any business system investment with an obligation in
excess of $1 million that was not certified during the preceding
fiscal year and reasons for the waiver. Our methodology relative to
each of the four requirements is as follows:
* To determine whether the DOD annual report described milestones and
actual performance against specified measures and any revisions, we
compared information contained in the annual report to what the act
required. Further, we compared the types of measures included in the
annual report to those commonly associated with program performance,
such as those described in prior GAO work related to performance
measures.[Footnote 50] In addition, we interviewed officials from the
Business Transformation Agency (BTA) and each of DOD's investment
review boards (IRB) to understand the process used to identify and
track milestones and other performance measures. We did not
independently validate the accuracy of the milestone dates included in
the report.
* To determine the extent to which DOD's annual report discussed
specific improvements in business operations and cost savings, we
reviewed each of the 18 case-in-point narratives included in the
annual report that described examples of business improvements and
other benefits. We compared this information with the act's reporting
requirements to identify any variances. We did not validate the
accuracy of the improvements or benefits discussed in the case-in-
point narratives.
* To determine the extent to which DOD's annual report identified
specific actions on each business systems investment submitted for
certification, we reviewed and analyzed all Defense Business Systems
Management Committee (DBSMC) certification approval memoranda as well
as IRB certification memoranda and IRB meeting minutes issued prior to
the DBSMC's final approval decisions for fiscal year 2009 and compared
the results to those certification actions described in the annual
report to identify differences. We also reviewed DOD IRB guidance to
understand the types of actions related to certification of business
system modernizations.
For certification actions included in the DBSMC and IRB memoranda but
not described in the annual report, we interviewed officials from the
BTA, IRBs, the Office of the Assistant Secretary of Defense (Networks
and Information Integration)/DOD Chief Information Officer (ASD(NII)/
DOD CIO), and the Office of the DOD Deputy Chief Management Officer
(DCMO) as to the reason for the differences.
For certification actions included in the report and described in
fiscal year 2009 DBSMC and IRB memoranda, we compared information
about specific DOD programs from recent GAO reports to the conditions
associated with certification actions described in the annual report
and the DBSMC and IRB memoranda to determine whether IRBs placed
certification conditions related to program weaknesses identified by
GAO and whether those conditions addressed those weaknesses.[Footnote
51] In addition, we interviewed DCMO, BTA, and IRB staff to discuss
conditions that were reported as part of certification actions and
what is submitted to the IRBs when individual systems request
certification.
* To determine if DOD's annual report identified any business system
investment with an obligation in excess of $1 million that was not
certified during the preceding fiscal year and the reasons for any
waivers granted, we reviewed DBSMC and IRB certification memoranda and
compared actions taken during fiscal year 2009 to the actions
described in DOD's annual report. We also interviewed DCMO and BTA
officials, as well as IRB support staff, to determine if any waivers
were issued during fiscal year 2009. Finally, we reviewed DOD's annual
reports from 2005 to present to determine the extent to which these
reports identify any waivers issued prior to fiscal year 2009.
We conducted this performance audit at DOD offices in Arlington,
Virginia, from January 2010 to May 2010, in accordance with generally
accepted government auditing standards. Those standards require that
we plan and perform the audit to obtain sufficient, appropriate
evidence to provide a reasonable basis for our findings and
conclusions based on our audit objectives.
[End of section]
Appendix II: Comments from the Department of Defense:
Department of Defense:
Office Of The Deputy Chief Management Officer:
9010 Defense Pentagon:
Washington, DC 20301-9010:
May 14 2010:
Mr. Randolph C. Hite:
Director, Information Technology Architecture and Systems Issues:
U.S. Government Accountability Office:
441 G Street, N.W.
Washington, DC 20548:
Dear Mr. Hite:
This is the Department of Defense (DOD) response to the GAO draft
report GAO-10-663, "Business Systems Modernization: Scope and Content
of DoD's Congressional Report and Executive Oversight of Investments
Need to Improve" dated May 4, 2010 (GAO Code 310682).
Sincerely,
Signed by:
Elizabeth A. McGrath:
Assistant Deputy Chief Management Officer:
[End of letter]
GAO Draft Report ” Dated May 4, 2010:
GAO Code 310682/GA0-10-663:
"Business System Modernization: Scope and Content of DoD's
Congressional Report and Executive Oversight of Investments Need to
Improve"
Department Of Defense Comments To The Recommendations:
Recommendation 1: The GAO recommends that the Secretary of Defense
direct the Deputy Secretary of Defense, as the chair of the Defense
Business Systems Management Committee (DBSMC), to ensure that the
scope and content of future DoD annual reports to Congress on
compliance with section 332 of the National Defense Authorization Act
(NDAA) of 2005, as amended, be expanded to include:
* Cost, capability, and benefit performance measures for each business
system modernization investment and actual performance against these
measures.
* All certification actions, as defined in DoD guidance, which were
taken in the previous year by the Department on its business system
modernization investments.
DOD Response: Concur. Consistent with the legislative requirements for
the Strategic Management Plan (SMP), the Department agrees that
capturing information on program performance against cost, capability,
and benefit measures would better inform Congress in its oversight of
business system modernization programs. The Department is committed to
capturing this information and is actively considering whether the
most appropriate forum is the Enterprise Transition Plan (ETP) or the
annual Congressional report on defense business operations. Once the
Department has made this determination. DoD will incorporate the
information incrementally, beginning with the Tier I business systems,
and will subsequently include Tier II and Tier III business systems.
The Department also concurs with the GAO's recommendation to include
all certification actions as defined in its guidance, and the
Department has broadened the information provided in its Congressional
Report since DoD issued its inaugural report in March 2006. As
required by statute, Congressional Reports have only counted the total
number of new certifications and would not include re-certifications
unless there was a change to milestones and performance measures. This
year's report added an additional level of detail regarding
certifications and included decertification information. DoD did not
include recertification in this year's report, but will include them
in future reports.
Recommendation 2: The GAO recommends that the Secretary of Defense
direct the Deputy Secretary of Defense to ensure that DoD guidance be
revised to include provisions that require Investment Review Board
(ERB) certification submissions disclose program weaknesses raised by
GAO and the status of actions to address our recommendations to
correct the weaknesses.
DOD Response: Concur. DoD agrees with the GAO recommendation to
require Investment Review Board (IRB) Certification submissions to
disclose program weaknesses raised by GAO and the status of actions to
address GAO's recommendations to correct the weaknesses, if applicable
or available. At this time, an update to the January 2009 version of
the DoD Information Technology (IT) Defense Business Systems
Investment Review Process Guidance (IRB Guidance) is in progress, and
the Department will include this requirement in the revision.
However, it should be noted that the current IRB Guidance does not
preclude a program from submitting additional information such as
program weaknesses raised by GAO. Current guidance provides: "Pre-
Certification Authorities (PCAs) are responsible for internal
Certification and review of system modernization funding requests by
Program Managers (PMs), in addition to ensuring that requests are
submitted to the IRB with complete, current and accurate documentation
and within the prescribed deadlines." PCA memoranda are also intended
to acknowledge additional program issues that cannot be addressed
within the confines of the DoD Information Technology Portfolio
Repository (DITPR) dashboard.
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Randolph Hite, (202) 512-3439 or hiter@gao.gov:
Staff Acknowledgments:
In addition to the contact person named above, key contributors to
this report were Carl Barden, Justin Booth, Nancy Glover, Michael
Holland, Neelaxi Lakhmani (Assistant Director), Kate Nielsen,
Constantine Papanastasiou, Christine San, Sylvia Shanks, Jennifer
Stavros-Turner, and Adam Vodraska.
[End of section]
Footnotes:
[1] Business systems support DOD's business operations such as
civilian personnel, finance, health, logistics, military personnel,
procurement, and transportation.
[2] GAO, High-Risk Series: An Update, [hyperlink,
http://www.gao.gov/products/GAO-09-271] (Washington, D.C.: Jan. 22,
2009).
[3] Pub. L. No. 108-375, § 332, 118 Stat. 1811, 1851-1856 (Oct. 28,
2004), as amended (10 U.S.C. § 2222).
[4] The act requires designated approval authorities to certify that a
defense business system modernization is (1) in compliance with the
enterprise architecture, (2) necessary to achieve critical national
security capability or address a critical requirement in an area such
as safety or security, or (3) necessary to prevent a significant
adverse effect on a project that is needed to achieve an essential
capability, taking into consideration the alternative solutions for
preventing such an adverse effect.
[5] 10 U.S.C. § 2222.
[6] See for example, GAO, Military Readiness: DOD Needs to Strengthen
Management and Oversight of the Defense Readiness Reporting System,
[hyperlink, http://www.gao.gov/products/GAO-09-518] (Washington, D.C.:
Sept. 25, 2009) and DOD Business Systems Modernization: Recent
Slowdown in Institutionalizing Key Management Controls Needs to Be
Addressed, [hyperlink, http://www.gao.gov/products/GAO-09-586]
(Washington, D.C.: May 18, 2009).
[7] Congress provided DOD with about $661 billion in appropriations
for fiscal year 2010.
[8] GAO, Business Systems Modernization: DOD Continues to Improve
Institutional Approach, but Further Steps Needed, [hyperlink,
http://www.gao.gov/products/GAO-06-658] (Washington, D.C.: May 15,
2006).
[9] See, for example, GAO, DOD Business Systems Modernization: Planned
Investment in Navy Program to Create Cashless Shipboard Environment
Needs to Be Justified and Better Managed, [hyperlink,
http://www.gao.gov/products/GAO-08-922] (Washington, D.C.: Sept. 8,
2008); DOD Travel Cards: Control Weaknesses Resulted in Millions of
Dollars of Improper Payments, [hyperlink,
http://www.gao.gov/products/GAO-04-576] (Washington, D.C.: June 9,
2004); Military Pay: Army National Guard Personnel Mobilized to Active
Duty Experienced Significant Pay Problems, [hyperlink,
http://www.gao.gov/products/GAO-04-89] (Washington, D.C.: Nov. 13,
2003); and Defense Inventory: Opportunities Exist to Improve Spare
Parts Support Aboard Deployed Navy Ships, [hyperlink,
http://www.gao.gov/products/GAO-03-887] (Washington, D.C.: Aug. 29,
2003).
[10] [hyperlink, http://www.gao.gov/products/GAO-09-271].
[11] These eight high-risk areas include DOD's overall approach to
business transformation, business systems modernization, financial
management, the personnel security clearance program, supply chain
management, support infrastructure management, weapon systems
acquisition, and contract management.
[12] The seven governmentwide high-risk areas include disability
programs, ensuring the effective protection of technologies critical
to U.S. national security interests, interagency contracting,
information systems and critical infrastructure, information sharing
for homeland security, human capital, and real property.
[13] Pub. L. No. 110-181, §904, 122 Stat. 3, 273 (Jan. 28, 2008).
[14] An enterprise architecture, or modernization blueprint, provides
a clear and comprehensive picture of an entity, whether it is an
organization (e.g., federal department or agency) or a functional or
mission area that cuts across more than one organization (e.g.,
financial management). This picture consists of snapshots of the
enterprise's current or "as is" operational and technological
environment and its target or "to be" environment, and contains a
capital investment road map for transitioning from the current to the
target environment. These snapshots consist of "views," which are
basically one or more architecture products that provide conceptual or
logical representations of the enterprise.
[15] According to DOD, the business mission area is responsible for
ensuring that capabilities, resources, and materiel are reliably
delivered to the warfighter. Specifically, the business mission area
addresses areas such as real property and human resources management.
[16] The act requires designated approval authorities to certify that
a defense business system modernization is (1) in compliance with the
enterprise architecture, (2) necessary to achieve critical national
security capability or address a critical requirement in an area such
as safety or security, or (3) necessary to prevent a significant
adverse effect on a project that is needed to achieve an essential
capability, taking into consideration the alternative solutions for
preventing such an adverse effect.
[17] These IRBs are for Financial Management, Weapon Systems Lifecycle
Management and Materiel Supply and Services Management, Real Property
and Installations Lifecycle Management, and Human Resources
Management. In August 2009, DOD's Enterprise Guidance Board was
chartered as the DOD CIO IRB.
[18] Pub. L. No. 110-181, §904.
[19] Pub. L. No. 110-417, §908, 122 Stat. 4356, 4569 (Oct. 14, 2008).
[20] Pub. L. No. 108-375, § 332, 118 Stat. 1811, 1851-1856 (Oct. 28,
2004), as amended (10 U.S.C. § 2222).
[21] The act requires designated approval authorities to certify that
a defense business system modernization is (1) in compliance with the
enterprise architecture, (2) necessary to achieve critical national
security capability or address a critical requirement in an area such
as safety or security, or (3) necessary to prevent a significant
adverse effect on a project that is needed to achieve an essential
capability, taking into consideration the alternative solutions for
preventing such an adverse effect.
[22] GAO, DOD Business Systems Modernization: Progress in Establishing
Corporate Management Controls Needs to Be Replicated Within Military
Departments, [hyperlink, http://www.gao.gov/products/GAO-08-705]
(Washington, D.C.: May 15, 2008); DOD Business Systems Modernization:
Progress Continues to Be Made in Establishing Corporate Management
Controls, but Further Steps Are Needed, [hyperlink,
http://www.gao.gov/products/GAO-07-733] (Washington, D.C.: May 14,
2007); Business Systems Modernization: DOD Continues to Improve
Institutional Approach, but Further Steps Needed, [hyperlink,
http://www.gao.gov/products/GAO-06-658] (Washington, D.C.: May 15,
2006); and DOD Business Systems Modernization: Important Progress Made
in Establishing Foundational Architecture Products and Investment
Management Practices, but Much Work Remains, [hyperlink,
http://www.gao.gov/products/GAO-06-219] (Washington, D.C.: Nov. 23,
2005).
[23] [hyperlink, http://www.gao.gov/products/GAO-09-586].
[24] According to the department, fiscal year 2009 planned milestones
are described in DOD's September 2008 Enterprise Transition Plan.
[25] Key milestones in the DOD acquisition process are Milestone A,
Milestone B, Milestone C, Initial Operational Capability, and Full
Operational Capability. These are decision points to determine whether
to initiate, continue, advance, change direction in, or terminate a
project or program work effort, which result in advancement to or
restriction from entering the next major acquisition process phase.
[26] According to DOD's annual report, DOD does not consider programs
at or beyond the initial operating capability phase and that are no
longer modernizing to be modernization programs and does not report
related milestones and performance measures in its fiscal year 2009
enterprise transition plan or March 2010 congressional report.
[27] DOD includes more than one fiscal year 2009 milestone for some
business systems, while other business systems did not have any
planned milestones during fiscal year 2009.
[28] The vast majority of business systems either do not involve
modernizations or are small investments (i.e., less than $1 million)
that are not subject to the IRB certification and DBSMC approval
processes. For purposes of reporting milestones in its fiscal year
2010 annual report, DOD only includes programs that have been approved
by the DBSMC and that had planned milestones for fiscal year 2009.
[29] GAO, Defense Acquisitions: Measuring the Value of DOD's Weapon
Programs Requires Starting with Realistic Baselines, [hyperlink,
http://www.gao.gov/products/GAO-09-543T] (Washington, D.C.: Apr. 1,
2009); DOD Business Systems Modernization: Planned Investment in Navy
Program to Create Cashless Shipboard Environment Needs to Be Justified
and Better Managed, [hyperlink,
http://www.gao.gov/products/GAO-08-922] (Washington, D.C.: Sept. 8,
2008); Managing for Results: Enhancing Agency Use of Performance
Information for Management Decision Making, [hyperlink,
http://www.gao.gov/products/GAO-05-927] (Washington, D.C.: Sept. 9,
2005); Information Technology: Customs Automated Commercial
Environment Program Progressing, but Need for Management Improvements
Continues, [hyperlink, http://www.gao.gov/products/GAO-05-267]
(Washington, D.C.: Mar. 14, 2005); Information Technology: DOD's
Acquisition Policies and Guidance Need to Incorporate Additional Best
Practices and Controls, [hyperlink,
http://www.gao.gov/products/GAO-04-722] (Washington, D.C.: July 30,
2004); Executive Guide Improving Mission Performance Through Strategic
Information Management and Technology, [hyperlink,
http://www.gao.gov/products/GAO/AIMD-94-115] (Washington, D.C.: May
2004).
[30] According to DOD, the Army Learning Management System is a subset
of the Army Distributed Learning System, which supports IT
infrastructure for delivering distributed learning content for
training.
[31] An ERP solution is an automated system using commercial off-the-
shelf software consisting of multiple, integrated functional modules
that perform a variety of business-related tasks such as payroll,
general ledger accounting, and supply chain management.
[32] The approval authorities, as discussed earlier in this report,
include the Under Secretary of Defense for Acquisition, Technology,
and Logistics; the Under Secretary of Defense (Comptroller); the Under
Secretary of Defense for Personnel and Readiness; the ASD(NII)/DOD
CIO; and the Deputy Secretary of Defense. They are responsible for the
review, approval, and oversight of business systems and must establish
investment review processes for systems under their cognizance.
[33] The act requires certification by designated approval authorities
that the defense business system modernization is (1) in compliance
with the enterprise architecture, (2) necessary to achieve critical
national security capability or address a critical requirement in an
area such as safety or security, or (3) necessary to prevent a
significant adverse effect on a project that is needed to achieve an
essential capability, taking into consideration the alternative
solutions for preventing such an adverse effect.
[34] Investments that are below the $1 million threshold but have been
designated as an interest program by an IRB are also subject to IRB
review and DBSMC approval.
[35] Among other things, DITPR, which is the department's
authoritative business systems inventory, contains information about
certifications, including any certification conditions placed on the
system.
[36] The NDAA for Fiscal Year 2010 adds the requirement that a
component's chief management officer assert that new investments have
undertaken sufficient business process reengineering efforts. Since
DOD's March 2010 report focuses on fiscal year 2009, our report does
not include information about this new requirement.
[37] The Standard Financial Information Structure is intended to
provide a standardized DOD-wide financial information structure to
improve cost accounting, analysis, and reporting.
[38] The Item Unique Identification Registry is a relational database
that is intended to store acquisition and logistics information to
track, catalog, and inventory items, such as equipment and spare
parts, via machine-readable item identifiers.
[39] DOD's reported numbers combine multiple certification actions
that took place on individual systems within specific categories of
certification actions. For example, if a system is certified twice,
both times without conditions, within fiscal year 2009, this system is
counted only once under the category of certification without
conditions. For the purposes of comparing our analysis to DOD reported
data, we also combined multiple certification actions that took place
on individual systems within specific categories of certification
actions.
[40] Although DOD provided additional documentation indicating the
four system certifications might be recertifications, the information
was not consistent with the DBSMC and IRB decision memoranda and
meeting minutes.
[41] [hyperlink, http://www.gao.gov/products/GAO-08-972].
[42] As described previously in this report, DOD does not report on
the results of recertifications in its annual report.
[43] DRRS is intended to provide timely, objective, and accurate data
about DOD force readiness.
[44] [hyperlink, http://www.gao.gov/products/GAO-09-518].
[45] GCSS-MC is intended to modernize the Marine Corps logistics
systems and to provide the Corps with timely and complete logistics
information to support the warfighter.
[46] GAO, DOD Business Systems Modernization: Key Marine Corps System
Acquisition Needs to be Better Justified, Defined, and Managed,
[hyperlink, http://www.gao.gov/products/GAO-08-822] (Washington, D.C.:
July 28, 2008).
[47] GAO, DOD Business Systems Modernization: Important Management
Controls Being Implemented on Major Navy Program, but Improvements
Needed in Key Areas, [hyperlink,
http://www.gao.gov/products/GAO-08-896] (Washington, D.C.: Sept. 8,
2008).
[48] According to DOD, the milestone decision authority is the
designated individual who has overall responsibility for an
investment. This person has the authority to approve an investment's
progression in the acquisition process and is responsible for
reporting cost, schedule, and performance results. For example, the
milestone decision authority for a Major Automated Information System
is the ASD(NII)/DOD CIO or a designee.
[49] Ronald W. Reagan National Defense Authorization Act for Fiscal
Year 2005, Pub. L. No. 108-375, § 332, 118 Stat. 1811, 1851-1856 (Oct.
28, 2004), as amended (10 U.S.C. § 2222).
[50] GAO, Managing for Results: Enhancing Agency Use of Performance
Information for Management Decision Making, [hyperlink,
http://www.gao.gov/products/GAO-05-927] (Washington, D.C.: Sept. 9,
2005); Information Technology: Customs Automated Commercial
Environment Program Progressing, but Need for Management Improvements
Continues, [hyperlink, http://www.gao.gov/products/GAO-05-267]
(Washington, D.C.: Mar. 14, 2005); Information Technology: DOD's
Acquisition Policies and Guidance Need to Incorporate Additional Best
Practices and Controls, [hyperlink,
http://www.gao.gov/products/GAO-04-722] (Washington, D.C.: July 30,
2004); and Executive Guide Improving Mission Performance Through
Strategic Information Management and Technology, [hyperlink,
http://www.gao.gov/products/GAO/AIMD-94-115] (Washington, D.C.: May
2004).
[51] GAO, DOD Business Systems Modernization: Planned Investment in
Navy Program to Create Cashless Shipboard Environment Needs to Be
Justified and Better Managed, [hyperlink,
http://www.gao.gov/products/GAO-08-922] (Washington, D.C.: Sept. 8,
2008); DOD Business Systems Modernization: Important Management
Controls Being Implemented on Major Navy Program, but Improvements
Needed In Key Areas, [hyperlink,
http://www.gao.gov/products/GAO-08-896] (Washington, D.C.: Sept. 8,
2008); DOD Business Systems Modernization: Key Navy Programs'
Compliance with DOD's Federated Business Enterprise Architecture Needs
to Be Adequately Demonstrated, [hyperlink,
http://www.gao.gov/products/GAO-08-972] (Washington, D.C.: Aug. 7,
2008); and DOD Business Systems Modernization: Lack of an Integrated
Strategy Puts the Army's Asset Visibility System Investments at Risk,
[hyperlink, http://www.gao.gov/products/GAO-07-860] (Washington, D.C.:
July 27, 2007).
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
