DOD Business Systems Modernization
Important Management Controls Being Implemented on Major Navy Program, but Improvements Needed in Key Areas Gao ID: GAO-08-896 September 8, 2008The Department of Defense (DOD) has long been challenged in implementing key information technology (IT) management controls on its thousands of business system investments. For this and other reasons, GAO has designated DOD's business systems modernization efforts as high-risk. One of the larger business system investments is the Department of the Navy's Enterprise Resource Planning (Navy ERP) program. Initiated in 2003, the program is to standardize the Navy's business processes, such as acquisition and financial management. It is being delivered in increments, the first of which is to cost about $2.4 billion over its useful life and be fully deployed in fiscal year 2013. GAO was asked to determine whether key IT management controls are being implemented on the program. To do this, GAO analyzed, for example, requirements management, economic justification, earned value management, and risk management.
DOD has implemented key IT management controls on its Navy ERP program to varying degrees of effectiveness. To its credit, the control associated with managing system requirements is being effectively implemented. In addition, important aspects of other controls have at least been partially implemented, including those associated with economically justifying investment in the program and proactively managing program risks. Nevertheless, other aspects of these controls, as well as the bulk of what is needed to effectively implement earned value management, which is a recognized means for measuring program progress, have not been effectively implemented. Among other things, these control weaknesses have contributed to the more than 2-year schedule delay and the almost $600 million cost overruns already experienced on the program since it began, and they will likely contribute to future delays and overruns if they are not corrected. Examples of the weaknesses are: (1) Investment in the program has been economically justified on the basis of expected benefits that far exceed estimated costs ($8.6 billion versus $2.4 billion over a 20-year life cycle). However, important estimating practices, such as using historical cost data from comparable programs and basing the cost estimate on a reliable schedule baseline were not employed. While these weaknesses are important because they limit the reliability of the estimates, GAO's analysis shows that they would not have altered the estimates to the point of not producing a positive return on investment. (2) Earned value management has not been effectively implemented. To its credit, the program office has elected to implement program-level earned value management. In doing so, however, basic prerequisites for effectively managing earned value have not been executed. In particular, the integrated master schedule was not derived in accordance with key estimating practices, and an integrated baseline review has not been performed on any of the first increment's releases. (3) A defined process for proactively avoiding problems, referred to as risk management, has been established, but risk mitigation strategies have not been effectively implemented for all significant risks, such as those associated with data conversion and organizational change management, as well the risks associated with the above-cited weaknesses. The reasons that program management and oversight officials cited for these practices not being executed range from the complexity and challenges of managing and implementing a program of this size to limitations in the program office's scope and authority. Notwithstanding the effectiveness with which important aspects of several controls have been implemented, the above-cited weaknesses put DOD at risk of investing in a system solution that does not optimally support corporate mission needs and mission performance, and meet schedule and cost commitments.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-08-896, DOD Business Systems Modernization: Important Management Controls Being Implemented on Major Navy Program, but Improvements Needed in Key Areas
This is the accessible text file for GAO report number GAO-08-896
entitled 'DOD Business Systems Modernization: Important Management
Controls Being Implemented on Major Navy Program, but Improvements
Needed in Key Areas' which was released on September 8, 2008.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Subcommittee on Readiness and Management Support,
Committee on Armed Services, U.S. Senate:
United States Government Accountability Office:
GAO:
September 2008:
DOD Business Systems Modernization:
Important Management Controls Being Implemented on Major Navy Program,
but Improvements Needed in Key Areas:
GAO-08-896:
GAO Highlights:
Highlights of GAO-08-896, a report to the Subcommittee on Readiness and
Management Support, Committee on Armed Services, U.S. Senate.
Why GAO Did This Study:
The Department of Defense (DOD) has long been challenged in
implementing key information technology (IT) management controls on its
thousands of business system investments. For this and other reasons,
GAO has designated DOD‘s business systems modernization efforts as high-
risk. One of the larger business system investments is the Department
of the Navy‘s Enterprise Resource Planning (Navy ERP) program.
Initiated in 2003, the program is to standardize the Navy‘s business
processes, such as acquisition and financial management. It is being
delivered in increments, the first of which is to cost about $2.4
billion over its useful life and be fully deployed in fiscal year 2013.
GAO was asked to determine whether key IT management controls are being
implemented on the program. To do this, GAO analyzed, for example,
requirements management, economic justification, earned value
management, and risk management.
What GAO Found:
DOD has implemented key IT management controls on its Navy ERP program
to varying degrees of effectiveness. To its credit, the control
associated with managing system requirements is being effectively
implemented. In addition, important aspects of other controls have at
least been partially implemented, including those associated with
economically justifying investment in the program and proactively
managing program risks. Nevertheless, other aspects of these controls,
as well as the bulk of what is needed to effectively implement earned
value management, which is a recognized means for measuring program
progress, have not been effectively implemented. Among other things,
these control weaknesses have contributed to the more than 2-year
schedule delay and the almost $600 million cost overruns already
experienced on the program since it began, and they will likely
contribute to future delays and overruns if they are not corrected.
Examples of the weaknesses are provided below.
* Investment in the program has been economically justified on the
basis of expected benefits that far exceed estimated costs ($8.6
billion versus $2.4 billion over a 20-year life cycle). However,
important estimating practices, such as using historical cost data from
comparable programs and basing the cost estimate on a reliable schedule
baseline were not employed. While these weaknesses are important
because they limit the reliability of the estimates, GAO‘s analysis
shows that they would not have altered the estimates to the point of
not producing a positive return on investment.
* Earned value management has not been effectively implemented. To its
credit, the program office has elected to implement program-level
earned value management. In doing so, however, basic prerequisites for
effectively managing earned value have not been executed. In
particular, the integrated master schedule was not derived in
accordance with key estimating practices, and an integrated baseline
review has not been performed on any of the first increment‘s releases.
* A defined process for proactively avoiding problems, referred to as
risk management, has been established, but risk mitigation strategies
have not been effectively implemented for all significant risks, such
as those associated with data conversion and organizational change
management, as well the risks associated with the above-cited
weaknesses.
The reasons that program management and oversight officials cited for
these practices not being executed range from the complexity and
challenges of managing and implementing a program of this size to
limitations in the program office‘s scope and authority.
Notwithstanding the effectiveness with which important aspects of
several controls have been implemented, the above-cited weaknesses put
DOD at risk of investing in a system solution that does not optimally
support corporate mission needs and mission performance, and meet
schedule and cost commitments.
What GAO Recommends:
GAO is making recommendations to the Secretary of Defense aimed at
improving cost and schedule estimating, earned value management, and
risk management weaknesses. DOD largely agreed with GAO‘s
recommendations and described actions planned or under way to address
them.
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-896]. For more
information, contact Randolph C. Hite at (202) 512-3439 or
hiter@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
Implementation of Key DOD and Related IT Acquisition Management
Controls on Navy ERP Varies:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Objective, Scope, And Methodology:
Appendix II: Comments From The Department Of Defense:
Appendix III: GAO Contact And Staff Acknowledgments:
Tables:
Table 1: Description and Status of the Navy ERP Pilots:
Table 2: Navy ERP Template 1 Releases:
Table 3: Organizations Responsible for Navy ERP Oversight and
Management:
Table 4: Description of Business System IT Acquisition Management
Controls:
Table 5: Summary of Cost Estimating Characteristics That the Cost
Estimate Satisfies:
Table 6: Satisfaction of Schedule Estimating Key Practices:
Figures:
Figure 1: Navy ERP Time Line:
Figure 2: Navy ERP Milestones for Beginning Deployment:
Figure 3: Navy ERP Life Cycle Cost Estimates in Fiscal Years 2003,
2004, and 2007:
Figure 4: Cumulative Cost Variance for Navy ERP over a 17-Month Period:
Figure 5: Cumulative Schedule Variance of the Navy ERP Program over a
17-Month Period:
Abbreviations:
BEA: business enterprise architecture:
BTA: Business Transformation Agency:
CIO: Chief Information Officer:
DBSMC: Defense Business Systems Management Committee:
DOD: Department of Defense:
DON: Department of the Navy:
EVM: earned value management:
FOC: full operational capability:
GCSS-MC: Global Combat Support System-Marine Corps:
IOC: initial operational capability:
IRB: investment review board:
IT: information technology:
MDA: milestone decision authority:
NAVAIR: Naval Air Systems Command:
NAVSEA: Naval Sea Systems Command:
NAVSUP: Naval Supply Systems Command:
Navy ERP: Navy Enterprise Resource Planning:
NTCSS: Naval Tactical Command Support System:
OMB: Office of Management and Budget:
PMB: performance measurement baseline:
SAP: Systems Applications and Products:
SPAWAR: Space and Naval Warfare Systems Command:
TC-AIMS II: Transportation Coordinators' Automated Information for
Movements System:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
September 8, 2008:
The Honorable Daniel K. Akaka:
Chairman:
The Honorable John Thune:
Ranking Member:
Subcommittee on Readiness and Management Support:
Committee on Armed Services:
United States Senate:
The Honorable John Ensign:
United States Senate:
For decades, the Department of Defense (DOD) has been challenged in
modernizing its timeworn business systems.[Footnote 1] In 1995, we
designated DOD's business systems modernization program as high risk,
and continue to do so today.[Footnote 2] Our reasons include the
modernization's large size, complexity, and its critical role in
addressing other long-standing transformation and financial management
challenges. Other reasons are that DOD has yet to institutionalize key
system modernization management controls, and it has not demonstrated
the ability to consistently deliver promised system capabilities and
benefits on time and within budget.
Nevertheless, DOD continues to invest billions of dollars in thousands
of business systems, including about a hundred that the department has
labeled as business transformational programs, 12 of which account for
about 50 percent of these programs' costs. The Navy Enterprise Resource
Planning (Navy ERP) program is one such program. Initiated in 2003,
Navy ERP is to standardize the Navy's acquisition, financial, program
management, maintenance, plant and wholesale supply, and workforce
management business processes across its dispersed organizational
environment. As envisioned, the program consists of a series of major
increments, the first of which includes three releases and is expected
to cost approximately $2.4 billion over its 20-year life cycle and to
be fully operational in fiscal year 2013.
As agreed, our objective was to determine whether the Department of the
Navy (DON)[Footnote 3] is effectively implementing information
technology (IT) management controls on Navy ERP. To accomplish this, we
focused on the program's first increment by analyzing a range of
program documentation and interviewing cognizant officials relative to
the following management areas: architectural alignment, economic
justification, earned value management (EVM), requirements management,
and risk management. We conducted this performance audit from June 2007
to September 2008, in accordance with generally accepted government
auditing standards. Those standards require that we plan and perform
the audit to obtain sufficient, appropriate evidence to provide a
reasonable basis for our findings and conclusions based on our audit
objective. We believe that the evidence obtained provides a reasonable
basis for our findings and conclusions based on our audit objective.
Additional details on our objective, scope, and methodology are in
appendix I.
Results In Brief:
DOD has implemented key IT management controls on its Navy ERP program
to varying degrees of effectiveness. To its credit, one of the controls
has been fully implemented; important aspects of other controls have
not. Collectively, these management controls are to ensure that a given
system investment represents the right solution to filling a mission
need, meaning that the system is defined to (1) minimize overlap and
duplication and maximize interoperability with related systems and (2)
produce mission benefits commensurate with costs over its useful life.
The controls are also to ensure that the system is acquired and
deployed the right way, meaning that it is done in a way to maximize
the chances of delivering defined system capabilities and benefits on
time and within budget. Given that deployment of Navy ERP is more than
2 years behind schedule and is to cost about $570 million[Footnote 4]
more than was originally envisioned, these goals have already not been
fully met, in part because DOD program management and oversight
entities have not fully implemented several key IT management controls.
As a result, the department has yet to adequately demonstrate that the
program's first increment, as it has been defined, is the right
solution, and it is likely that the department will continue to add to
the program's cost overruns and schedule delays that the program has
already experienced to date. The strengths and weaknesses associated
with each of the IT management controls that we evaluated are described
here:
* Navy ERP compliance with DOD's federated business enterprise
architecture (BEA) has not been sufficiently demonstrated. To its
credit, the program office followed DOD's architecture compliance
assessment guidance and used the related assessment tool. However, this
guidance and tool do not adequately provide for addressing all relevant
aspects of architectural compliance. Specifically, the program's
compliance assessment (1) did not include all relevant architecture
products, such as those that describe technical and system elements;
(2) was not used to identify potential areas of duplication across
programs; and (3) did not address compliance with the DON enterprise
architecture. These important steps were not performed because of
policy, guidance, and tool limitations, and because aspects of the
corporate BEA and the DON enterprise architecture, which are both major
components of DOD's federated BEA, have yet to be sufficiently defined
to permit thorough compliance determinations in these areas. In
addition, program oversight and approval authorities did not validate
the program office's compliance assessments. As a result, the
department does not have a sufficient basis for knowing if Navy ERP has
been defined to minimize overlap with, and duplication of other
programs' functions, and is being defined and designed to maximize
interoperability among related programs.
* Investment in Navy ERP has been economically justified on the basis
of expected life cycle benefits that far exceed estimated life cycle
costs ($8.6 billion versus $2.4 billion over a 20-year life cycle).
However, these benefit estimates have not been subject to analysis of
how uncertainty in assumptions and data could impact them, as
prescribed in relevant guidance. According to program officials, such
uncertainty analysis is not warranted because they have taken and
continue to take steps to validate the assumptions and the data, such
as using the latest budget data associated with retiring legacy
systems, and monitoring changes to the systems' retirement dates. While
these steps are positive, they do not eliminate the need for
uncertainty analysis. Accordingly, we assessed key uncertainty
variables and found that while the inherent uncertainty in the
estimates would reduce expected savings, the reduction would be small
relative to a total benefit estimate of $8.6 billion.
With respect to the cost estimate, our analysis showed that it was not
derived using all key estimating practices contained in relevant
guidance. For example, the estimate was not grounded in a historical
record of comparable data from similar programs and was not based on a
reliable schedule baseline, which are both necessary to having a cost
estimate that can be considered credible and accurate. These practices
were not employed for various reasons, including DOD's lack of
historical data from similar programs and the lack of an integrated
master schedule for the program's first increment that includes all
three releases. Notwithstanding the fact that these limitations could
materially increase the $2.4 billion cost estimate, it is nevertheless
unlikely that they would increase the cost estimate to a level close to
the uncertainty-adjusted benefit expectations. Therefore, we have no
reason to believe that Navy ERP will not produce a positive return on
investment.
* EVM, which is a means for determining and disclosing actual program
performance in comparison with estimates, is not being effectively
implemented in Navy ERP. To its credit, the program office has elected
to implement program-level EVM.[Footnote 5] In doing so, however, basic
EVM activities have not been executed, which has produced, and will
likely continue to produce, actual program costs and schedules that do
not track close to estimates. For example, an integrated baseline
review, which is to verify that the program's costs and schedule are
reasonable given the program's scope of work and associated risks, has
not been performed on any of the first increment's releases. According
to program officials, this is because of the time it took to establish
program-level EVM capabilities and because of their focus on deploying
and stabilizing the first release. However, they recently stated that
one has been tentatively scheduled for August 2008. By not having an
integrated master schedule that has been subject to a baseline review,
as well as not employing other industry standards as discussed in this
report, Navy ERP will be challenged in implementing EVM effectively,
and cost overruns and lengthy schedule delays beyond those already
experienced by the program will likely occur. Our analysis of the
latest estimate for completing just the budgeted development work for
all three releases, which is about $844 million, shows that this
estimate will most likely have an overrun of about $152 million.
* An important requirements management activity has been effectively
implemented in Navy ERP. Specifically, the program office has ensured
that system requirements for the first release are traceable, as
evidenced by our analysis of 60 randomly selected system-level
requirements in which we confirmed that 58 are traceable backward to
operational requirements and forward to design specifications and test
results. Such traceability is important because it increases the
chances of delivering a system that performs as intended. Our analysis
of requirements in this sample also confirmed that system requirements
that had been reallocated from the first release to the other releases
were traceable, thus demonstrating traceability among product releases.
* The program office has defined a process for proactively managing
risks that reflects key practices governing how this IT management
control should be performed. However, it has not effectively
implemented this process for all identified risks. In particular, steps
taken to mitigate the risks associated with converting data from
existing systems to the new system and positioning user organizations
for the operational changes associated with the new system were not
effective. According to program officials, these mitigation strategies
could not be effectively implemented because the program office does
not have the authority to compel the user organizations to execute
their part of the mitigation strategy. As a result, the first user
organization to receive the Navy ERP experienced significant problems
during recent operational testing, and these problems will take
additional time and resources to correct. In addition, not all known
risks have been captured in the risk inventory. For example, the risks
associated with the two above discussed control weaknesses (not having
adequately demonstrated the program's architectural alignment and not
having implemented program-level EVM according to industry practices)
are not included in the risk inventory and are thus not being disclosed
and addressed. This is important because not having effectively
addressed such risks has contributed to schedule delays and will likely
contribute to more cost and schedule shortfalls.
Notwithstanding the effectiveness with which several program management
controls have been implemented on Navy ERP, the above-cited weaknesses
in implementing other management controls put DOD at risk of investing
in a system solution that does not optimally support corporate mission
needs and mission performance and continuing to fall short of program
schedule and cost commitments. Accordingly, we are making
recommendations to the Secretary of Defense aimed at addressing the
cost and schedule estimating, EVM, and risk management weaknesses,
thereby better ensuring that the program is managed to deliver the
right solution, the right way. We are not making recommendations in
this report for addressing the architecture compliance weaknesses
because we recently completed other work that is more broadly focused
on this management control across multiple programs.
In written comments on a draft of this report, signed by the Deputy
Under Secretary of Defense (Business Transformation) and reprinted in
appendix II, the department stated that it concurred with two, and
partially concurred with two, of our four recommendations. Further, it
stated that it has taken steps to address some of our recommendations,
adding that it is committed to implementing recommendations that
contribute to the program's success.
In partially concurring with two of the recommendations, DOD concurred
with most aspects of both. Nevertheless, for our recommendation aimed
at improving the program's cost estimates, it stated that it did not
concur with one aspect--ensuring that future cost estimates reflect the
risk of not having cost data for comparable programs. In doing so, DOD
stated that while the program had limited comparable data on which to
base the program's cost estimate, it had accounted for this limitation
in the cost estimate's uncertainty analysis. We do not agree that this
risk was reflected in the uncertainty analysis. We examined this
analysis as part of our review and found that it did not recognize this
risk. Moreover, DOD's comments did not include any evidence to the
contrary.
In addition, for our recommendation aimed at improving the program's
integrated master schedule, the department partially concurred with one
of the five components of the recommendation--defining a critical path
that incorporates all three releases of the system. In doing so, DOD
stated what our report already recognized, namely that the schedule
reflects a separate critical path for each release and that this is due
to the size and complexity of the releases. However, DOD offers no new
information in its comments. Further, our report also recognizes that
to be successful, large and complex programs that involve thousands of
activities need to ensure that their schedules integrate these
activities. In this regard, we support the department's commitment to
explore the feasibility of implementing this part of our
recommendation.
While concurring with all components of our other two recommendations,
the department nevertheless provided comments relative to the program's
use of EVM to explain why Release 1.0 of the system was not subject to
an integrated baseline review. For several reasons discussed in the
agency comments section of this report, we do not agree with these
additional comments. In addition, the department's comments described
actions planned or under way to address our recommendations. If fully
and properly implemented, DOD's described actions should go a long way
in addressing the weaknesses that we identify in the report.
Background:
DON's primary mission is to organize, train, maintain, and equip combat-
ready naval forces capable of winning the global war on terrorism and
any other armed conflict, deterring aggression by would-be foes,
preserving freedom of the seas, and promoting peace and security. To
support this mission, DON performs a variety of interrelated and
interdependent business functions (e.g., acquisition and financial
management), relying heavily on IT systems. In fiscal year 2008, DON's
budget for business systems and associated infrastructure was about
$2.7 billion, of which $2.2 billion was allocated to operations and
maintenance of existing systems and the remaining $500 million to
systems in development and modernization. Of the approximately 3,000
business systems that DOD reports in its current inventory, DON
accounts for 904, or about 30 percent, of the total. Navy ERP is one
such system investment.
Navy ERP: A Brief Description:
In July 2003, the Assistant Secretary of the Navy for Research,
Development, and Acquisition established Navy ERP to "converge" four
separate pilot programs that were under way at four separate Navy
commands.[Footnote 6] This program is to leverage a commercial, off-
the-shelf software known as an enterprise resource planning product.
Such products consist of multiple, integrated functional modules that
perform a variety of business-related tasks, such as acquisition and
financial management. Table 1 provides a brief description and status
of each of the pilots.
Table 1: Description and Status of the Navy ERP Pilots:
Pilot: SIGMA;
Description and status: Deployed at Naval Air Systems Command
(NAVAIR)[A] to support and link such business functions as program
management, contracting, financial, and human resources management. It
was retired in the first quarter of fiscal year 2008.
Pilot: CABRILLO;
Description and status: Deployed at Space and Naval Warfare Systems
Command (SPAWAR)[B] to support Navy Working Capital Fund financial
management. It is to be retired in fiscal year 2010.
Pilot: NEMAIS;
Description and status: Deployed at Naval Sea Systems Command
(NAVSEA)[C] to support regional maintenance, including intermediate-
level maintenance[D] management and human resources. It is to be
retired in fiscal year 2011.
Pilot: SMART;
Description and status: Deployed at Naval Supply Systems Command
(NAVSUP)[E] and NAVAIR to support national and local supply management,
intermediate-level maintenance management and to interface with
aviation depots. It was retired in fiscal year 2005.
Source: GAO analysis of DON data.
[A] NAVAIR is responsible for developing, delivering, and supporting
aircraft and weapons used by sailors and marines.
[B] SPAWAR is responsible for developing, delivering, and supporting
specialized command and control technologies, business information
technology, and space capabilities.
[C] NAVSEA is responsible for acquiring and maintaining the Navy's
ships and submarines.
[D] Intermediate-level maintenance is for repair or maintenance of
items that do not have to go to the depot level for major work but
cannot be maintained or repaired at the organizational level.
[E] NAVSUP is responsible for supply, fuel, and transportation, as well
as other logistics programs.
[End of table]
According to DOD, Navy ERP is to address the Navy's long-standing
problems related to financial transparency and asset visibility.
Specifically, the program is intended to standardize the Navy's
acquisition, financial, program management, maintenance, plant and
wholesale supply, and workforce management business processes across
its dispersed organizational components. When the program is fully
implemented, it is to support over 86,000 users.
Navy ERP is being developed in a series of increments using the Systems
Applications and Products (SAP) commercial software package, augmented
as needed by customized software. SAP consists of multiple, integrated
functional modules that perform a variety of business related tasks,
such as finance and acquisition. The first increment, called Template
1, is currently the only funded portion of the program and consists of
three releases: 1.0 Financial and Acquisition, 1.1 Wholesale and Retail
Supply, and 1.2 Intermediate-Level Maintenance. Release 1.0 is the
largest of the three releases in terms of the functional requirements
being addressed. Specifically, it is to provide about 56 percent of
Template 1 requirements. See table 2 for a description of these
releases.
Table 2: Navy ERP Template 1 Releases:
Release: 1.0 Financial and Acquisition;
Functionality: General Fund and Navy Working Capital Fund finance
applications, such as billing, budgeting, and cost planning;
Acquisition applications, such as activity based costing, contract
awards, and budget exhibits; Workforce management applications, such as
personnel administration, and training and events management.
Release: 1.1 Wholesale and Retail Supply;
Functionality: Wholesale applications, such as supply and demand
planning, order fulfillment, and supply forecasting; Retail supply
applications, such as inventory management, supply and demand
processing, and warehouse management.
Release: 1.2 Intermediate-Level Maintenance;
Functionality: Maintenance applications, such as maintenance
management, quality management, and calibration management.
Source: GAO analysis of DON data.
[End of table]
DON estimates the life cycle cost for the program's first increment to
be about $2.4 billion, including about $1 billion for acquisition, and
$1.4 billion for operations and maintenance. The life cycle cost of the
entire program has not yet been determined because future increments
have not been defined. The program office reported that approximately
$400 million was spent from fiscal year 2004 through fiscal year 2007
on the first increment. For fiscal year 2008, about $200 million is
planned to be spent.
Program Oversight and Management Roles and Responsibilities:
To manage the acquisition and deployment of Navy ERP, DON established a
program management office within the Program Executive Office for
Executive Information Systems. The program office manages the program's
scope and funding and is responsible for ensuring that the program
meets its objectives. To accomplish this, the program office is
responsible for key program management areas, such as architectural
alignment, economic justification, earned value management,
requirements management, and risk management. In addition, various DOD
and DON organizations share program oversight and review activities. A
listing of key entities and their roles and responsibilities is in
table 3.
Table 3: Organizations Responsible for Navy ERP Oversight and
Management:
Entity: Under Secretary of Defense for Acquisition, Technology, and
Logistics;
Roles and responsibilities: Serves as the milestone decision authority
(MDA), which according to DOD, has overall responsibility for the
program, to include approving the program to proceed through its
acquisition cycle on the basis of, for example, the acquisition plan,
an independently evaluated economic analysis, and the Acquisition
Program Baseline.
Entity: Assistant Secretary of the Navy, Research, Development, and
Acquisition;
Roles and responsibilities: Serves as DON's oversight organization for
the program, to include enforcement of Under Secretary of Defense for
Acquisition, Technology, and Logistics policies and procedures.
Entity: Department of the Navy, Program Executive Office for Executive
Information Systems;
Roles and responsibilities: Oversees a portfolio of large-scale
projects and programs designed to enable common business processes and
provide standard capabilities, to include reviewing the acquisition
plan, economic analysis, and the Acquisition Program Baseline prior to
approval by the MDA.
Entity: Department of the Navy Chief Information Officer (CIO);
Roles and responsibilities: Supports the department's planning,
programming, budgeting, and execution processes by ensuring that the
program has achievable and executable goals and conforms to financial
management regulations, and DON, DOD, and federal IT policies in
several areas (e.g., security, architecture, and investment
management); works closely with the program office during milestone
review assessments.
Entity: Office of the Secretary of Defense, Office of the Director for
Program Analysis and Evaluation;
Roles and responsibilities: Verifies and validates the reliability of
cost and benefit estimates found in economic analyses and provides its
results to the MDA.
Entity: Naval Center for Cost Analyses;
Roles and responsibilities: Performs independent costs estimates.
Entity: Defense Business Systems Management Committee (DBSMC);
Roles and responsibilities: Serves as the highest ranking governance
body for business systems modernization activities and approves
investments costing more than $1 million, as, for example, being
compliant with the BEA.
Entity: Investment Review Board (IRB);
Roles and responsibilities: Reviews business system investments and has
responsibility for recommending certification for all business system
investments costing more than $1 million that are asserted as compliant
with the BEA.
Entity: Business Transformation Agency (BTA);
Roles and responsibilities: Coordinates business transformation efforts
across DOD and supports the IRBs and DBSMC.
Entity: Navy ERP Program Management Office;
Roles and responsibilities: Performs day-to-day program management and,
as such, is the single point of accountability for managing the
program's objectives through development, deployment, and sustainment.
Source: DOD.
[End of table]
Overview of Navy ERP's Status:
The first increment of Navy ERP is currently in the production and
deployment phase of the defense acquisition system.[Footnote 7] The
system consists of five key program life cycle phases and three related
milestone decision points. These five phases and related milestones,
along with a summary of key program activities completed during or
planned for each phase, are as follows:
1. Concept Refinement: The purpose of this phase is to refine the
initial system solution (concept) and create a strategy for acquiring
the solution. This phase began in July 2003, at which time DON began to
converge the four pilot programs into Navy ERP and developed its first
cost estimate in September 2003. This phase of the program was combined
with the next phase, thus creating a combined Milestone A/B decision
point.
2. Technology Development: The purpose of this phase is to determine
the appropriate set of technologies to be integrated into the
investment solution by iteratively assessing the viability of the
various technologies while simultaneously refining user requirements.
During the combined Concept Refinement and Technology Development
phase, the program office prepared a concept of operations and
operational requirements document; performed an analysis of
alternatives, business case analysis, and economic analysis; and
established its first Acquisition Program Baseline. It also selected
SAP as the commercial off-the-shelf ERP software. The combined phase
was completed in August 2004, when the MDA approved Milestone A/B to
allow the program to move to the next phase.
3. System Development and Demonstration: The purpose of this phase is
to develop a system and demonstrate through developer testing that the
system can function in its target environment. This phase was completed
in September 2007, when Release 1.0 passed development testing and its
deployment to NAVAIR began. This was 17 months later than the program's
original schedule set in August 2004 but on time according to the
revised schedule set in December 2006.
In September 2004, the program office awarded a $176 million system
integration contract to BearingPoint for full system design,
development, and delivery using SAP's off-the-shelf product and related
customized software. In January 2006, the program office (1) reduced
the contractor's scope of work from development and integration of the
first increment to only development of the first release and (2)
assumed responsibility and accountability for overall system
integration. According to the program office, reasons for this change
included the need to change the development plan to reflect
improvements in the latest SAP product released and the lack of
authority by the contractor to adjudicate and reconcile differences
among the various Navy user organizations (i.e., Navy commands).
In December 2006, the program office revised its Acquisition Program
Baseline to reflect an increase of about $461 million in the life cycle
cost estimate due, in part, to restructuring the program (e.g.,
changing the order of the releases, changing the role of system
integrator from contractor to the program office) and resolving
problems related to, among other things, converting data from legacy
systems to run on Navy ERP and establishing interfaces between legacy
systems and Navy ERP. In addition, the program office awarded a $151
million contract for Release 1.1 and 1.2 configuration and development
to IBM in June 2007. In September 2007, prior to entering the next
phase, the program revised its Acquisition Program Baseline again to
reflect a $9 million decrease in the life cycle cost estimate and a 5-
month increase in its program schedule. Soon after, the MDA approved
Milestone C to move to the next phase.
4. Production and Deployment: The purpose of this phase is to achieve
an operational capability that satisfies the mission needs, as verified
through independent operational test and evaluation, and to implement
the system at all applicable locations. This phase began in September
2007, focusing first on achieving initial operational capability (IOC)
of Release 1.0 at NAVAIR by May 2008. This date is 22 months later than
the baseline established for Milestone A/B in August 2004, and 4 months
later than the new baseline established in September 2007. According to
program documentation, these delays were due, in part, to challenges
experienced at NAVAIR in converting data from legacy systems to run on
the new system and implementing new business procedures associated with
the system.
In light of the delays at NAVAIR in achieving IOC, the deployment
schedules for the other commands were also revised. Specifically,
Release 1.0 is still to be deployed at NAVSUP on October 2008, but
Release 1.0 deployment at SPAWAR is now scheduled 18 months later than
planned (October 2009), and deployment at NAVSEA general fund and Navy
Working Capital Fund is now scheduled to be 12 months later than
planned (October 2010 and 2011, respectively). Because of the Release
1.0 delays, Release 1.1 is now planned for deployment at NAVSUP 7
months later than planned (February 2010). Release 1.2 is still
scheduled to be released at Regional Maintenance Centers in October
2010.
The program office is currently in the process of again re-baselining
the program, and DON plans to address any cost overruns through
reprogramming of fiscal year 2008 DON funds.[Footnote 8] It estimates
that this phase will be completed with full operational capability
(FOC)[Footnote 9] by August 2013 (26 months later than the baseline
established in 2004, and 5 months later than the re-baseline
established in September 2007).
5. Operations and Support: The purpose of this phase is to
operationally sustain the system in the most cost-effective manner over
its life cycle. In this phase, the program plans to provide centralized
support to its users across all system commands. Each deployment site
is expected to perform complementary support functions, such as data
maintenance.
Overall, Increment 1 was originally planned to reach FOC in fiscal year
2011, and its estimated life cycle cost was about $1.87 billion.
[Footnote 10] The estimate was later baselined[Footnote 11] in August
2004 at about $2.0 billion.[Footnote 12] In December 2006 and again in
September 2007, the program was re-baselined. FOC is now planned for
fiscal year 2013, and the estimated life cycle cost is about $2.4
billion (31 percent increase over the original estimate).[Footnote 13]
Key activities for each phase are depicted in figure 1, changes in the
deployment schedule are depicted in figure 2, and cost estimates are
depicted in figure 3.
Figure 1: Navy ERP Time Line:
[See PDF for image]
This figure is an illustration of the Navy ERP time line, as follows:
Phase: Concept refinement and technology development;
Program established: Prior to the 2004 plan.
Phase: System development and demonstration;
2004 plan: mid-2004 through mid-2006;
* Template 1 contract rescoped to Release 1.0 in early 2006.
2007 plan: mid-2004 through end 2007;
* Template 1 contract awarded late 2004;
* Rebaselined early 2007;
* Releases 1.1 and 1,2 contract awarded, mid-2007;
* Rebaselined, end 2007.
Phase: production and deployment;
2004 plan: mid-2006 through mid-2011;
* IOC, late 2006;
* FOC, late 2011;
2007 plan: late 2007 through late 2013;
* IOC, early 2008;
* FOC, late 2013.
Source: GAO analysis of DON data.
[End of figure]
Figure 2: Navy ERP Milestones for Beginning Deployment:
[See PDF for image]
This figure is an illustration of the Navy ERP milestones for beginning
deployment, as follows:
Release: 1.0, Financial and Acquisition:
Milestone: NAVAIR: late 2007 (2007 plan);
Milestone: NAVSUP: late 2008 (2007 plan and 2008 plan);
Milestone: SPAWAR: mid-2008 (2007 plan); early 2010 (2008 plan);
Milestone: NAVSEA dor General Fund: early 2010 (2007 plan); early 2011
(2008 plan);
Milestone: NAVSEA for Working Capital Fund: early 2011 (2007 plan);
early 2012 (2008 plan).
Release: 1.1, Wholesale and Retail Supply;
Milestone: NAVSUP: late 2009 (2007 plan); mid-2010 (2008 plan).
Release 1.2, Intermediate-level Maintenance;
Milestone: Regional Maintenance Centers: early 2011 (2007 plan and 2008
plan).
Source: GAO analysis of DON data.
[End of figure]
Figure 3: Navy ERP Life Cycle Cost Estimates in Fiscal Years 2003,
2004, and 2007:
[See PDF for image]
This figure is a vertical bar graph depicting the following data:
Navy ERP Life Cycle Cost Estimates in Fiscal Years 2003, 2004, and
2007:
Fiscal year: 2003: $1.87 billion;
Fiscal year: 2004: $1.99 billion;
Fiscal year: 2007: $2.44 billion.
Source: GAO analysis of DON data.
[End of figure]
Use of IT Acquisition Management Controls Maximizes Chances for
Success:
IT acquisition management controls are tried and proven methods,
processes, techniques, and activities that organizations define and use
to minimize program risks and maximize the chances of a program's
success. Using these controls can result in better outcomes, including
cost savings, improved service and product quality, and a better return
on investment. For example, two software engineering analyses of nearly
200 systems acquisition projects indicate that teams using systems
acquisition controls that reflected best practices produced cost
savings of at least 11 percent over similar projects conducted by teams
that did not employ the kind of rigor and discipline embedded in these
practices.[Footnote 14] In addition, our research shows that these
controls are a significant factor in successful acquisition outcomes,
including increasing the likelihood that programs and projects will be
executed within cost and schedule estimates.[Footnote 15]
We and others have identified and promoted the use of a number of IT
acquisition management controls associated with acquiring IT
systems.[Footnote 16] See table 4 for a description of several of these
activities.
Table 4: Description of Business System IT Acquisition Management
Controls:
Business system acquisition practice: Architectural alignment; To
ensure that the acquisition is consistent with the organization's
enterprise architecture;
Description: Architectural alignment is the process for analyzing and
verifying that the proposed architecture of the system being acquired
is consistent with the enterprise architecture for the organization
acquiring the system. Such alignment is needed to ensure that acquired
systems can interoperate and are not unnecessarily duplicative of one
another.
Business system acquisition practice: Economic justification; To ensure
that system investments are economically justified;
Description: Economic justification is the process for ensuring that
acquisition decisions are based on reliable analyses of the proposed
investment's likely costs versus benefits over its useful life, as well
as an analysis of the risks associated with actually realizing the
acquisition's forecasted benefits for its estimated costs. Economic
justification is not a one-time event but rather is performed
throughout an acquisition's life cycle in order to permit informed
investment decision making.
Business system acquisition practice: Earned value management; To
ensure that actual progress against cost and schedule expectations is
being measured;
Description: EVM is a tool that integrates the technical, cost, and
schedule parameters of a contract and measures progress against them.
During the planning phase, an integrated program baseline is developed
by time phasing budget resources for defined work. As work is performed
and measured against the baseline, the corresponding budget value is
"earned." Using this earned value metric, cost and schedule variances,
as well as cost and time to complete estimates, can be determined and
analyzed.
Business system acquisition practice: Requirements management; To
ensure that requirements are traceable, verifiable, and controlled;
Description: Requirements management is the process for ensuring that
the requirements are traceable, verifiable, and controlled.
Traceability refers to the ability to follow a requirement from origin
to implementation and is critical to understanding the interconnections
and dependencies among the individual requirements and the impact when
a requirement is changed. Requirements management begins when the
solicitation's requirements are documented and ends when system
responsibility is transferred to the support organization.
Business system acquisition practice: Risk management; To ensure that
risks are identified and systematically mitigated;
Description: Risk management is the process for identifying potential
acquisition problems and taking appropriate steps to avoid their
becoming actual problems. Risk management occurs early and continuously
in the acquisition life cycle.
Source: GAO.
[End of table]
Prior GAO Reviews Have Identified IT Acquisition Management Control
Weaknesses on DOD Business System Investments:
We have previously reported[Footnote 17] that DOD has not effectively
managed a number of business system investments. Among other things,
our reviews of individual system investments have identified weaknesses
in such things as architectural alignment and informed investment
decision making, which are also the focus areas of the Fiscal Year 2005
Defense Authorization Act business system provisions.[Footnote 18] Our
reviews have also identified weaknesses in other system acquisition and
investment management areas--such as EVM, economic justification,
requirements management, and risk management.
In July 2007, we reported that the Army's approach for investing about
$5 billion over the next several years in its General Fund Enterprise
Business System, Global Combat Support System-Army Field/Tactical,
[Footnote 19] and Logistics Modernization Program did not include
alignment with the Army enterprise architecture or use of a portfolio-
based business system investment review process.[Footnote 20] Moreover,
we reported that the Army did not have reliable analyses, such as
economic analyses, to support its management of these programs. We
concluded that, until the Army adopts a business system investment
management approach that provides for reviewing groups of systems and
making enterprise decisions on how these groups will collectively
interoperate to provide a desired capability, it runs the risk of
investing significant resources in business systems that do not provide
the desired functionality and efficiency. Accordingly, we made
recommendations aimed at improving the department's efforts to achieve
total asset visibility and enhancing its efforts to improve its control
and accountability over business system investments. The department
agreed with our recommendations.
We also reported that DON had not, among other things, economically
justified its ongoing and planned investment in the Naval Tactical
Command Support System (NTCSS)[Footnote 21] and had not invested in
NTCSS within the context of a well-defined DOD or DON enterprise
architecture. In addition, we reported that DON had not effectively
performed key measurement, reporting, budgeting, and oversight
activities and had not adequately conducted requirements management and
testing activities. We concluded that, without this information, DON
could not determine whether NTCSS, as defined, and as being developed,
is the right solution to meet its strategic business and technological
needs. Accordingly, we recommended that the department develop the
analytical basis to determine if continued investment in NTCSS
represents prudent use of limited resources and to strengthen
management of the program, conditional upon a decision to proceed with
further investment in the program. The department largely agreed with
our recommendations.
In addition, we reported that the Army had not defined and developed
its Transportation Coordinators' Automated Information for Movements
System II (TC-AIMS II)--a joint services system with the goal of
helping to manage the movement of forces and equipment within the
United States and abroad--in the context of a DOD enterprise
architecture.[Footnote 22] In addition, we reported that the Army had
not economically justified the program on the basis of reliable
estimates of life cycle costs and benefits and had not effectively
implemented risk management. As a result, we concluded that the Army
did not know if its investment in TC-AIMS II, as planned, was warranted
or represented a prudent use of limited DOD resources. Accordingly, we
recommended that the department, among other things, develop the
analytical basis needed to determine if continued investment in TC-AIMS
II represents prudent use of limited defense resources. In response,
the department agreed with our recommendations and has since reduced
the program's scope by canceling future investments.
Furthermore, in 2005, we reported that DON had invested approximately
$1 billion in the four previously cited ERP pilots without marked
improvement in its day-to-day operations.[Footnote 23] More
specifically, we reported that the program office had not implemented
an EVM system. We also identified significant challenges and risks as
the project moved forward, such as developing and implementing system
interfaces, converting data from legacy systems into the ERP system,
meeting its estimated completion date of 2011 at an estimated cost of
$800 million, and achieving alignment with DOD's BEA. To address these
areas, we made recommendations that DOD improve oversight of Navy ERP,
including developing quantitative metrics to evaluate the program. DOD
generally agreed with our recommendations.
Implementation Of Key DOD And Related IT Acquisition Management
Controls On Navy ERP Varies:
DOD IT-related acquisition policies and guidance, along with other
relevant guidance, provide an acquisition management control framework
within which to manage business system programs like Navy ERP.
Effective implementation of this framework can minimize program risks
and better ensure that system investments are defined in a way to
optimally support mission operations and performance, as well as
deliver promised system capabilities and benefits on time and within
budget. To varying degrees of effectiveness, Navy ERP has been managed
in accordance with aspects of this framework. However, implementation
of key management controls has not been effective. Specifically,
* compliance with DOD's federated BEA has not been sufficiently
demonstrated;
* investment in the program has been economically justified on the
basis of expected life cycle benefits that will likely exceed estimated
life cycle costs, although some estimating limitations nevertheless
exist;
* earned value management has not been effectively implemented;
* an important requirements management activity has been effectively
implemented; and:
* a risk management process has been defined, but not effectively
implemented for all risks.
The reasons that program management and oversight officials cited for
why these key practices have not been sufficiently executed range from
limitations in the applicable DOD guidance and tools to the complexity
and challenges of managing and implementing a program of this size.
Each of these reasons is described in the applicable sections of this
report. By not effectively implementing all the above key IT
acquisition management functions, the program is at increased risk of
(1) not being defined in a way that best meets corporate mission needs
and enhances performance and (2) adding to the more than 2 years in
program schedule delays and about $570 million in program cost
increases experienced to date.
Navy ERP Compliance with DOD's Federated BEA Has Not Been Sufficiently
Demonstrated:
DOD and other guidance,[Footnote 24] recognize the importance of
investing in business systems within the context of an enterprise
architecture.[Footnote 25] Moreover, the Fiscal Year 2005 Defense
Authorization Act requires that defense business systems be compliant
with DOD's federated BEA.[Footnote 26] Our research and experience in
reviewing federal agencies show that not making investments within the
context of a well-defined enterprise architecture often results in
systems that are duplicative, are not well integrated, are
unnecessarily costly to interface and maintain, and do not optimally
support mission outcomes.[Footnote 27]
To its credit, the program office has followed DOD's BEA compliance
guidance.[Footnote 28] However, this guidance does not adequately
provide for addressing all relevant aspects of BEA compliance.
Moreover, DON's enterprise architecture, which is a major component of
DOD's federated BEA, as well as key aspects of DOD's corporate BEA,
have yet to be sufficiently defined to permit thorough compliance
determinations. In addition, current policies and guidance do not
require DON investments to comply with its enterprise architecture.
This means that the department does not have a sufficient basis for
knowing if Navy ERP has been defined to minimize overlap with and
duplication of other programs' functionality and maximize
interoperability among related programs. Each of these architecture
alignment limitations is discussed here:
* The program's compliance assessments did not include all relevant
architecture products. In particular, the program did not assess
compliance with the BEA's technical standards profile, which outlines,
for example, the standards governing how systems physically communicate
with other systems and how they secure data from unauthorized access.
This is particularly important because systems like Navy ERP need to
share information with other systems and, for these systems to
accomplish this effectively and efficiently, they need to employ common
standards. A case in point is the relationship between Navy ERP and the
Global Combat Support System--Marine Corps (GCSS-MC) program.[Footnote
29] Specifically, Navy ERP has identified 25 technical standards that
are not in the BEA technical standards profile, and GCSS-MC has
identified 13 technical standards that are not in the profile. Among
these non-BEA standards are program-unique information sharing
protocols, which could limit information sharing between Navy ERP and
GCSS-MC, and with other systems.
In addition, the program office did not assess compliance with the BEA
products that describe system-level characteristics. This is important
because doing so would create a body of information about programs that
could be used to identify common system components and services that
could potentially be shared by the programs, thus avoiding wasteful
duplication. For example, our analysis of Navy ERP program
documentation shows that it contains system functions related to
receiving goods, taking physical inventories, and returning goods,
which are also system functions cited by the GCSS-MC program. However,
because compliance with the BEA system products was not assessed, the
extent to which these functions are potentially duplicative was not
considered.
Furthermore, the program office did not assess compliance with BEA
system products that describe data exchanges among systems. As we
previously reported, establishing and using standard system interfaces
is a critical enabler to sharing data.[Footnote 30] For example, Navy
ERP program documentation indicates that it is to exchange inventory
order and status data with other systems. System interfaces are
important for understanding how information is to be exchanged between
systems. However, since the program was not assessed for compliance
with these products, it does not have the basis for understanding how
its approach to exchanging information differs from that of other
systems that it is to interface with. Compliance against each of these
BEA products was not assessed because DOD's compliance guidance does
not provide for doing so and, according to BTA officials, because some
BEA system products are not sufficiently defined. According to these
officials, BTA plans to continue to define these products as the BEA
evolves.
* The compliance assessment was not used to identify potential areas of
duplication across programs, which DOD has stated is an explicit goal
of its federated BEA and associated investment review and decision-
making processes. More specifically, even though the compliance
guidance provides for assessing programs' compliance with the BEA
product that defines DOD operational activities, and Navy ERP was
assessed for compliance with this product, the results were not used to
identify programs that support the same operational activities and
related business processes. Given that the federated BEA is intended to
identify and avoid not only duplications within DOD components, but
also between components, it is important that such commonality be
addressed. For example, BEA compliance assessments for Navy ERP and
GCSS-MC, as well as two Air Force programs (Defense Enterprise
Accounting and Management System--Air Force and the Air Force
Expeditionary Combat Support System) show that each program supports at
least six of the same BEA operational activities (e.g., conducting
physical inventory, delivering property and services) and three of
these four programs support at least 18 additional operational
activities (e.g., performing budgeting, managing receipt and
acceptance). However, since the potential overlap among these and other
programs was not assessed, these programs may be investing in
duplicative functionality. Reasons for this were that the compliance
guidance does not provide for such analyses to be conducted and
programs have not been granted access rights to use this functionality
in the compliance tool.
* The program's compliance assessment did not address compliance
against DON's enterprise architecture, which is one of the biggest
members of the federated BEA. This is particularly important given that
DOD's approach to fully satisfying the architecture requirements of the
Fiscal Year 2005 Defense Authorization Act is to develop and use a
federated architecture in which component architectures are to provide
the additional details needed to supplement the thin layer of corporate
policies, rules, and standards included in the corporate BEA.[Footnote
31] As we recently reported,[Footnote 32] DON's enterprise architecture
is not mature because, among other things, it is missing a sufficient
description of its current and future environments in terms of business
and information/data. However, certain aspects of an architecture
nevertheless exist and, according to DON CIO officials, these aspects
will be leveraged in its efforts to develop a complete enterprise
architecture. For example, the FORCEnet architecture is intended to
document Navy's technical infrastructure. Therefore, opportunities
exist for DON to assess its programs in relation to these architecture
products, and to understand where its programs are exposed to risks
because products do not exist, are not mature, or at odds with other
Navy programs. According to DOD officials, compliance with the DON
architecture was not assessed because DOD compliance policy is limited
to compliance with the corporate BEA, and a number of aspects of the
DON enterprise architecture have yet to be sufficiently developed.
* The program's compliance assessment was not validated by DON or DOD
investment oversight and decision-making authorities. More
specifically, neither the DOD IRBs nor the DBSMC, nor the BTA in
supporting both of these investment oversight and decision-making
authorities, reviewed the program's assessments. According to BTA
officials, under DOD's tiered approach to investment accountability,
these entities are not responsible for validating programs' compliance
assessments. Rather, this is a component responsibility, and thus they
rely on the military departments and defense agencies to validate the
assessments.
However, DON Office of the CIO, which is responsible for precertifying
investments as compliant before they are reviewed by the IRB, did not
validate any of the program's compliance assessments. According to
Office of the CIO officials, they rely on Functional Area Managers to
validate a program's compliance assessments. However, no DON policy or
guidance exists that describes how the Functional Area Managers should
conduct such validations. CIO officials stated that this is because
these authorities do not have the resources that they need to validate
the assessments, and because a number of aspects of the DON
architecture are not yet sufficiently developed.
Validation of program assessments is further complicated by the absence
of information captured in the assessment tool about what program
documentation or other source materials were used by the program office
in making its compliance determinations. Specifically, the tool is only
configured, and thus was only used, to capture the results of a
program's comparison of program architecture products to BEA products.
Thus, it was not used to capture the system products used in making
these determinations.
The limitations in existing BEA compliance-related policy and guidance,
the supporting compliance assessment tool, and the federated BEA, put
programs like Navy ERP at increased risk of being defined and
implemented in a way that does not sufficiently ensure interoperability
and avoid duplication and overlap. We recently completed a review
examining multiple programs' compliance with the federated BEA,
including Navy ERP, for the Senate Armed Services Committee,
Subcommittee on Readiness and Management Support. We addressed the
architectural compliance guidance, tool, and validation limitations as
part of this review.[Footnote 33]
Investment in Navy ERP Has Been Economically Justified, but Important
Estimating Practices Were Not Implemented:
The investment in Navy ERP has been economically justified on the basis
of expected life cycle benefits that far exceed estimated life cycle
costs. According to the program's benefit/cost analysis, Navy ERP will
produce about $8.6 billion in estimated benefits for an estimated cost
of about $2.4 billion over its 20-year life cycle. While these benefit
estimates were not subject to any analysis of how uncertainty in
assumptions and data could impact the estimates, as called for by
relevant guidance, our examination of key uncertainty variables, such
as the timing of legacy systems' retirement, showed that the savings
impact would be relatively minor. However, the reliability of the cost
estimate is limited because it was derived using several, but not all,
key estimating practices. For example, the estimate was not grounded in
a historical record of comparable data from similar programs and was
not based on a reliable schedule baseline, which are both necessary to
having a cost estimate that can be considered credible and accurate.
These practices were not employed for various reasons, including DOD's
lack of historical data from similar programs and the lack of an
integrated master schedule for the program that includes all releases.
Notwithstanding the fact that these limitations could materially
increase the $2.4 billion cost estimate, it is nevertheless unlikely
that these factors would increase the estimate to a level approaching
the program's benefit expectations. Therefore, we have no reason to
believe that Navy ERP will not produce a positive return on investment.
Benefit Estimates Are Sufficiently Reliable Despite Absence of
Uncertainty Analysis:
Forecasting expected benefits over the life of a program is a key
aspect of economically justifying an investment. The Office of
Management and Budget (OMB) guidance[Footnote 34] advocates
economically justifying investments on the basis of expected benefits,
costs, and risks. Since estimates of benefits can be uncertain because
of the imprecision in both the underlying data and modeling assumptions
used, the guidance also provides for analyzing and reporting the
effects of this uncertainty. By doing this, informed investment
decision making can occur through the life of the program, and a
baseline can be established against which to compare the accrual of
actual benefits from deployed system capabilities.
The most recent economic analysis, dated August 2007, includes
monetized benefit estimates for fiscal years 2004-2023, in three key
areas--about $2.7 billion in legacy system cost savings, $3.3 billion
in cost savings from inventory reductions, and $2.7 billion in cost
savings from labor productivity improvements. Collectively, these
benefits total about $8.6 billion.[Footnote 35]
The program office calculated expected benefits in terms of cost
savings, which is consistent with established practices and guidance.
For example, the program is to result in the retirement of 138 legacy
systems (including the 4 pilot systems) between fiscal years 2005 and
2015, and the yearly maintenance costs for a single system are expected
to be as high as about $39 million.[Footnote 36] According to relevant
guidance, cost saving estimates should also be analyzed in terms of how
uncertainty in assumptions and data could impact them. However, the
program office did not perform such uncertainty analysis. According to
program officials, uncertainty analysis is not warranted because they
have taken and continue to take steps to validate the assumptions and
the data, such as using the latest budget data associated with the
legacy systems, and monitoring changes to the systems' retirement
dates. While these steps are positive, they do not eliminate the need
for uncertainty analysis. Accordingly, we assessed key uncertainty
variables, such as the timing of the legacy systems' retirement, and
found that the retirement dates of some of these systems have changed
since the estimate was prepared, due to, among other things, schedule
delays in the program. While the inherent uncertainty in these dates
would reduce expected savings (e.g., only $11 million based on the 134
legacy systems that we examined),[Footnote 37] the reduction would be
small relative to a total benefit estimate of $8.6 billion.
The Cost Estimate Was Not Reliably Derived:
A reliable cost estimate is a key variable in calculating return on
investment, and it provides the basis for informed investment decision
making, realistic budget formulation and program resourcing, meaningful
progress measurement, proactive course correction, and accountability
for results. According to OMB,[Footnote 38] programs must maintain
current and well-documented cost estimates, and these estimates must
encompass the full life cycle of the program. OMB states that
generating reliable cost estimates is a critical function necessary to
support OMB's capital programming process. Without reliable estimates,
programs are at increased risk of experiencing cost overruns, missed
deadlines, and performance shortfalls.
Our research has identified a number of practices that are the basis of
effective program cost estimating. We have issued guidance that
associates these practices with four characteristics of a reliable cost
estimate.[Footnote 39] Specifically, these four characteristics are as
follows:
* Comprehensive: The cost estimates should include both government and
contractor costs over the program's full life cycle, from the inception
of the program through design, development, deployment, and operation
and maintenance to retirement. They should also provide an appropriate
level of detail to ensure that cost elements are neither omitted nor
double counted and include documentation of all cost-influencing ground
rules and assumptions.
* Well-documented: The cost estimates should have clearly defined
purposes and be supported by documented descriptions of key program or
system characteristics (e.g., relationships with other systems,
performance parameters). Additionally, they should capture in writing
such things as the source data used and their significance, the
calculations performed and their results, and the rationale for
choosing a particular estimating method or reference. Moreover, this
information should be captured in such a way that the data used to
derive the estimate can be traced back to, and verified against, their
sources. The final cost estimate should be reviewed and accepted by
management on the basis of confidence in the estimating process and the
estimate produced by the process.
* Accurate: The cost estimates should provide for results that are
unbiased and should not be overly conservative or optimistic (i.e.,
they should represent most likely costs). In addition, the estimates
should be updated regularly to reflect material changes in the program,
and steps should be taken to minimize mathematical mistakes and their
significance. Among other things, the estimate should be grounded in a
historical record of cost estimating and actual experiences on
comparable programs.
* Credible: The cost estimates should discuss any limitations in the
analysis performed due to uncertainty or biases surrounding data or
assumptions. Further, the estimates' derivation should provide for
varying any major assumptions and recalculating outcomes based on
sensitivity analyses, and their associated risks and inherent
uncertainty should be disclosed. Also, the estimates should be verified
based on cross-checks using other estimating methods and by comparing
the results with independent cost estimates.
The $2.4 billion life cycle cost estimate for Navy ERP reflects many of
the practices associated with a reliable cost estimate, including all
practices associated with being comprehensive and well-documented, and
several related to being accurate and credible (see table 5). However,
several important practices related to accuracy and credibility were
not performed. To be reliable, a cost estimate should be comprehensive,
well-documented, accurate, and credible.
Table 5: Summary of Cost Estimating Characteristics That the Cost
Estimate Satisfies:
Characteristic of reliable estimates: Comprehensive;
Satisfied?[A]: Yes.
Characteristic of reliable estimates: Well-documented;
Satisfied?[A]: Yes.
Characteristic of reliable estimates: Accurate;
Satisfied?[A]: Partially.
Characteristic of reliable estimates: Credible;
Satisfied?[A]: Partially.
Source: GAO analysis of DON data.
[A] "Yes" means that the program office provided documentation
demonstrating satisfaction of the criterion. "Partially" means that the
program office provided documentation demonstrating satisfaction of
part of the criterion. "No" means that the program office has yet to
provide documentation demonstrating satisfaction of the criterion.
[End of table]
The cost estimate is comprehensive because it includes both the
government and contractor costs specific to development, acquisition
(nondevelopment), implementation, and operations and support over the
program's 20-year life cycle. Moreover, the estimate clearly describes
how the various subelements are aggregated to produce amounts for each
cost category, thereby ensuring that all pertinent costs are included,
and no costs are double counted. Finally, cost-influencing ground rules
and assumptions, such as the program's schedule, labor rates, and
inflation rates are documented.
The cost estimate is also well-documented in that the purpose of the
cost estimate is clearly defined, and the technical baseline includes,
among other things, the hardware components and planned performance
parameters. Furthermore, the calculations and results used to derive
the estimate are documented, including descriptions of the
methodologies used and evidence of traceability back to source data
(e.g., vendor quotes, salary tables). Also, the cost estimate was
reviewed by the Naval Center for Cost Analysis and the Office of the
Secretary of Defense, Director for Program Analysis and Evaluation,
which adds a level of confidence in the estimating process and the
estimate produced.
However, the estimate lacks accuracy because not all important
practices related to this characteristic were performed. Specifically,
while the estimate is grounded in documented assumptions (e.g.,
hardware refreshment every 5 years) and periodically updated to reflect
changes to the program, it is not adequately grounded in historical
experience with comparable programs. While the program office did
leverage historical cost data from the Navy ERP pilot programs, program
officials told us that the level of cost accounting on these programs
did not provide sufficient data. As stated in our guide, estimates
should be based on historical records of cost and schedule estimates
from comparable programs, and such historical data should be maintained
and used for evaluation purposes and future estimates on comparable
programs. The importance of doing so is evident by the fact that Navy
ERP's cost estimate has increased by about $570 million since fiscal
year 2003, which program officials attributed to, among other things,
site implementation costs (e.g., training and converting legacy system
data) not included in the original cost estimate, schedule delays, and
the lack of historical data from similar ERP programs.
This lack of cost data for large-scale ERP programs is, in part, due to
DOD not having a standardized cost element structure for these programs
that can be used for capturing actual cost data, which is a
prerequisite to capturing and maintaining the kind of historical data
that can inform cost estimates on similar programs. This means that
programs like Navy ERP will not be able to ground their cost estimates
in actual costs from comparable programs. According to officials with
the Defense Cost and Resource Center,[Footnote 40] such cost element
structures are needed, along with a requirement for programs to report
on their costs, but approval and resources have yet to be gained for
either these structures or the reporting of their costs. We recently
completed work that addressed standardization of DOD's ERP cost element
structure and maintenance of a database for historical ERP cost data
for use on ERP programs.[Footnote 41]
Compounding the estimate's limited accuracy are limitations in its
credibility. Specifically, while the estimate satisfies some of the key
practices for a credible cost estimate (e.g., confirming key cost
drivers, performing sensitivity analyses,[Footnote 42] and having an
independent cost estimate prepared by the Naval Center for Cost
Analysis that was within 11 percent of the program's estimate), the
program lacks a reliable schedule baseline, which is a key component of
a reliable cost estimate because it serves as the basis for future work
to be performed. Other factors that limit confidence in the cost
estimate's accuracy are (1) past increases in the program's cost
estimate (as discussed earlier) and (2) trends in EVM data (as
discussed later). Taken together, the program's cost estimate is not
sufficiently credible and accurate and thus not reliable.
While important cost estimating practices were not implemented, it is
nevertheless unlikely that these limitations would materially increase
the $2.4 billion cost estimate to a level approaching the program's
$8.6 billion benefit expectations.
Earned Value Management Has Not Been Effectively Implemented:
Measuring and reporting progress against cost and schedule commitments
(i.e., baselines) is a vital element of effective program management.
EVM provides a proven means for measuring such progress and thereby
identifying potential cost overruns and schedule delays early, when
their impact can be minimized.
To its credit, the program has elected to implement program-level EVM,
which is a best practice that has rarely been implemented in the
federal government. In doing so, however, basic EVM activities have not
been executed. In particular, an integrated baseline review, which is
to verify that the program's cost and schedule are reasonable given the
program's scope of work and associated risks, has not been performed.
Moreover, other accepted industry standards have not been sufficiently
implemented, and surveillance of EVM implementation by an entity
independent of the program office has not occurred. Not performing
these important practices has contributed to the cost overruns and
lengthy schedule delays already experienced on Release 1.0, and they
will likely result in more. In fact, our analysis of the latest
estimate to complete just the budgeted development work for all three
releases, which is about $844 million, shows that this estimate will
most likely be exceeded by about $152 million.
The Program Has Elected to Implement Program-Level EVM:
As we previously reported,[Footnote 43] EVM offers many benefits when
done properly. In particular, it allows performance to be measured, and
it serves as an early warning system for deviations from plans. It,
therefore, enables a program office to mitigate the risks of cost and
schedule overruns. OMB policy recognizes the use of EVM as an important
part of program management and decision making.[Footnote 44]
Implementing EVM at the program level rather than just the contract
level is considered a best practice, and OMB recently began requiring
it to measure how well a program's approved cost, schedule, and
performance goals are being met. According to OMB, integrating
government and contractor cost, schedule, and performance status should
result in better program execution through more effective management.
In addition, integrated EVM data can be used to better justify budget
requests.
To minimize the risk associated with its decision to transition
responsibility for Navy ERP system integration from the contractor to
the government and to improve cost and schedule performance, the
program office elected in October 2006 to perform EVM at the program
level. We support the use of program-level EVM. However, if not
implemented effectively, this program-level approach will be of little
value.
An Integrated Baseline Review Has Not Been Performed:
A fundamental aspect of effective EVM is the development of a
performance measurement baseline (PMB), which represents the cumulative
value of planned work and serves as the baseline against which
variances are calculated. According to relevant best practice guidance,
a PMB consists of:
* a complete work breakdown structure;
* a complete integrated master schedule, and:
* accurate budgets for all planned work.[Footnote 45]
To validate the PMB, an integrated baseline review is performed to
obtain stakeholder agreement on the baseline. According to DOD guidance
and best practices, such a review should be held within 6 months of a
contract award and conducted on an as needed basis throughout the life
of a program to ensure that the baseline reflects (1) all tasks in the
statement of work, (2) adequate resources (staff and materials) to
complete the tasks, and (3) integration of the tasks into a well-
defined schedule. Further, the contract performance reports that are to
be used to monitor performance against the PMB should be validated
during the integrated baseline review.[Footnote 46]
The program office has satisfied some of the prerequisites for having a
reliable PMB, such as developing a work breakdown structure and
specifying the contract performance reports that are to be used to
monitor performance. However, it has not conducted an integrated
baseline review. Specifically, a review was not conducted for Release
1.0, even though the contract was finalized about 30 months ago
(January 2006). Also, while the review for Release 1.1 was recently
scheduled for August 2008, this is 8 months later than when such a
review should be held, according to DOD guidance and best practices.
This means that the reasonableness of the program's scope and schedule
relative to the program risks has not been assured, and has likely
been, and will likely continue to be a primary contributor to future
cost increases and schedule delays.
According to program officials, a review was not performed on the first
release because development of this release was largely complete by the
time the program office established the underlying capabilities needed
to perform program-level EVM. In addition, program officials stated
that an integrated baseline review has yet to be performed on the other
two releases because their priority has been on deploying and
stabilizing the first release. In our view, not assuring the validity
of the PMB precludes effective implementation of EVM. Until a review is
conducted, DOD will not have reasonable assurance that the program's
scope and schedule are achievable, and thus, additional cost and
schedule overruns are likely.
Industry EVM Standards Have Not Been Fully Implemented And
Independently Surveilled:
In 1996, DOD adopted industry EVM guidance[Footnote 47] that identifies
32 essential practices organized into five categories: (1)
organization; (2) planning, scheduling and budgeting; (3) accounting;
(4) analysis and management reports; and (5) revisions and data
maintenance. DOD requires that all programs' implementation of EVM
undergo a compliance audit against the 32 industry practices. In
addition, DOD policy and guidance[Footnote 48] state that independent
surveillance of EVM should occur over the life of the program to
guarantee the validity of the performance data and ensure that EVM is
being used effectively to manage cost, schedule, and technical
performance.
On Navy ERP, compliance with the 32 accepted industry practices has not
been verified, and surveillance of EVM by an independent entity has not
occurred. Therefore, the program does not have the required basis for
ensuring that EVM is being effectively implemented on Navy ERP.
According to program officials, surveillance was performed by NAVAIR
for Release 1.0. However, NAVAIR officials said that they did not
perform such surveillance because they did not receive the Release 1.0
cost performance data needed to do so. Program officials also stated
that DON's Center for Earned Value Management[Footnote 49] has
conducted an initial assessment of their EVM management system, and
that they intend to have the Center perform surveillance. However, they
did not have a plan for accomplishing this. Until compliance with the
standards is verified and continuous surveillance occurs, and
deviations are addressed, the program will likely continue to
experience cost overruns and schedule delays.
Estimated Schedule Baseline Was Not Derived In Accordance With All Key
Schedule Estimating Practices:
The success of any program depends in part on having a reliable
schedule of when the program's work activities will occur, how long
they will take, and how they are related to one another. As such, the
schedule not only provides a road map for the systematic execution of a
program but also provides the means by which to estimate costs, gauge
progress, identify and address potential problems, and promote
accountability. Our research has identified nine practices associated
with effective schedule estimating.[Footnote 50] These practices are
(1) capturing key activities, (2) sequencing key activities, (3)
establishing the duration of key activities, (4) assigning resources to
key activities, (5) integrating key activities horizontally and
vertically, (6) establishing the critical path for key activities, (7)
identifying "float time"[Footnote 51] between key activities, (8)
distributing reserves to high-risk activities, and (9) performing a
schedule risk analysis.
The program's estimated schedule was developed using some of these
practices, but several key practices were not fully employed that are
fundamental to having a schedule that provides a sufficiently reliable
basis for estimating costs, measuring progress and forecasting
slippages. On the positive side, the schedule for the first two
releases captures key activities and their durations and is integrated
horizontally and vertically, meaning that multiple teams executing
different aspects of the program can effectively work to the same
master schedule. Moreover, for these two releases, the program has
established float time between key activities and distributed schedule
reserve to high-risk activities. However, the program has not
adequately sequenced and assigned resources to key program activities.
Moreover, the estimated schedule for the first increment is not
grounded in an integrated master schedule of all the releases, and thus
the schedule for this increment does not reflect the program's critical
path of work that must be performed to achieve the target completion
date. Also, it does not reflect the results of a schedule-risk analysis
across all three releases with schedule reserve allocated to high-risk
activities because such risks were not examined. See table 6 for the
results of our analyses relative to each of the nine practices.
Table 6: Satisfaction of Schedule Estimating Key Practices:
Practice: Capturing key activities;
Explanation: The schedule should reflect all key activities (e.g.,
steps, events, outcomes) as defined in the program's work breakdown
structure, to include activities to be performed by both the government
and its contractors;
Satisfied?[A]: Yes;
GAO analysis: The program's estimated schedules for the first two
releases reflect both government and contractor activities, such as
development and testing of the software components, as well as key
milestones for measuring progress.
Practice: Sequencing key activities;
Explanation: The schedule should be planned so that it can meet
critical program dates. To meet this objective, key activities need to
be logically sequenced in the order that they are to be carried out. In
particular, activities that must be finished prior to the start of
other activities (i.e., predecessor activities), as well as activities
that cannot begin until other activities are completed (i.e., successor
activities) should be identified. By doing so, interdependencies among
activities that collectively lead to the accomplishment of events or
milestones can be established and used as a basis for guiding work and
measuring progress;
Satisfied?[A]: Partially;
GAO analysis: The schedules for the first two releases include the
logical sequencing of most, but not all, activities. For example, 234
of 2,445 activities in the Release 1.1 schedule were not sequenced
properly. By not identifying the correct interdependencies and properly
sequencing key activities, the schedule may not facilitate the
meaningful tracking of progress.
Practice: Establishing the duration of key activities;
Explanation: The schedule should realistically reflect how long each
activity will take to execute. In determining the duration of each
activity, the same rationale, historical data, and assumptions used for
cost estimating should be used for schedule estimating. Further, these
durations should be as short as possible, and they should have specific
start and end dates. Excessively long periods needed to execute an
activity should prompt further decomposition of the activity so that
shorter execution durations will result;
Satisfied?[A]: Yes;
GAO analysis: The schedules for the first two releases establish
realistic durations of key activities. For example, the schedule for
Release 1.1 is based on historical data from Release 1.0, which
provides a level of confidence that the durations are reasonable.
Practice: Assigning resources to key activities;
Explanation: The schedule should reflect what resources (i.e., labor,
material, and overhead) are needed to do the work, whether all required
resources will be available when they are needed, and whether any
funding or time constraints exist;
Satisfied?[A]: Partially;
GAO analysis: The schedules for the first two releases include the
allocation of resources, such as personnel, to activities, but it does
not reflect whether all resources will be available when they are
needed because the identified resources are shared across the three
releases. Restated, personnel are assigned to activities across
multiple releases, each of which is managed according to a separate
schedule. Therefore, if one of the schedules were to be delayed, the
other schedules that required the same resources would likely also be
delayed.
Practice: Integrating key activities horizontally and vertically;
Explanation: The schedule should be horizontally integrated, meaning
that it should link the products and outcomes associated with already
sequenced activities. These links are commonly referred to as
"handoffs" and serve to verify that activities are arranged in the
right order to achieve aggregated products or outcomes. The schedule
should also be vertically integrated, meaning that traceability exists
among varying levels of activities and supporting tasks and subtasks.
Such mapping or alignment among levels enables different groups to work
to the same master schedule;
Satisfied?[A]: Yes;
GAO analysis: The schedules for the first two releases are both
horizontally and vertically integrated, meaning that the activities
across the multiple teams are arranged in the right order to achieve
aggregated products or outcomes. In addition, traceability exists among
varying levels of activities, which allows multiple teams (i.e.,
development, testing) to work to the same master schedule.
Practice: Establishing the critical path for key activities;
Explanation: Using scheduling software, the critical path--the longest
duration path through the sequenced list of key activities--should be
identified. The establishment of a program's critical path is necessary
for examining the effects of any activity slipping along this path.
Potential problems that might occur along or near the critical path
should also be identified and reflected in the scheduling of the time
for high-risk activities (see next practice);
Satisfied?[A]: Partially;
GAO analysis: While the program has established the critical path for
the first two releases separately, it has not done so for the entire
first increment. This is because the program maintains separate
schedules for each release, and in doing so, cannot identify the
longest duration of tasks through the entire first increment. Without
doing so, the effects of any slippage along the critical path on future
releases cannot be determined.
Practice: Identifying "float time" between key activities;
Explanation: The schedule should identify float time--the time that a
predecessor activity can slip before the delay affects successor
activities--so that schedule flexibility can be determined. As a
general rule, activities along the critical path typically have the
least amount of float time;
Satisfied?[A]: Yes;
GAO analysis: The schedules for the first two releases identify float
time between key activities.
Practice: Distributing reserves to high-risk activities;
Explanation: The baseline schedule should include a buffer or a reserve
of extra time. Schedule reserve for contingencies should be calculated
by performing a schedule risk analysis (see next practice). As a
general rule, the reserve should be applied to high-risk activities,
which are typically found along the critical path;
Satisfied?[A]: Partially;
GAO analysis: The schedule allocates reserve for high-risk activities
on the critical path for Release 1.0. However, because the program has
not established a critical path for the first increment, it cannot
allocate reserve for the high-risk activities on the entire program's
critical path.
Practice: Schedule risk analysis should be performed;
Explanation: A schedule risk analysis should be performed using
statistical techniques to predict the level of confidence in meeting a
program's completion date. This analysis focuses not only on critical
path activities but also on activities near the critical path, since
they can potentially affect program status;
Satisfied?[A]: Partially;
GAO analysis: A schedule risk analysis on the entire program was not
performed. A schedule risk analysis has been done on Release 1.0, and
the program office plans to do one for Release 1.1. However, without
analyzing the risks associated with the program's entire schedule, the
program cannot determine the level of confidence in meeting the
program's completion date.
Source: GAO analysis of DON data.
[A] "Yes" means that the program provided documentation demonstrating
satisfaction of the practice. "Partially" means that the program
provided documentation demonstrating satisfaction of part of the
practice. "No" means that the program has yet to provide documentation
demonstrating satisfaction of the practice.
[End of table]
According to program documentation, they have plans to address the
logical sequencing of activities (to ensure that it reflects how work
is to be performed), but program officials stated that they do not plan
to combine all three releases into a single integrated master schedule
for the entire first increment of the program because doing so would
produce an overly complex and nonexecutable schedule involving as many
as 15,000 activities. However, our research of and experience in
evaluating major programs' use of EVM and integrated master schedules
show that while large, complex programs necessitate schedules involving
thousands of activities, successful programs ensure that their
schedules integrate these activities. In our view, not adequately
performing these practices does not allow the program to effectively
assign resources, identify the critical path, and perform a schedule
risk analysis that would allow it to understand, disclose, and
compensate for its schedule risks. This means that the program is not
well-positioned to understand progress and forecast its impact. To
illustrate, the program recently experienced delays in deploying its
first release at NAVAIR, which according to a recent operational test
and evaluation report[Footnote 52] has significantly affected the
schedule's critical path. These schedule impacts are because resources
supporting the deployment at NAVAIR began to shift to the next
scheduled deployment site and thus are no longer available to resolve
critical issues at NAVAIR. Since the schedule baseline is not
integrated across all releases, the impact of this delay on other
releases, and thus the program as a whole, cannot be readily
determined.
Trends in EVM Data Show Pattern of Cost Overruns and Schedule
Slippages:
Program data show a pattern of actual cost overruns and schedule delays
between January 2007 and May 2008. Moreover, our analysis of the data
supports a most likely program cost growth of about $152 million to
complete all three releases.
Differences from the PMB are measured in both cost and schedule
variances.[Footnote 53] Positive variances indicate that activities are
costing less or are completed ahead of schedule. Negative variances
indicate that activities are costing more or are falling behind
schedule. These cost and schedule variances can then be used in
forecasting the cost and time needed to complete the program.
Based on program-provided data for the first increment over a 17-month
period ending May 2008, the program has experienced negative cost
variances. Specifically, while these cost variances have fluctuated
during this period, they have consistently been negative. (See fig. 4.)
Moreover, our analysis of the cost to complete just the budgeted
development work (also known as the PMB) for all three releases, which
is about $844 million,[Footnote 54] will be exceeded by between about
$102 million and $316 million, with a most likely overrun of about $152
million.[Footnote 55] In contrast, the program office reports that the
overrun at completion will be $55 million but has yet to provide us
with documentation supporting this calculation. Moreover, our
calculation does not reflect the recent problems discovered during the
operational test and evaluation at NAVAIR and thus the overrun is
likely to be higher.
Figure 4: Cumulative Cost Variance for Navy ERP over a 17-Month Period:
[See PDF for image]
This figure is a line graph depicting the following data:
Date: January, 2007:
Cost variance: $2.567 million.
Date: February, 2007:
Cost variance: $0.387 million.
Date: March, 2007:
Cost variance: -$1.628 million.
Date: April, 2007:
Cost variance: $0.905 million.
Date: May, 2007:
Cost variance: -$4.047 million.
Date: June, 2007:
Cost variance: -$15.995 million.
Date: July, 2007:
Cost variance: -$16.36 million.
Date: August, 2007:
Cost variance: -$15.2 million.
Date: September, 2007:
Cost variance: -$10.544 million.
Date: October, 2007:
Cost variance: -$12.828 million.
Date: November, 2007:
Cost variance: -$24.932 million.
Date: December, 2007:
Cost variance: -$24.747 million.
Date: January, 2008:
Cost variance: -$19.044 million.
Date: February, 2008:
Cost variance: -$19.073 million.
Date: March, 2008:
Cost variance: -$15.685 million.
Date: April, 2008:
Cost variance: -$15.171 million.
Date: May, 2008:
Cost variance: -$17.289 million.
Source: GAO analysis based on Navy ERP data.
[End of figure]
During this same 17-month period, the program has experienced negative
schedule variances and, since January 2008, they have almost doubled
each month. Further, as of May 2008, the program had not completed
about $24 million in scheduled work. (See fig. 5.) An inability to meet
schedule performance is a frequent indication of future cost increases,
as more spending is often necessary to resolve schedule delays.
Figure 5: Cumulative Schedule Variance of the Navy ERP Program over a
17-Month Period:
[See PDF for image]
This figure is a line graph depicting the following data:
Date: January, 2007:
Cost variance: -$1.258 million.
Date: February, 2007:
Cost variance: -$1.655 million.
Date: March, 2007:
Cost variance: -$1.885 million.
Date: April, 2007:
Cost variance: -$1.84 million.
Date: May, 2007:
Cost variance: -$2.384 million.
Date: June, 2007:
Cost variance: -$4.98 million.
Date: July, 2007:
Cost variance: -$6.938 million.
Date: August, 2007:
Cost variance: -$7.646 million.
Date: September, 2007:
Cost variance: -$4.542 million.
Date: October, 2007:
Cost variance: -$4.715 million.
Date: November, 2007:
Cost variance: -$1.863 million.
Date: December, 2007:
Cost variance: -$2.548 million.
Date: January, 2008:
Cost variance: -$4.504 million.
Date: February, 2008:
Cost variance: -$8.069 million.
Date: March, 2008:
Cost variance: -$10.93 million.
Date: April, 2008:
Cost variance: -$17.703 million.
Date: May, 2008:
Cost variance: -$25.62 million.
Source: GAO analysis based on Navy ERP data.
[End of figure]
Because the program office has not performed important reliability
checks, such as EVM validation and integrated baseline reviews, as
discussed above, we cannot be certain that the PMB is reliable (i.e.,
reflects all the work to be done and has identified all the risks). As
a result, the overrun that we are forecasting could be higher.
[See PDF for image]
[End of figure]
By not executing basic EVM practices, the program has and will likely
continue to experience cost and schedule shortfalls. Until the program
office implements these important EVM practices, it will likely not be
able to track actual program costs and schedules close to estimates.
Important Requirements Management Activity Has Been Effectively
Implemented:
Well-defined and managed requirements are recognized by DOD guidance as
essential, and they can be viewed as a cornerstone of effective system
acquisition. One aspect of effective requirements management is
requirements traceability.[Footnote 56] By tracing requirements both
backward from system requirements to higher level business or
operational requirements and forward to system design specifications
and test plans, the chances of the deployed product satisfying
requirements is increased, and the ability to understand the impact of
any requirement changes and thus make informed decisions about such
changes, is enhanced.
The program office is effectively implementing requirements
traceability for its 1,733 Release 1.0 system requirements. To verify
this traceability, we randomly selected and analyzed 60 of the 1,733
system requirements and confirmed that 58 of the 60 were traceable both
backward to higher level requirements and forward to design
specifications and test results. The remaining 2 had been allocated to
the other releases, and thus we also confirmed the program's ability to
maintain traceability between product releases. In doing so, the
program utilized a tool called DOORS, which if implemented properly,
allows each requirement to be linked from its most conceptual
definition to its most detailed definition, as well as to design
specifications and test cases. In effect, the tool maintains the
linkages among requirement documents, design documents, and test cases
even if requirements change.
If DON continues to effectively implement requirements traceability, it
will increase the chances that system requirements will be met by the
deployed system.
A Risk Management Process Has Been Defined, but All Risks Have Not Been
Effectively Mitigated:
Proactively managing program risks is a key acquisition management
control and, if defined and implemented properly, it can increase the
chances of programs delivering promised capabilities and benefits on
time and within budget. To the program office's credit, it has defined
a risk management process that meets relevant guidance. However, it has
not effectively implemented the process for all identified risks. As a
result, these risks have not been proactively mitigated and either have
contributed to cost and schedule shortfalls, or could potentially
contribute to such shortfalls.
DOD acquisition management guidance, as well as other relevant guidance
advocates identifying facts and circumstances that can increase the
probability of an acquisition's failing to meet cost, schedule, and
performance commitments and then taking steps to reduce the probability
of their occurrence and impact.[Footnote 57] In brief, effective risk
management consists of: (1) establishing a written plan for managing
risks; (2) designating responsibility for risk management activities;
(3) encouraging program-wide participation in the identification and
mitigation of risks; (4) defining and implementing a process that
provides for the identification, analysis, and mitigation of risks; and
(5) examining the status of identified risks in program milestone
reviews.
The program office has developed a written plan for managing risks and
established a process that together provide for the above-cited risk
management practices. Moreover, it has largely followed its plan and
process as per the following examples:
* The program manager has been assigned overall responsibility for
managing risks and serves as the chair of the risk management board.
[Footnote 58] Also, a functional team lead (i.e., subject matter
expert) is assigned responsibility for analyzing and mitigating each
identified risk.
* Program-wide participation in the identification, analysis, and
mitigation of risks is encouraged. Specifically, a manager for each
release is responsible for providing risk management guidance to the
staff, which includes staff identification and analysis of risks. Also,
according to the program office's risk management plan, all program
personnel can submit a risk for approval. In addition, stakeholders
participate in risk management activities during acquisition milestone
reviews.
* The program office has identified and categorized individual risks.
As of June 2008, the risk database contained 15 active risks--3 high, 8
medium, and 4 low.[Footnote 59]
* Program risks are considered during program milestone reviews. For
example, during the program's critical design review, which is a key
event of the system development and demonstration phase, key risks
regarding implementing new business processes and legacy system changes
were discussed. Furthermore, the program manager receives a monthly
risk report that describes the status of program risks.
However, the program office has not consistently followed other aspects
of its process. In particular, it has not effectively implemented steps
for mitigating the risks associated with (1) converting data from
NAVAIR's legacy systems to run on Navy ERP and (2) positioning NAVAIR
for adopting the new business processes embedded in Navy ERP. As we
have previously reported, it is important for organizations that are to
operate and use commercial off-the-shelf software products, such as
Navy ERP, to proactively manage and position themselves for the
organizational impact of introducing functionality embedded in the
commercial products. If they do not, the organization's performance
will suffer.[Footnote 60]
To the program office's credit, it identified numerous risks associated
with data conversion and organizational change management and developed
and implemented strategies that were intended to mitigate these risks.
However, it closed these risks even though they were never effectively
mitigated, as evidenced by the results of recently completed DON
operational test and evaluation. According to the June 2008 operational
test and evaluation report for NAVAIR, significant problems relating to
both legacy system data conversion and adoption of new business
processes were experienced. The report states that these problems have
contributed to increases in the costs to operate the system, including
unexpected manual effort. It further states that these problems have
rendered the deployed version not operationally effective and that
deployment of the system to other sites should not occur until the
change management process has been analyzed and improved. It also
attributed the realization of the problems to the program office and
NAVAIR not having adequately engaged and communicated early with each
other to coordinate and resolve differences in organizational
perspectives and priorities and provide intensive pre-deployment
preparation and training. Program officials acknowledged these
shortcomings and attributed them to their limited authority over the
commands. In this regard, they have previously surfaced these risks
with department oversight and approval authorities, but actions were
not taken by these authorities that ensured that the risks were being
effectively mitigated.
Beyond not effectively mitigating these risks, the program office has
not ensured that all risks are captured in the risk inventory. For
example, the inventory does not include the risks described in this
report that are associated with not having adequately demonstrated the
program's alignment to the federated BEA and not having implemented
program-level EVM in a manner that reflects industry practices. This
means that these risks are not being disclosed or mitigated.
By not effectively addressing all risks associated with the program,
these risks can and have become problems that contribute to cost and
schedule shortfalls. Until all significant risks are proactively
addressed, to include ensuring that all associated mitigation steps are
implemented and that they accomplished their intended purpose, the
program will likely experience further problems at subsequent
deployment sites.
Conclusions:
DOD's success in delivering large-scale business systems, such as Navy
ERP, is in large part determined by the extent to which it employs the
kind of rigorous and disciplined IT management controls that are
reflected in department policies and related guidance. While
implementing these controls does not guarantee a successful program, it
does minimize a program's exposure to risk and thus the likelihood that
it will fall short of expectations. In the case of Navy ERP, living up
to expectations is important because the program is large, complex, and
critical to addressing the department's long-standing problems related
to financial transparency and asset visibility.
The effectiveness to which key IT management controls have been
implemented in Navy ERP varies, with one control and several aspects of
others being effectively implemented, and others less so. Moreover,
those controls that have not been effectively implemented have, in
part, contributed to the sizable cost and schedule shortfalls
experienced to date on the program. Unless this changes, more
shortfalls can be expected.
While the program office is primarily responsible for ensuring that
effective IT management controls are implemented, other oversight and
stakeholder organizations share responsibility. For example, even
though the program has not demonstrated its alignment with the
federated BEA, it nevertheless followed established DOD architecture
compliance guidance and used the related compliance assessment tool in
assessing and asserting its compliance. The root cause for not
demonstrating compliance thus is not traceable to the program office
but rather is due to, among other things, the limitations of the
compliance guidance and tool, and the program's oversight entities not
validating the compliance assessment and assertion. Also, the reason
that the program's cost estimate was not informed by the cost
experiences of other programs of the same size and scope is because DOD
does not have a standard ERP cost element structure and has not
maintained a historical database of costs for like programs to use. In
contrast, effective implementation of other management controls, such
as implementing EVM, requirements traceability, and risk management is
the responsibility of the program office.
All told, addressing the management control weaknesses requires the
combined efforts of the various organizations that share responsibility
for managing and overseeing the program. By doing so, the department
can better assure itself that Navy ERP will optimally support its
performance goals and will deliver promised capabilities and benefits
on time and within budget.
Because we recently completed work that more broadly addresses the
above cited architectural alignment and comparable program cost data
limitations, we are not making recommendations in this report for
addressing them.
Recommendations for Executive Action:
To strengthen Navy ERP management control and better provide for the
program's success, we are making the following recommendations:
* To improve the reliability of Navy ERP benefit estimates and cost
estimates, we recommend that the Secretary of Defense direct the
Secretary of the Navy, through the appropriate chain of command, to
ensure that future Navy ERP estimates include uncertainty analyses of
estimated benefits, reflect the risks associated with not having cost
data for comparable ERP programs, and are otherwise derived in full
accordance with the other key estimating practices, and economic
analysis practices discussed in this report.
* To enhance Navy ERP's use of EVM, we recommend that the Secretary of
Defense direct the Secretary of the Navy, through the appropriate chain
of command, to ensure that (1) an integrated baseline review on the
last two releases of the first increment is conducted, (2) compliance
against the 32 accepted industry EVM practices is verified, and (3) a
plan to have an independent organization perform surveillance of the
program's EVM system is developed and implemented.
* To increase the quality of the program's integrated master schedule,
we recommend that the Secretary of Defense direct the Secretary of the
Navy, through the appropriate chain of command, to ensure that the
schedule (1) includes the logical sequencing of all activities, (2)
reflects whether all required resources will be available when needed,
(3) defines a critical path that integrates all three releases, (4)
allocates reserve for the high-risk activities on the entire program's
critical path, and (5) incorporates the results of a schedule risk
analysis for all three releases and recalculates program cost and
schedule variances to more accurately determine a most likely cost and
schedule overrun.
* To improve Navy ERP's management of program risks, we recommend that
the Secretary of Defense direct the Secretary of the Navy, through the
appropriate chain of command, to ensure that (1) the plans for
mitigating the risks associated with converting data from legacy
systems to Navy ERP and positioning the commands for adopting the new
business processes embedded in the Navy ERP are re-evaluated in light
of the recent experience with NAVAIR and adjusted accordingly, (2) the
status and results of these and other mitigation plans' implementation
are periodically reported to program oversight and approval
authorities, (3) these authorities ensure that those entities
responsible for implementing these strategies are held accountable for
doing so, and (4) each of the risks discussed in this report are
included in the program's inventory of active risks and managed
accordingly.
Agency Comments and Our Evaluation:
In written comments on a draft of this report, signed by the Deputy
Under Secretary of Defense (Business Transformation) and reprinted in
appendix II, DOD stated that it concurred with two of our four
recommendations and partially concurred with the remaining two.
Further, it stated that it has taken steps to address some of our
recommendations, adding that it is committed to implementing
recommendations that contribute to the program's success. The
department's comments relative to both of the recommendations that it
partially concurred with, as well as additional comments, are discussed
below.
For our recommendation associated with improving the program's benefit
and cost estimates, DOD concurred with two of the recommendation's
three parts, but it did not concur with one part--ensuring that future
cost estimates reflect the risk of not having cost data for comparable
programs. While acknowledging that the program had limited cost data
from comparable programs on which to base its cost estimate, DOD stated
that an uncertainty analysis had been applied to the estimate to
account for the risk associated with not having such data. The
department further stated that actual experience on the program will
continue to be used to refine the program's cost estimating
methodology. While we support DOD's stated commitment to using actual
program cost experience in deriving future estimates, we do not agree
that the latest estimate accounted for the risk of not having cost data
from comparable programs. We examined the uncertainty analysis as part
of our review, and found that it did not recognize this risk. Moreover,
DOD's comments offered no new evidence to the contrary.
For our recommendation associated with improving the program's schedule
estimating, DOD concurred with four of the recommendation's five parts,
and partially concurred with one part--ensuring that the schedule
defines a critical path that integrates all releases. In taking this
position, the department stated that a critical path has been
established for each release rather than across all three releases, and
it attributes this to the size and complexity of the program. We do not
take issue with either of these statements, as they are already
recognized in our report. However, DOD offers no new information in its
comments. Further, our report also recognizes that to be successful,
large and complex programs that involve thousands of activities need to
ensure that their schedules integrate these activities. In this regard,
we support the department's commitment to explore the feasibility of
implementing this part of our recommendation.
In addition, while stating that it concurred with all parts of our
recommendation associated with improving the program's use of EVM, DOD
nevertheless provided additional comments as justification for having
not conducted an integrated baseline review on Release 1.0.
Specifically, it stated that when it rebaselined this release in
December 2006, the release's development activities were essentially
complete and the release was in the latter stages of testing. Further,
it stated that the risks associated with the Release 1.0 schedule were
assessed 3 months after this rebaselining, and these risks were
successfully mitigated. To support this statement, it said that Release
1.0 achieved its "Go-Live" as scheduled at NAVAIR. We do not agree with
these comments for several reasons. First, at the time of the
rebaselining, about 9 months of scheduled Release 1.0 development
remained, and thus the release was far from complete. Moreover, the
significance of the amount of work that remained, and still remains
today on Release 1.0 is acknowledged in DOD's own comment that the
scheduled integrated baseline review for Release 1.1 will also include
remaining Release 1.0 work. Second, the Release 1.0 contract was
awarded in January 2006, and DOD's own guidance requires that an
integrated baseline review be conducted within 6 months of a contract's
award. Third, although DOD states that the program achieved "Go-Live"
as scheduled on October 1, 2007, the program achieved initial
operational capability 7 months later than established in the December
2006 baseline.
In addition to these comments, the department also described actions
under way or planned to address our recommendations. We support the
actions described, as they are consistent with the intent of our
recommendations. If fully and properly implemented, these actions will
go a long way in addressing the management control weaknesses that our
recommendations are aimed at correcting.
We are sending copies of this report to interested congressional
committees; the Director, Office of Management and Budget; the
Congressional Budget Office; the Secretary of Defense; and the
Department of Defense Office of the Inspector General. We also will
make copies available to others upon request. In addition, the report
will be available at no charge on the GAO Web site at [hyperlink,
http://www.gao.gov].
If you or your staff have any questions about this report, please
contact me at (202) 512-3439 or hiter@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. GAO staff who made major contributions to
this report are listed in appendix III.
Signed by:
Randolph C. Hite:
Director, Information Technology Architecture and Systems Issues:
[End of section]
Appendix I: Objective, Scope, and Methodology:
[End of section]
Our objective was to determine whether the Department of the Navy is
effectively implementing information technology management controls on
the Navy Enterprise Resource Planning (Navy ERP) program. To accomplish
this, we focused on the first increment of Navy ERP and the following
management areas (1) architectural alignment, (2) economic
justification, (3) earned value management (EVM), (4) requirements
management, and (5) risk management.
* To determine whether Navy ERP was aligned with the Department of
Defense's (DOD) federated business enterprise architecture (BEA), we
reviewed the program's BEA compliance assessments and system
architecture products, as well as Versions 4.0, 4.1, and 5.0 of the
BEA, and compared them with the BEA compliance requirements described
in the Fiscal Year 2005 Defense Authorization Act[Footnote 61] and
DOD's BEA compliance guidance, and we evaluated the extent to which the
compliance assessments addressed all relevant BEA products. We also
determined the extent to which the program-level architecture
documentation supported the BEA compliance assessments. We obtained
documentation, such as the BEA compliance assessments from the Navy ERP
and Global Combat Support System--Marine Corps programs, as well as the
Air Force's Defense Enterprise Accounting and Management System and Air
Force Expeditionary Combat Support System programs. We then compared
these assessments to identify potential redundancies or opportunities
for reuse and determined if the compliance assessments examined
duplication across programs, and if the tool that supports these
assessments is being used to identify such duplication. In doing so, we
interviewed program officials and officials from the Department of the
Navy, Office of the Chief Information Officer and reviewed recent GAO
reports to determine the extent to which the programs were assessed for
compliance against the Department of the Navy enterprise architecture.
We also interviewed program officials and officials from the Business
Transformation Agency and the Department of the Navy, including the
logistics Functional Area Manager, and obtained guidance documentation
from these officials to determine the extent to which the compliance
assessments were subject to oversight or validation.
* To determine whether the program had economically justified its
investment in Navy ERP, we reviewed the latest economic analysis to
determine the basis for the cost and benefit estimates. This included
evaluating the analysis against Office of Management and Budget
guidance and GAO's Cost Assessment Guide.[Footnote 62] In doing so, we
interviewed cognizant program officials, including the program manager
and cost analysis team, regarding their respective roles,
responsibilities, and actual efforts in developing and/or reviewing the
economic analysis. We also interviewed officials at the Office of
Program Analysis and Evaluation and Naval Center for Cost Analysis as
to their respective roles, responsibilities, and actual efforts in
developing and/or reviewing the economic analysis. We did not verify
the validity of the source data used to calculate estimated benefits,
such as those data used to determine the yearly costs associated with
legacy systems planned for retirement.
* To determine the extent to which the program had effectively
implemented EVM, we reviewed relevant documentation, such as contract
performance reports, acquisition program baselines, performance
measurement baseline, and schedule estimates and compared them with DOD
policies and guidance.[Footnote 63] To identify trends that could
affect the program baseline in the future, we assessed cost and
schedule performance and, in doing so, we applied earned value analysis
techniques[Footnote 64] to data from contract performance reports. We
compared the cost of work completed with the budgeted costs for
scheduled work over a 17-month period, from January 2007 to May 2008,
to show trends in cost and schedule performance. We also used data from
the reports to estimate the likely costs at completion of the program
through established earned value formulas. This resulted in three
different values, with the middle value being the most likely. We
checked EVM data to see if there were any mathematical errors or
inconsistencies that would lead to the data being unreliable. We
interviewed cognizant officials from the Naval Air Systems Command and
program officials to determine whether the program had conducted an
integrated baseline review, whether the EVM system had been validated
against industry guidelines,[Footnote 65] and to better understand the
anomalies in the EVM data and determine what outside surveillance was
being done to ensure that the industry standards are being met. We also
reviewed the program's schedule estimates and compared them with
relevant best practices[Footnote 66] to determine the extent to which
they reflect key estimating practices that are fundamental to having a
reliable schedule. In doing so, we interviewed cognizant program
officials to discuss their use of best practices in creating the
program's current schedule.
* To determine the extent to which the program has effectively
implemented requirements management, we reviewed relevant program
documentation, such as the program management plan and baseline list of
requirements. To determine the extent to which the program has
maintained traceability backward to high-level business operation
requirements and system requirements, and forward to system design
specifications, and test plans, we randomly selected 60 program
requirements and traced them both backward and forward. This sample was
designed with a 5 percent tolerable error rate at the 95 percent level
of confidence so that, if we found 0 problems in our sample, we could
conclude statistically that the error rate was less than 5 percent. In
addition, we interviewed program officials involved in the requirements
management process to discuss their roles and responsibilities for
managing requirements.
* To determine the extent to which the program implemented risk
management, we reviewed relevant risk management documentation, such as
the program's risk management plan and risk database reports
demonstrating the status of the program's major risks and compared the
program office's activities with DOD acquisition management guidance
and related industry practices.[Footnote 67] We also reviewed the
program's mitigation process with respect to key risks to determine the
extent to which these risks were effectively managed. In doing so, we
interviewed cognizant program officials, such as the program manager
and risk manager, to discuss their roles and responsibilities and
obtain clarification on the program's approach to managing risks
associated with acquiring and implementing Navy ERP.
We conducted this performance audit at DOD offices in the Washington,
D.C., metropolitan area and Annapolis, Md., from June 2007 to September
2008, in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objective. We
believe that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objective.
[End of section]
Appendix II: Comments from the Department of Defense:
Office Of The Under Secretary Of Defense:
Acquisition Technology And Logistics:
3000 Defense Pentagon:
Washington, DC 20301-3000:
August 19, 2008:
Mr. Randolph C. Hite:
Director, Information Technology Architecture and Systems Issues:
U.S. Government Accountability Office:
441 G Street, N.W.
Washington, DC 20548:
Dear Mr. Hite:
This is the Department of Defense (Dot)) response to the GAO draft
report GAO-08-896, "Defense Business Systems Modernization: Important
Management Controls Being Implemented on Major Navy Program, But
Improvements Needed in Key Areas," dated July 18, 2008 (GAO Code
310659). Detailed comments on the report recommendations are enclosed.
The Department concurred with two recommendations and partially
concurred with the remaining two. The Department has already taken
steps to address some of GAO's recommendations and the Navy Enterprise
Resource Planning (ERP) program office is committed to implementing
recommendations that will contribute to the program's success.
We appreciate the GAO's input on the Department's progress with its
business transformation efforts and continue to value our partnership.
Sincerely,
Signed by:
Paul A. Brinkley:
Deputy Under Secretary of Defense (Business Transformation):
Enclosure: As stated:
GAO Draft Report Dated July 18, 2008:
GAO-08-896 (GAO Code 310659):
"DOD Business Systems Modernization: Important Management Controls
Being Implemented On Major Navy Program, But Improvements Needed In Key
Areas"
Department Of Defense Comments To The GAO Recommendations:
Recommendation 1: The GAO recommends that the Secretary of Defense
direct the Secretary of the Navy, through the appropriate chain of
command, to ensure that future Navy Enterprise Resource Planning (ERP)
estimates include uncertainty analyses of estimated benefits, reflect
the risk associated with not having cost data for comparable ERP
programs, and are otherwise derived in full accordance with the other
key estimating practices, and economic analysis practices discussed in
this report. (Page 52/GAO Draft Report)
DOD Response: Partially Concur. Future Navy ERP estimates will include
uncertainty analyses of estimated benefits and be derived in full
accordance with key estimating and economic analysis practices. In
future estimates, uncertainty surrounding the assumptions driving the
benefit estimate will have probability distributions applied. The point
estimate reported will be generated using the expected values of the
probability distributions.
The Department does not concur with the second element of
Recommendation 1, that future Navy ERP estimates reflect the risk
associated with not having cost data for comparable ERP programs. At
Milestone .A/B, the program estimate was based on commercial best
practices and standards used to implement System Application Program
(SAP) functionality as well as engineering inputs and expertise
developed during the Navy's pilot ERP programs. This approach was
considered the best approach given the limited comparable enterprise
program system implementation data currently available in DoD.
Furthermore, the industry standards and the functional scope definition
had uncertainty analysis applied to account for the risk associated
with not having comparable DoD programmatic data. As the program began
execution, the methodology was and continues to be refined as actual
experience is gained by the Navy ERP program.
Recommendation 2: The GAO recommends that the Secretary of Defense
direct the Secretary of the Navy, through the appropriate chain of
command, to ensure that: (1) an integrated baseline review on the last
two releases of the first increment is conducted: (2) compliance
against the 32 accepted industry earned value management (EVM)
practices is verified: and (3) a plan to have an independent
organization perform surveillance of the program's EVM system is
developed and implemented. (Page 52/GAO Draft Report)
DOD Response: Concur. The Department appreciates the importance of
Earned Value Management (EVM) as an effective management tool that
promotes efficient management of programs such as Navy ERP. Navy ERP
has been implementing "Program-Level" EVM since 2006. During that time,
the Navy ERP Program Office has received detailed guidance and advice
from various government and industry groups since Program-Level EVM is
rare and requires tailoring to achieve compliance with best practices.
A key part of this implementation is the Integrated Baseline Review
(IBR), which has been scheduled for Navy ERP Release 1.1 in August
2008. Although this IBR will focus on Release 1.1, it will also include
the remaining Release 1.0 deployments. The IBR will focus on the health
of the baseline, assessing schedule and associated resourcing. Navy ERP
will also conduct IBR on future releases.
An IBR was not conducted on Release 1.0 because once the rebaselining
was approved in December 2006, development was essentially complete and
the release was in the latter stages of testing. However, a Schedule
Risk Assessment (SRA) of Release 1.0 was conducted in March 2007. The
main finding was that the Integrated Master Schedule (IMS) was high
risk for achieving Go-Live due to the lack of integration of the Navy
ERP schedule with the schedule of Naval Air System Command (NAVAIR),
the customer. Navy ERP took corrective measures to mitigate this risk
by developing a cut-over schedule that integrated with NAVAIR's
schedule. Navy ERP has made it a requirement to incorporate customer
schedules in all future releases. The corrective measures successfully
mitigated the risk associated with 1.0 as indicated by the Program
achieving Go-Live as scheduled.
Navy ERP follows the direction of the Assistant Secretary of the Navy
for Research. Development and Acquisition (ASN(RDA)) to work with the
Navy Center for Earned Value Management (CEVM) to validate that
management processes meet the intent of the 32 guidelines described in
the American National Standards Institute/Electronic Industries
Alliance (ANSI/EIA) standard ANSI/EIA-748. CEVM will also perform
ongoing surveillance of Navy ERP processes and EVM data, addressing the
last element of GAO's recommendation. Additionally, as of August 2008,
Navy ERP is complying with the Under Secretary of Defense for
Acquisition, Technology, and Logistics (AT&L) memo issued July 11, 2007
on the implementation of the Central Repository System. This memo
requires that all Acquisition Category I programs submit monthly cost
and schedule performance reports via the Defense Cost and Resource
Center (DCARC) Earned Value Repository.
Recommendation 3: The GAO recommends that the Secretary of Defense
direct the Secretary of the Navy, through the appropriate chain of
command, to ensure that the schedule: (1) includes the logical
sequencing of all activities; (2) reflects whether all required
resources will be available when needed; (3) defines a critical path
that integrates all three releases; (4) allocates reserve for the high-
risk activities on the entire program's critical path; and (5)
incorporates the results of a schedule risk analysis for all three
releases, and recalculates program cost and schedule variances to more
accurately determine a most likely cost and schedule overrun. (Page
52/GAO Draft Report)
DOD Response: Partially Concur. The Department's EVM policy recognizes
that the preparation of an Integrated Master Schedule (IMS) is a best
practice. The Department concurs with the following elements of the
recommendation and the Navy ERP Program Office will ensure that the
schedule:
* Includes the logical sequencing of all activities: Navy ERP has
implemented a logically sequenced schedule for Release 1.1 of all
activities based on experience gained from Navy ERP Release 1.0.
Improvements have been made and the schedule is continuously monitored
through weekly schedule metrics.
* Reflects whether all required resources will be available when
needed: The Navy ERP IMS has a resource-loaded, time-phased schedule
for each release that identifies resource allocation and is aligned
with the Program's approved staffing plan.
* Allocates reserve for the high-risk activities on the entire
program's critical path: Navy ERP has actively begun refining the risk
identification process to incorporate risk and potential impact to the
program schedule utilizing schedule inputs. As this process matures,
the result will be used to assist in identify adequate resources
required in managing integrated program activities across all three
releases.
* Incorporates the results of a schedule risk analysis (SRA) for all
releases: The Results of the SRA for Release 1.1 will be compared to
the results of the SRA for Release 1.0 and incorporated into the
variance analysis for cost and schedule to avoid any potential
overruns. Navy ERP is implementing an estimate at completion (EAC)
process for quarterly updates beginning in Fiscal Year 2009.
The Department partially concurs with the third element of the
recommendation, that the schedule defines a critical path that
integrates all three releases. Navy ERP utilizes a critical path for
each release. This approach recognizes the size and complexity of the
Navy's implementation releases and still allows the Navy to remain
focused on the true dependencies the individual releases have with one
another. The Program Office will review this recommendation with Navy
leadership to determine feasibility for Navy [RP.
Recommendation 4: The GAO recommends that the Secretary of Defense
direct the Secretary of the Navy, through the appropriate chain of
command, to ensure that: (1) the plans for mitigating the risks
associated with converting data from legacy systems to Navy ERP and
positioning the commands for adopting the new business processes
embedded in the Navy ERP are re-evaluated in light of the recent
experience with Naval Air Systems Command (NAVAIR) and adjust
accordingly; (2) the status and results of these and other mitigation
plans' implementation are periodically reported to program oversight
and approval authorities; (3) these authorities ensure that those
entities responsible for implementing these strategies are held
accountable for doing so; and (4) each of the risks discussed in this
report are included in the program's inventory of active risks, and
managed accordingly. (Page 53/GAO Draft Report)
DOD Response: Concur. Navy ERP incorporated lessons learned from NAVAIR
into the Navy ERP Program Command Implementation Guidance. The
guidance, a copy of which is attached for reference, incorporates key
lessons learned from prior deployments and provides a detailed overview
of the implementation process and key information required in
structuring a Command's implementation team and activities needed for
success. The Navy ERP Program Command Implementation Guidance also
identifies critical success factors based on extensive commercial and
Navy ERP implementation experience and provides a time-phased checklist
to help focus a Command's resources on the right actions, at the right
time. It has been reviewed by the Navy System Command Commanders and is
being used by the Navy Supply System Command (NAVSUP) in preparation
for the next deployment of Navy ERP Release 1.0.
The Readiness Assessment checklist in the Navy ERP Program Command
Implementation Guidance helps to identify risks, and through the Navy
ERP Risk Program process, elevate the risk appropriately to the right
level of management to facilitate mitigation planning. In addition to
those risks identified by the program, Navy ERP will add the risks
identified by GAO to its risk inventory. Risks and mitigation plans are
formally captured by the Navy ERP Risk Management Board in its Risk
Radar tracking tool.
Furthermore, in light of the experience learned with deployment to
NAVAIR, Navy ERP has completed a full review and reevaluation of all
active risks and corresponding mitigation plans. This review included
two areas noted by GAO in the report: (I) conversion of data from
legacy systems to Navy ERP and (2) transformation change management to
position Commands to adopt Navy ERP.
Since Navy's ERP's deployment of Release 1.0 to NAVAIR, the enterprise
governance structure that reviews program risks has evolved. Governance
now includes the Navy Component Acquisition Executive's review process,
the Navy ERP Senior Integration Board, and Enterprise Risk Assessment
Methodology (ERAM) assessments. The Navy will work through these
governance bodies and with other appropriate oversight and approval
authorities to determine additional reporting mechanisms for risks with
associated mitigation plans, to ensure commitment and accountability
across the Navy Enterprise. Direction received from these authorities
with respect to risk and risk management will be incorporated into the
Navy ERP Risk Management Program. Through the Navy and Office of the
Secretary of Defense (OSD) governance bodies. including the Milestone
Decision Authority (MDA), Deputy Under Secretary of Defense (Business
Transformation), as well as Navy Enterprise Senior Integration Board
(NESIB), the Navy ERP Program Manager is held accountable for overall
program performance, including implementing risk mitigation strategies.
This aligns to Department of Defense Directive 5000.1, paragraph 3.5,
which states that the Program Manager is accountable for credible cost,
schedule, and performance reporting to the MDA.
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Randolph C. Hite, (202) 512-3439, or hiter@gao.gov:
Staff Acknowledgments:
In addition to the individual named above, key contributors to this
report were Neelaxi Lakhmani, Assistant Director; Monica Anatalio;
Harold Brumm; Neil Doherty; Cheryl Dottermusch; Nancy Glover; Mustafa
Hassan; Michael Holland; Ethan Iczkovitz; Anh Le; Josh Leiling; Emily
Longcore; Lee McCracken; Madhav Panwar; Karen Richey; Melissa
Schermerhorn; Karl Seifert; Sushmita Srikanth; Jonathan Ticehurst; and
Adam Vodraska.
[End of section]
Footnotes:
[1] Business systems are information systems, including financial and
nonfinancial systems, that support DOD business operations, such as
civilian personnel, finance, health, logistics, military personnel,
procurement, and transportation.
[2] GAO, High-Risk Series: An Update, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-07-310] (Washington, D.C.:
January 2007).
[3] The Department of the Navy is a major component of DOD, consisting
of two uniformed services: the Navy and the Marine Corps.
[4] This amount is the difference between the September 2007 estimated
life cycle cost of $2,445.0 million and the September 2003 estimated
life cycle cost of $1,873.1 million.
[5] Program-level means that EVM covers all aspects of the program,
regardless of whether they are performed by the government or the
contractor. In contrast, EVM has historically been implemented on a
contract-by-contract basis, and has not included government performed
work.
[6] These commands are Naval Air, Naval Sea, Space and Naval Warfare,
and Naval Supply Systems.
[7] The defense acquisition system is a framework-based approach that
is intended to translate mission needs and requirements into stable,
affordable, and well-managed acquisition programs.
[8] Congressional defense committees have established reprogramming
guidelines, including setting dollar thresholds that direct DOD to seek
the prior approval of the committees before executing the reprogramming
of appropriated funds. In accordance with these guidelines, DOD's
financial management regulation provides that DOD does not need
congressional committee approval when the amount to be reprogrammed
falls below certain thresholds (referred to as a "below-threshold
reprogramming"). As relevant here, as of fiscal year 2005, the
threshold is $10 million or 20 percent of the program's appropriation,
whichever is less. To fund shortfalls in Navy ERP, DON plans to
reprogram amounts below this threshold from other programs.
[9] FOC means that the system has been successfully deployed in all
intended locations.
[10] This 2003 estimate, which was prepared to assist in budget
development and support the Milestone A/B approval, was for
development, deployment, and sustainment costs in fiscal years 2003
through 2021.
[11] According to DOD's acquisition guidebook, an Acquisition Program
Baseline is a program manager's estimated cost, schedule, and
performance goals. Goals consist of objective values, which represent
what the user desires and expects, and threshold values, which
represent acceptable limits. When the program manager determines a
current cost, schedule or performance threshold value will not be
achieved, the MDA must be notified, and a new baseline developed,
reviewed by decision makers, and, if the program is to continue,
approved by the MDA.
[12] According to the August 2004 Acquisition Program Baseline, this
estimate is for acquisition, operations, and support for fiscal years
2004 through 2021.
[13] According to the September 2007 Acquisition Program Baseline, this
estimate is for acquisition, operations, and support for fiscal years
2004 through 2023.
[14] Donald E. Harter, Mayuram S. Krishnan, and Sandra A. Slaughter,
"Effects of Process Maturity on Quality, Cycle Time, and Effort in
Software Product Development," Management Science, 46, no. 4 (2000);
and Bradford K. Clark, "Quantifying the Effects of Process Improvement
on Effort," IEEE Software (November/December 2000).
[15] GAO, Information Technology: DOD's Acquisition Policies and
Guidance Need to Incorporate Additional Best Practices and Controls,
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-722] (Washington,
D.C.: July 30, 2004).
[16] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-722].
[17] See, for example, GAO, DOD Business Transformation: Lack of an
Integrated Strategy Puts the Army's Asset Visibility System Investments
at Risk, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-860]
(Washington, D.C.: July 27, 2007); GAO, Information Technology: DOD
Needs to Ensure That Navy Marine Corps Intranet Program Is Meeting
Goals and Satisfying Customers, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-07-51] (Washington, D.C.: Dec. 8, 2006); GAO, Defense
Travel System: Reported Savings Questionable and Implementation
Challenges Remain, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-
980] (Washington, D.C.: Sept. 26, 2006); GAO, DOD Systems
Modernization: Uncertain Joint Use and Marginal Expected Value of
Military Asset Deployment System Warrant Reassessment of Planned
Investment, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-171]
(Washington, D.C.: Dec. 15, 2005); and GAO, DOD Systems Modernization:
Planned Investment in the Navy Tactical Command Support System Needs to
Be Reassessed, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-
215] (Washington, D.C.: Dec. 5, 2005).
[18] Ronald W. Reagan National Defense Authorization Act for Fiscal
Year 2005, Pub. L. No. 108-375, § 332 (2004) (codified at 10 U.S.C. §§
186 and 2222).
[19] Field/tactical refers to Army units that are deployable to
locations around the world, such as Iraq or Afghanistan.
[20] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-860].
[21] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-215].
[22] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-171].
[23] GAO, DOD Business Systems Modernization: Navy ERP Adherence to
Best Business Practices Critical to Avoid Past Failures, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-05-858] (Washington, D.C.: Sept.
29, 2005).
[24] Department of Defense Architecture Framework, Version 1.0, Volume
1 (February 2004); GAO, Information Technology: A Framework for
Assessing and Improving Enterprise Architecture Management (Version
1.1), [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-584G]
(Washington, D.C.: Apr. 1, 2003); Chief Information Officer Council, A
Practical Guide to Federal Enterprise Architecture, Version 1.0
(February 2001); and Institute of Electrical and Electronics Engineers
(IEEE), Standard for Recommended Practice for Architectural Description
of Software-Intensive Systems 1471-2000 (Sept. 21, 2000).
[25] A well-defined enterprise architecture provides a clear and
comprehensive picture of an entity, whether it is an organization
(e.g., a federal department) or a functional or mission area that cuts
across more than one organization (e.g., personnel management). This
picture consists of snapshots of both the enterprise's current or "as
is" environment and its target or "to be" environment, as well as a
capital investment road map for transitioning from the current to the
target environment. These snapshots consist of integrated "views,"
which are one or more architecture products that describe, for example,
the enterprise's business processes and rules; information needs and
flows among functions, supporting systems, services, and applications;
and data and technical standards and structures.
[26] DOD has adopted a federated approach for developing its business
mission area enterprise architecture, which includes the corporate BEA
representing the thin layer of DOD-wide corporate architectural
policies, capabilities, rules, and standards; component architectures
(e.g., Navy enterprise architecture); and program architectures (e.g.,
Navy ERP architecture).
[27] See, for example, GAO, Information Technology: FBI Is Taking Steps
to Develop an Enterprise Architecture, but much Remains to Be
Accomplished, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-05-363]
(Washington, D.C.: Sept. 9, 2005); GAO, Homeland Security: Efforts
Under Way to Develop Enterprise Architecture, but Much Work Remains,
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-777] (Washington,
D.C.: Aug. 6, 2004); GAO, Information Technology: Architecture Needed
to Guide NASA's Financial Management Modernization, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-04-43] (Washington, D.C.: Nov.
21, 2003); GAO, DOD Business Systems Modernization: Important Progress
Made to Develop Business Enterprise Architecture, but Much Work
Remains, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-1018]
(Washington, D.C.: Sept. 19, 2003); GAO, Information Technology: DLA
Should Strengthen Business Systems Modernization Architecture and
Investment Activities, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-01-631] (Washington, D.C.: June 29, 2001); and GAO,
Information Technology: INS Needs to Better Manage the Development of
Its Enterprise Architecture, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO/AIMD-00-212] (Washington, D.C.: Aug. 1, 2000).
[28] DOD, Business Enterprise Architecture Compliance Guidance (Apr.
10, 2006).
[29] Initiated in 2003, GCSS-MC is to modernize the Marine Corps
logistics systems and thereby provide the decision makers with timely
and complete logistics information to support the warfighter. Moreover,
according to program officials, both GCSS-MC and Navy ERP are under the
leadership of Navy's Program Executive Office for Enterprise
Information Systems, which is responsible for developing, acquiring,
and deploying seamless enterprise-wide information technology systems.
[30] GAO, DOD Business Systems Modernization: Progress in Establishing
Corporate Management Controls Needs to be Replicated Within Military
Departments, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-705]
(Washington, D.C.: May 15, 2008).
[31] As we recently reported, while the corporate BEA includes
corporate policies, capabilities, rules, and standards, it is still
evolving and will continue to add additional details. See [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-08-705].
[32] GAO, DOD Business Systems Modernization: Military Departments Need
to Strengthen Management of Enterprise Architecture Programs,
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-519] (Washington,
D.C.: May 12, 2008).
[33] GAO, DOD Business Systems Modernization: Key Navy Programs'
Compliance with DOD's Federated Business Enterprise Architecture Needs
to Be Adequately Demonstrated, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-08-972] (Washington, D.C.: Aug. 7, 2008).
[34] Office of Management and Budget, Circular No. A-94: Guidelines and
Discount Rates for Benefit-Cost Analysis of Federal Programs (Oct. 29,
1992).
[35] The benefits estimates for the areas are rounded; therefore, they
add to more than $8.6 billion.
[36] For example, the Uniform Accounting Data Process System-Inventory
Control Points has yearly maintenance costs of $39.3 million.
[37] These reductions in expected savings represent the costs of
maintaining the legacy systems during the period in which the systems'
retirement dates have been delayed.
[38] OMB, Circular No. A-11, Preparation, Submission, and Execution of
the Budget (Washington, D.C.: Executive Office of the President, June
2006); Circular No. A-130 Revised, Management of Federal Information
Resources (Washington, D.C.: Executive Office of the President, Nov.
28, 2000); and Capital Programming Guide: Supplement to Circular A-11,
Part 7, Preparation, Submission, and Execution of the Budget
(Washington, D.C.: Executive Office of the President, June 2006).
[39] GAO, Cost Assessment Guide: Best Practices for Estimating and
Managing Program Costs, Exposure Draft, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP] (Washington, D.C.:
July 2, 2007).
[40] The Defense Cost and Resource Center is responsible for collecting
current and historical major defense acquisition program cost and
software resource data in a joint service environment and making those
data available for use by authorized government analysts when
estimating the cost of ongoing and future government programs.
[41] GAO, DOD Business Systems Modernization: Key Marine Corps System
Acquisition Needs to Be Better Justified, Defined, and Managed,
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-822] (Washington,
D.C.: July 28, 2008).
[42] Sensitivity analysis reveals how the cost estimate is affected by
a change in a single assumption or cost driver while holding all other
variables constant.
[43] GAO, Missile Defense: Additional Knowledge Needed in Developing
System for Intercepting Long-Range Missiles, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-03-600] (Washington, D.C.: Aug.
21, 2003).
[44] OMB, Preparation, Submission, and Execution of the Budget,
Circular A-11 (Washington, D.C.: Executive Office of the President,
June 2006), part 7, Planning, Budgeting, Acquisition, and Management of
Capital Assets, sec. 300. [hyperlink,
http://www.whitehouse.gov/omb/circulars/index.html].
[45] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP].
[46] Defense Contract Management Agency, Department of Defense Earned
Value Management Implementation Guide (Washington, D.C.: October 2006);
and [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP].
[47] American National Standards Institute (ANSI)/Electronic Industries
Alliance (EIA) Standard, EVM Systems Standard, ANSI/EIA-748-A-1998
(R2002), approved May 19, 1998; revised January 2002.
[48] Defense Contract Management Agency, Department of Defense Earned
Value Management Implementation Guide (Washington, D.C.: October 2006).
See also DOD Memorandum: Revision to DOD Earned Value Management Policy
(Washington, D.C.: Mar. 7, 2005).
[49] DON established the Center for Earned Value Management in April
2007 to, among other things, work with program offices to improve the
accuracy of EVM data and to provide independent assistance to program
managers when requested.
[50] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP].
[51] Float time is the amount of time an activity can slip before
affecting the critical path.
[52] Department of the Navy, Commander, Operational Test and Evaluation
Force, Navy Enterprise Resource Planning System Initial Operational
Test and Evaluation, OT-C1 Final Report to the Chief of Naval
Operations (Norfolk, VA: June 13, 2008).
[53] Cost variances compare the earned value of the completed work with
the actual cost of the work performed. For example, if a contractor
completed $5 million worth of work, and the work actually cost $6.7
million, there would be a negative $1.7 million cost variance. Schedule
variances are also measured in dollars, but they compare the earned
value of the work actually completed as of a point in time to the value
of work that was expected to be completed. For example, if a contractor
completed $5 million worth of work at the end of the month but was
budgeted to complete $10 million worth of work, there would be a
negative $5 million schedule variance.
[54] This figure is the total estimated budget for the program. It
represents the cumulative value of the budgeted cost of work scheduled
over the life of the program.
[55] To make these assessments, we applied earned value analysis
techniques to data from the program's contract performance reports.
These techniques compare budget versus actual costs versus project
status in dollar amounts. For our analysis, we used standard earned
value formulas to calculate cost and schedule variance and forecast the
range of cost overrun at completion.
[56] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004).
Software Engineering Institute, Software Acquisition Capability
Maturity Model® version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA:
March 2002).
[57] Department of Defense, Risk Management Guide for DOD Acquisition,
6th Edition, Version 1.0., [hyperlink,
http://www.acq.osd.mil/sse/ed/docs/2006-RM-Guide-4Aug06-final-
version.pdf] (accessed March 13, 2008) and Software Engineering
Institute, CMMI for Acquisition, Version 1.2, CMU/SEI-2007- TR-017
(Pittsburgh, PA: November 2007).
[58] The risk management board oversees risk management activities by
assigning risk owners, approving and directing resources to facilitate
risk handling strategies, and reviewing risk handling status.
[59] Risk levels of high, medium or low are assigned using quantitative
measurements of the probability of the risk occurring and the potential
impact to the program's cost, schedule, and performance. Based on that
assessment, a risk level is assigned to represent the risk's
significance.
[60] See, for example, GAO, Office of Personnel Management: Retirement
Systems Modernization Program Faces Numerous Challenges, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-05-237] (Washington, D.C.: Feb.
28, 2005); and GAO, Information Technology: Customs Automated
Commercial Environment Program Progressing, but Need for Management
Improvements Continues, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-05-267] (Washington, D.C.: Mar. 14, 2005).
[61] Ronald W. Reagan National Defense Authorization Act for Fiscal
Year 2005, Pub. L. No. 108-375, § 332 (2004) (codified at 10 U.S.C. §§
186 and 2222).
[62] Office of Management and Budget, Circular No. A-94: Guidelines and
Discount Rates for Benefit-Cost Analysis of Federal Program (Oct. 29,
1992), [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP].
[63] Defense Contract Management Agency, Department of Defense Earned
Value Management Implementation Guide (Washington, D.C.: Apr. 7, 2005).
See also DOD Memorandum: Revision to DOD Earned Value Management Policy
(Washington, D.C.: Mar. 7, 2005).
[64] The earned value concept is applied as a means of placing a dollar
value on project status. The techniques we used compared the program's
budget to actual costs and project status in dollar amounts. We used
standard earned value formulas to calculate cost and schedule variance
and forecast the range of cost overrun at program completion.
[65] American National Standards Institute (ANSI) /Electronic
Industries Alliance (EIA) EVM Systems Standard, ANSI/EIA-748-A-1998
(R2002), approved May 19, 1998; revised January 2002.
[66] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP].
[67] Department of Defense, Risk Management Guide for DOD Acquisition,
6th Edition, Version 1.0., [hyperlink,
http://www.acq.osd.mil/sse/ed/docs/2006-RM-Guide-4Aug06-final-
version.pdf] (accessed March 13, 2008) and Software Engineering
Institute, CMMI for Acquisition, Version 1.2, CMU/SEI-2007-TR-017
(Pittsburgh, PA: November 2007).
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
