DOD Business Systems Modernization
Key Marine Corps System Acquisition Needs to Be Better Justified, Defined, and Managed Gao ID: GAO-08-822 July 28, 2008GAO has designated the Department of Defense's (DOD) business systems modernization as a high-risk program because, among other things, it has been challenged in implementing key information technology (IT) management controls on its thousands of business systems. The Global Combat Support System-Marine Corps program is one such system. Initiated in 2003, the program is to modernize the Marine Corps logistics systems. The first increment is to cost about $442 million and be deployed in fiscal year 2010. GAO was asked to determine whether the Department of the Navy is effectively implementing IT management controls on this program. To accomplish this, GAO analyzed the program's implementation of several key IT management disciplines, including economic justification, earned value management, risk management, and system quality measurement.
DOD has not effectively implemented key IT management controls provided for in DOD and related acquisition guidance on this program. If implemented effectively, these and other IT management disciplines increase the likelihood that a given system investment will produce the right solution to fill a mission need and that this system solution will be acquired and deployed in a manner that maximizes the chances of delivering promised system capabilities and benefits on time and within budget. Neither of these outcomes is being fully realized on this program, as evidenced by the fact that its first increment has already slipped more than 3 years and is expected to cost about $193 million more than envisioned. These slippages and cost overruns can be attributed in part to the management control weaknesses discussed in this report and summarized below. Moreover, additional slippages and overruns are likely if these and other IT management weaknesses are not addressed. Investment in the system has not been economically justified on the basis of reliable estimates of both benefits and costs. Specifically, while projected benefits were risk-adjusted to compensate for limited data and questionable assumptions, the cost side of the benefit/cost equation is not sufficiently reliable because it was not derived in accordance with key cost estimating practices. In particular, it was not based on historical data from similar programs and it did not account for schedule risks, both of which are needed for the estimate to be considered accurate and credible. Earned value management that the program uses to measure progress has not been adequately implemented. Specifically, the schedule baseline against which the program gauges progress is not based on key estimating practices provided for in federal guidance, such as assessing schedule risks and allocating schedule reserves to address these risks. As a result, program progress cannot be adequately measured, and likely program completion dates cannot be projected based on actual work performed. Some significant program risks have not been adequately managed. While a well-defined risk management plan and supporting process have been put in place, the process has not always been followed. Specifically, mitigation steps for significant risks either have not been implemented or proved ineffective, allowing the risks to become actual problems. The data needed to produce key indicators of system quality, such as trends in the volume of significant and unresolved problems and change requests, are not being collected. Without such data, it is unclear whether the system is becoming more or less mature and stable. The reasons for these weaknesses range from limitations of DOD guidance and tools, to not collecting relevant data. Until they are addressed, DOD is at risk of delivering a solution that does not cost-effectively support mission operations and falls short of cost, schedule, and capability expectations.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-08-822, DOD Business Systems Modernization: Key Marine Corps System Acquisition Needs to Be Better Justified, Defined, and Managed
This is the accessible text file for GAO report number GAO-08-822
entitled 'DOD Business Systems Modernization: Key Marine Corps System
Acquisition Needs to Be Better Justified, Defined, and Managed' which
was released on July 28, 2008.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Subcommittee on Readiness and Management Support,
Committee on Armed Services, U.S. Senate:
United States Government Accountability Office:
GAO:
July 2008:
DOD Business Systems Modernization:
Key Marine Corps System Acquisition Needs to Be Better Justified,
Defined, and Managed:
GAO-08-822:
GAO Highlights:
Highlights of GAO-08-822, a report to the Subcommittee on Readiness and
Management Support, Committee on Armed Services, U.S. Senate.
Why GAO Did This Study:
GAO has designated the Department of Defense‘s (DOD) business systems
modernization as a high-risk program because, among other things, it
has been challenged in implementing key information technology (IT)
management controls on its thousands of business systems. The Global
Combat Support System-Marine Corps program is one such system.
Initiated in 2003, the program is to modernize the Marine Corps
logistics systems. The first increment is to cost about $442 million
and be deployed in fiscal year 2010. GAO was asked to determine whether
the Department of the Navy is effectively implementing IT management
controls on this program. To accomplish this, GAO analyzed the
program‘s implementation of several key IT management disciplines,
including economic justification, earned value management, risk
management, and system quality measurement.
What GAO Found:
DOD has not effectively implemented key IT management controls provided
for in DOD and related acquisition guidance on this program. If
implemented effectively, these and other IT management disciplines
increase the likelihood that a given system investment will produce the
right solution to fill a mission need and that this system solution
will be acquired and deployed in a manner that maximizes the chances of
delivering promised system capabilities and benefits on time and within
budget. Neither of these outcomes is being fully realized on this
program, as evidenced by the fact that its first increment has already
slipped more than 3 years and is expected to cost about $193 million
more than envisioned. These slippages and cost overruns can be
attributed in part to the management control weaknesses discussed in
this report and summarized below. Moreover, additional slippages and
overruns are likely if these and other IT management weaknesses are not
addressed.
* Investment in the system has not been economically justified on the
basis of reliable estimates of both benefits and costs. Specifically,
while projected benefits were risk-adjusted to compensate for limited
data and questionable assumptions, the cost side of the benefit/cost
equation is not sufficiently reliable because it was not derived in
accordance with key cost estimating practices. In particular, it was
not based on historical data from similar programs and it did not
account for schedule risks, both of which are needed for the estimate
to be considered accurate and credible.
* Earned value management that the program uses to measure progress has
not been adequately implemented. Specifically, the schedule baseline
against which the program gauges progress is not based on key
estimating practices provided for in federal guidance, such as
assessing schedule risks and allocating schedule reserves to address
these risks. As a result, program progress cannot be adequately
measured, and likely program completion dates cannot be projected based
on actual work performed.
* Some significant program risks have not been adequately managed.
While a well-defined risk management plan and supporting process have
been put in place, the process has not always been followed.
Specifically, mitigation steps for significant risks either have not
been implemented or proved ineffective, allowing the risks to become
actual problems.
* The data needed to produce key indicators of system quality, such as
trends in the volume of significant and unresolved problems and change
requests, are not being collected. Without such data, it is unclear
whether the system is becoming more or less mature and stable.
The reasons for these weaknesses range from limitations of DOD guidance
and tools, to not collecting relevant data. Until they are addressed,
DOD is at risk of delivering a solution that does not cost-effectively
support mission operations and falls short of cost, schedule, and
capability expectations.
What GAO Recommends:
GAO is making recommendations to the Secretary of Defense aimed at
limiting investment in the program and addressing its cost and schedule
estimating, risk management, and system quality measurement weaknesses.
DOD agreed in full or in part with GAO‘s recommendations and described
ongoing and planned actions intended to address the recommendations.
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-822]. For more
information, contact Randolph C. Hite at (202) 512-3439 or
hiter@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
Key DOD and Related IT Acquisition Management Controls Have Not Been
Fully Implemented on GCSS-MC:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Objective, Scope, and Methodology:
Appendix II: Comments from the Department of Defense:
Appendix III: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Description of Legacy Systems Scheduled for Retirement in
2010:
Table 2: Organizations Responsible for GCSS-MC Oversight and
Management:
Table 3: Summary of Selected System Acquisition Best Practices:
Table 4: Summary of Cost Estimating Characteristics That the Cost
Estimate Satisfies:
Table 5: IMS Satisfaction of Nine Schedule Estimating Key Practices:
Table 6: MCITS Priority Levels:
Table 7: Change Request Priorities:
Figures:
Figure 1: GCSS-MC Program Schedule Status:
Figure 2: GCSS-MC Program Cost Status:
Abbreviations:
BEA: business enterprise architecture:
BTA: Business Transformation Agency:
CIO: Chief Information Officer:
DAS: defense acquisition system:
DBSMC: Defense Business Systems Management Committee:
DOD: Department of Defense:
DON: Department of the Navy:
ERP: enterprise resource planning:
EVM: earned value management:
FOC: full operational capability:
GCSS-MC: Global Combat Support System-Marine Corps:
I-RMIS: Issues-Risk Management Information System:
IEEE: Institute of Electrical and Electronics Engineers:
IMS: integrated master schedule:
IRB: investment review board:
IT: information technology:
MCITS: Marine Corps Issue Tracking System:
MDA: milestone decision authority:
NTCSS: Naval Tactical Command Support System:
OMB: Office of Management and Budget:
TC-AIMS II: Transportation Coordinators' Automated Information for
Movements System II:
USD AT&L: Under Secretary of Defense for Acquisition, Technology and
Logistics:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
July 28, 2008:
The Honorable Daniel K. Akaka:
Chairman:
The Honorable John Thune:
Ranking Member:
Subcommittee on Readiness and Management Support:
Committee on Armed:
Services United States Senate:
The Honorable John Ensign:
United States Senate:
For decades, the Department of Defense (DOD) has been challenged in
modernizing its timeworn business systems.[Footnote 1] In 1995, we
designated DOD's business systems modernization program as high risk
and continue to do so today.[Footnote 2] Our reasons include the
modernization's large size, complexity, and critical role in addressing
other long-standing transformation and financial management challenges.
Other reasons are that DOD has yet to institutionalize key system
modernization management controls, and it has not demonstrated the
ability to consistently deliver promised system capabilities and
benefits on time and within budget.
Nevertheless, DOD continues to invest billions of dollars in thousands
of business systems, including about a hundred that the department has
labeled as business transformational programs, 12 of which account for
about 50 percent of these programs' costs. The Global Combat Support
System-Marine Corps (GCSS-MC) is one such program. Initiated in 2003,
GCSS-MC is to modernize the Marine Corps logistics systems and thereby
provide decision makers with timely and complete logistics information
to support the warfighter. As envisioned, the program consists of a
series of major increments, the first of which is expected to cost
approximately $442 million and be fully deployed in fiscal year 2010.
As agreed, our objective was to determine whether the Department of the
Navy is effectively implementing information technology (IT) management
controls on GCSS-MC. To accomplish this, we focused on the first
increment of GCSS-MC by analyzing a range of program documentation and
interviewing cognizant officials relative to the following management
areas: architectural alignment, economic justification, earned value
management, requirements management, risk management, and system
quality measurement. We conducted our performance audit from June 2007
to July 2008, in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objective. We
believe that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objective. Additional
details on our objective, scope, and methodology are in appendix I.
Results in Brief:
DOD has not effectively implemented key IT management controls on its
GCSS-MC program. Collectively, these management controls are intended
to reasonably ensure that investment in a given system represents the
right solution to fill a mission need--and if it is, that the system is
acquired and deployed the right way, meaning that it is done in a
manner that maximizes the chances of delivering defined system
capabilities and benefits on time and within budget. Given that
deployment of GCSS-MC is more than 3 years behind schedule and expected
to cost about $193 million more than envisioned, these goals are
already not being met, in part because DOD program management and
oversight entities have not adequately implemented several key IT
management controls. As a result, the department does not have a
sufficient basis for knowing that GCSS-MC, as defined, is the best
system solution to meeting its mission needs, and the program is likely
to experience further schedule slips and cost overruns, along with
reduced system capabilities. Weaknesses associated with DOD's
implementation of five key IT management controls, as well as recent
actions to correct weaknesses with another management control, are as
follows:
* GCSS-MC compliance with DOD's federated business enterprise
architecture (BEA) has not been sufficiently demonstrated. To its
credit, the program office has followed DOD's BEA compliance guidance.
However, the program's compliance assessment (1) did not include all
relevant architecture products, such as those that describe technical
and system elements; (2) was not used to identify potential areas of
duplication across programs; and (3) did not address compliance with
the Department of the Navy's enterprise architecture. These important
steps were not performed because of policy, guidance, and tool
limitations, and because aspects of the corporate BEA and the
Department of the Navy's enterprise architecture, which are both major
components of DOD's federated BEA, have yet to be sufficiently defined
to permit thorough compliance determinations in these areas. In
addition, program oversight and approval authorities did not validate
the program office's compliance assessments. As a result, the
department does not have a sufficient basis for knowing if GCSS-MC has
been defined to optimize the DOD and Department of the Navy business
operations.
* Investment in GCSS-MC has not been economically justified. According
to the program's economic analysis, the first increment will have an
estimated life cycle cost of about $442 million and deliver about $1.04
billion in risk-adjusted, estimated benefits. While the most recent
cost estimate was derived using some effective estimating practices, it
was not based on other practices that are essential to having an
accurate and credible estimate. For example, the estimate was not
grounded in a historical record of comparable data from similar
programs and did not account for significant risks associated with the
program's aggressive schedule, both of which limit the estimate's
accuracy and credibility. These important practices were not employed
for various reasons, including a lack of historical data from similar
programs and a schedule risk analysis to assess the cost estimate's
variability. As a result, an adequate basis for informed investment
decision making does not exist, and actual program costs will likely
not be consistent with estimates.
* Earned value management (EVM), which is a means for determining and
disclosing actual program cost and schedule performance in comparison
with estimates, is not being effectively performed because the schedule
baseline is not reliable. Specifically, while the program's current
schedule baseline was derived using some key estimating practices, it
is not reflective of other important practices, such as conducting a
schedule risk assessment and allocating schedule reserve. These
important practices were not followed, according to program officials,
because doing so would have pushed back the estimated completion date
for the first increment, which they said would not have been consistent
with DOD oversight and approval authorities' direction to complete the
increment as soon as possible. In other words, the program office
adopted what amounted to an imposing completion date but did not adjust
the scope and schedule of work to be completed to make this date
attainable. The result is a schedule that is not reliable and does not
provide a sufficient baseline for performing EVM.
* Despite limitations in earlier efforts to manage GCSS-MC's
requirements, improvements have since been made. During the initial
phase of our review, the program office could not trace all of its
1,375 system-level requirements to design specifications and test
documentation. For example, about 30 percent of the program's design
documents had yet to be validated, approved, and linked to the
requirements. This was significant because it had already contributed
to lengthy delays to the program's schedule and, without adequate
traceability, the risk of the system not performing as intended and
requiring expensive rework is increased. Following our inquiries into
the traceability process being used, program officials changed the
process. Since then, our analysis of 61 randomly selected, system-level
requirements confirmed that 60 are now traceable backward to
operational requirements and forward to design specifications and test
plans. If implemented effectively, the new process should address
previous requirements' traceability weaknesses and thereby avoid a
repeat of past problems.
* Despite having a well-defined process for managing program risks, not
all program risks have been adequately managed. For example, of 25
medium risks that the program office reported as closed as of February
2008, 4 were closed because they became actual issues (problems). In
each of these cases, the risk mitigation strategy was not fully
implemented. In addition, 4 were closed because, even though the risk
strategies were implemented, the strategies did not mitigate the risks,
resulting in each becoming an actual problem. Program officials
attributed the lack of success in mitigating these risks to, among
other things, resource constraints and an aggressive program schedule.
Unless program risks are effectively mitigated, GCSS-MC will experience
further cost, schedule, and performance shortfalls.
* Sufficient data for measuring trends in the number of high priority,
unresolved system issues (problems) and system change requests, both of
which are recognized indicators of a system's quality or maturity, are
not available. On the positive side, the program office has established
processes for (1) collecting and tracking data on the status of program
issues, including problems discovered during early test events, and (2)
capturing data on the status of requests for changes to the system.
Moreover, program documentation emphasizes the importance of monitoring
trends in program problems and change requests that have yet to be
addressed or resolved. However, an effective means for producing the
full complement of data that are needed for a reliable picture of such
trends does not exist. In particular, data on problems and change
request priority levels and closure dates are not consistently
maintained, and program oversight of contractor-identified issues or
defects is limited. Program officials stated that they intend to
address these data limitations, but stated that oversight of contractor-
identified issues is not their responsibility. Without sufficient data
available to understand trends in the volume and severity of program
problems and changes, it is unclear whether GCSS-MC's quality and
stability are moving in the right direction.
Until the above discussed IT management controls are effectively
implemented, DOD is at risk of investing in incremental GCSS-MC system
solutions that do not optimally support corporate mission needs and
mission performance, and do not satisfy defined requirements and meet
schedule and cost commitments. These risks have already been realized,
as evidenced by the more than 3-year delay and $193 million increase in
the expected costs to deploy the first system increment. Accordingly,
we are making recommendations to the Secretary of Defense aimed at
ensuring that any decision to invest in the next acquisition phase of
GCSS-MC's first increment is made in light of the status of efforts to
address the risks discussed in this report and that investment in all
future increments be conditional upon the program having fully
addressed the control weaknesses discussed in this report. We are also
making recommendations intended to correct the cost estimating,
schedule estimating, risk management, and system quality measurement
control weaknesses discussed in this report. However, we are not making
recommendations in this report relative to addressing the architecture
compliance weaknesses because we have work under way that is more
broadly focused on this area across multiple programs, and we will be
making recommendations in that report. We are also not making
recommendations regarding requirements traceability because the
weaknesses that we found have recently been corrected.
In written comments on a draft of this report, signed by the Deputy
Under Secretary of Defense (Business Transformation) and reprinted in
appendix II, the department stated that it concurred with two of our
recommendations and partially concurred with the remaining five. In
general, it partially concurred with the five recommendations because
it said that efforts were either under way or planned that will address
some of the weaknesses that these recommendations are aimed at
correcting. We support these efforts because they are generally
consistent with the intent of the recommendations and believe that, if
fully and properly implemented, they will go a long way in addressing
the management control weaknesses that our recommendations are intended
to correct. DOD's comments are reprinted in their entirety in appendix
II of this report.
Background:
The Department of the Navy (DON) is a major component of DOD,
consisting of two uniformed services: the Navy and the Marine Corps.
The Marine Corps' primary mission is to serve as a "total force in
readiness" by responding quickly in a wide spectrum of
responsibilities, such as attacks from sea to land in support of naval
operations, air combat, and security of naval bases. As the only
service that operates in three dimensions--in the air, on land, and at
sea, the Marine Corps must be equipped to provide rapid and precise
logistics[Footnote 3] support to operating forces in any environment.
The Marine Corps' many and dispersed organization components rely
heavily on IT to perform their respective mission-critical operations
and related business functions, such as logistics and financial
management. For fiscal year 2008, the Marine Corps budget for IT
business systems is about $1.3 billion, of which $746 million (57
percent) is for operations and maintenance of existing systems and $553
million (43 percent) is for systems development and modernization. Of
the approximately 904 systems in DON's current inventory, the Marine
Corps accounts for 81, or about 9 percent, of the total. The GCSS-MC is
one such system investment. According to DOD, it is intended to address
the Marine Corps' long-standing problem of stove-piped logistics
systems that collectively provide limited data visibility and access,
are unable to present a common, integrated logistics picture in support
of the warfighter, and do not provide important decision support tools.
GCSS-MC: A Brief Description:
In September 2003, the Marine Corps initiated GCSS-MC[Footnote 4] to
(1) deliver integrated functionality across the logistics areas (e.g.,
supply and maintenance), (2) provide timely and complete logistics
information to authorized users for decision making, and (3) provide
access to logistics information and applications regardless of
location. The system is intended to function in three operational
environments--deployed operations (i.e., in theater of war or exercise
environment on land or at sea), in-transit, and in garrison.[Footnote
5] When GCSS-MC is fully implemented, it is to support about 33,000
users located around the world.
GCSS-MC is being developed in a series of large and complex increments
using commercially available enterprise resource planning
(ERP)[Footnote 6] software and hardware components. The first increment
is currently the only funded portion of the program and is to provide a
range of asset management capabilities, including:
* planning inventory requirements to support current and future
demands;
* requesting and tracking the status of products (e.g., supplies and
personnel) and services (e.g., maintenance and engineering);
* allocating resources (e.g., inventory, warehouse capacity, and
personnel) to support unit demands for specific products; and:
* scheduling maintenance resources (e.g., manpower, equipment, and
supplies) for specific assets, such as vehicles.
Additionally, the first increment is to replace four legacy systems
scheduled for retirement in 2010. Table 1 describes these four systems.
Table 1: Description of Legacy Systems Scheduled for Retirement in
2010:
System: Supported Activities and Supply System;
Description: Thirty-five year old mainframe ground supply system that
is used for procuring, distributing, and managing Marine Corps'
supplies.
System: Marine Corps Integrated Maintenance Management System;
Description: Twenty-nine year old mainframe system that is used for
maintaining ground equipment, including planning and managing work
orders and parts, and for performing equipment readiness reporting and
status tracking.
System: Asset Tracking and Logistics System;
Description: Fifteen-year-old data entry system dedicated to supporting
the Supported Activities and Supply System and the Marine Corps
Integrated Maintenance Management System for controlling, distributing,
and replenishing equipment and supplies in assigned areas of operation
and receiving supply support from and providing it to other military
departments.
System: Personal Computer Marine Corps Integrated Maintenance
Management System;
Description: Fourteen-year-old stand-alone system that is used for
performing a limited number of ground maintenance management logistics
functions.
Source: DOD.
[End of table]
Future increments are to provide additional functionality (e.g.,
transportation and wholesale inventory management), enhance existing
functionality, and potentially replace up to 44 additional legacy
systems.
The program office estimates the total life cycle cost for the first
increment to be about $442 million, including $169 million for
acquisition and $273 million for operations and maintenance.[Footnote
7] The total life cycle cost of the entire program has not yet been
determined because future increments are currently in the planning
stages and have not been defined. As of April 2008, the program office
reported that approximately $125 million has been spent on the first
increment.
Program Oversight and Management Roles and Responsibilities:
To manage the acquisition and deployment of GCSS-MC, the Marine Corps
established a program management office within the Program Executive
Office for Executive Information Systems. The program office is led by
the Program Manager who is responsible for managing the program's scope
and funding and ensuring that the program meets its objectives. To
accomplish this, the program office is responsible for key acquisition
management controls, such as architectural alignment, economic
justification, EVM, requirements management, risk management, and
system quality measurement. In addition, various DOD and DON
organizations share program oversight and review activities relative to
these and other acquisition management controls. A listing of key
entities and their roles and responsibilities is in table 2.
Table 2: Organizations Responsible for GCSS-MC Oversight and
Management:
Entity: Under Secretary of Defense for Acquisition, Technology, and
Logistics (USD AT&L);
Roles and responsibilities: Serves as the milestone decision authority
(MDA), which according to DOD, has overall responsibility for the
program, to include approving the program to proceed through its
acquisition cycle on the basis of, for example, the acquisition plan,
an independently evaluated economic analysis, and the Acquisition
Program Baseline.
Entity: Assistant Secretary of the Navy, Research, Development, and
Acquisition;
Roles and responsibilities: Serves as DON's oversight organization for
the program, to include enforcement of USD AT&L policies and
procedures.
Entity: Department of the Navy, Program Executive Office for Executive
Information Systems;
Roles and responsibilities: Oversees a portfolio of large-scale
projects and programs designed to enable common business processes and
provide standard capabilities, to include reviewing the acquisition
plan, economic analysis, and the Acquisition Program Baseline prior to
approval by the MDA.
Entity: Department of the Navy Chief Information Officer (CIO);
Roles and responsibilities: Supports the department's planning,
programming, budgeting, and execution processes by ensuring that the
program has achievable and executable goals and conforms to financial
management regulations, and DON, DOD, and federal IT policies in
several areas (e.g., security, architecture, and investment
management); it also works closely with the program office during
milestone review assessments.
Entity: Office of the Secretary of Defense, Office of the Director for
Program Analysis and Evaluation;
Roles and responsibilities: Verifies and validates the reliability of
cost and benefit estimates in economic analyses and provides its
results to the MDA.
Entity: Naval Center for Cost Analysis;
Roles and responsibilities: Performs independent program cost
estimates.
Entity: Defense Cost and Resource Center;
Roles and responsibilities: Collects current and historical major
defense acquisition program cost and software resource data in a joint
service environment and makes those data available for use by
authorized government analysts when estimating the cost of ongoing and
future government programs.
Entity: Deputy Commandant, Installations and Logistics, Headquarters,
Marine Corps;
Roles and responsibilities: Provides guidance and direction for overall
logistics modernization effort and develops/approves formal capability
requirements for the program.
Entity: Defense Business Systems Management Committee (DBSMC);
Roles and responsibilities: Serves as the highest ranking governance
body for business systems modernization activities and approves
investments costing more than $1 million, as, for example, being
compliant with the BEA.
Entity: Investment Review Board (IRB);
Roles and responsibilities: Reviews business system investments and has
responsibility for recommending certification for all business system
investments costing more than $1 million that are asserted as compliant
with the BEA.
Entity: Business Transformation Agency (BTA);
Roles and responsibilities: Coordinates business transformation efforts
across DOD and supports the IRBs and DBSMC.
Entity: GCSS-MC Program Management Office;
Roles and responsibilities: Performs day-to-day program management and,
as such, is the single point of accountability for managing the
program's objectives through development, production, and sustainment,
such as risk management and coordinating all testing activities with
requirements, including measuring system quality and managing change
requests.
Source: DOD.
[End of table]
Overview of GCSS-MC's Status:
The program reports that the first increment of GCSS-MC is currently in
the system development and demonstration phase of the defense
acquisition system (DAS).[Footnote 8] The DAS consists of five key
program life cycle phases and three related milestone decision points.
These five phases and related milestones are described along with a
summary of key program activities completed during, or planned, for
each phase as follows:
1. Concept refinement: The purpose of this phase is to refine the
initial system solution (concept) and create a strategy for acquiring
the investment solution. During this phase, the program office defined
the acquisition strategy and analyzed alternative solutions. The first
increment completed this phase on July 23, 2004, which was 1 month
later than planned, and the MDA approved a Milestone A decision to move
to the next phase.
2. Technology development: The purpose of this phase is to determine
the appropriate set of technologies to be integrated into the
investment solution by iteratively assessing the viability of various
technologies while simultaneously refining user requirements. During
this phase, the program office selected Oracle's E-Business Suite
[Footnote 9] as the commercial off-the-shelf ERP software. In addition,
the program office awarded Accenture the system integration contract
to, among other things, configure the software, establish system
interfaces, and implement the new system. This system integration
contract was divided into two phases--Part 1 for the planning,
analysis, and conceptual design of the solution and Part 2 for detailed
design, build, test, and deployment of the solution. The program office
did not exercise the option for Part 2 of the contract to Accenture and
shortly thereafter established a new program baseline in June 2006. In
November 2006, it awarded a time-and-materials system integration
contract[Footnote 10] valued at $28.4 million for solution design to
Oracle. The first increment completed this phase on June 8, 2007, which
was 25 months later than planned due in part to contractual performance
shortfalls, and the MDA approved a Milestone B decision to move to the
next phase.
3. System development and demonstration: The purpose of this phase is
to develop the system and demonstrate through developer testing that
the system can function in its target environment. During this phase,
the program office extended the solution design contract and increased
funding to $67.5 million due, in part, to delays in completing the
detailed design activities. As a result, the program office has not yet
awarded the next contract (which includes both firm-fixed-price and
time-and-materials task orders) for build and testing activities,
originally planned for July 2007. Instead, it entered what it termed a
"transition period" to complete detailed design activities. According
to the program's baseline, the MDA is expected to approve a Milestone C
decision to move to the next phase in October 2008. However, program
officials stated that Milestone C is now scheduled for April 2009,
which is 35 months later than originally planned.
4. Production and deployment: The purpose of this phase is to achieve
an operational capability that satisfies the mission needs, as verified
through independent operational test and evaluation, and implement the
system at all applicable locations. The program office plans to award a
separate firm-fixed-price plus award fee contract[Footnote 11] for
these activities with estimated costs yet to be determined.
5. Operations and support: The purpose of this phase is to
operationally sustain the system in the most cost-effective manner over
its life cycle. The details of this phase have not yet been defined.
Overall, GCSS-MC was originally planned to reach full operational
capability (FOC)[Footnote 12] in fiscal year 2007 at an estimated cost
of about $126 million over a 7-year life cycle.[Footnote 13] This cost
estimate was later revised in 2005 to about $249 million over a 13-year
life cycle.[Footnote 14] However, the program now expects to reach FOC
in fiscal year 2010 at a cost of about $442 million over a 12-year life
cycle. Figures 1 and 2 show the program's current status against
original milestones and original, revised, and current cost estimates.
Figure 1: GCSS-MC Program Schedule Status:
This figure is a timeline of the GCSS-MC Program schedule status, as
follows:
DAS Phases: Concept refinement;
May 2004 schedule: Late 2003 to Late 2004;
March 2008 schedule: Late 2003 to Late 2004.
DAS Phases: Technology development;
May 2004 schedule: Late 2004 to mid-2005;
March 2008 schedule: Late 2004 to late 2007 (rebaselined mid-2006).
DAS Phases: System development and demonstration (solution design and
build/test activities);
May 2004 schedule: Mid-2005 to mid-2006;
March 2008 schedule: Mid 2007 to mid-2009;
DAS Phases: Production and deployment;
May 2004 schedule: Mid-2006 to mid-2007 (FOC);
March 2008 schedule: Mid-2009 to mid-2010 (FOC).
July 28, 2008: Issue date of GAO-08-822.
Source: GAO analysis of Marine Corps data.
[End of figure]
Figure 2: GCSS-MC Program Cost Status:
[See PDF for image]
This figure is a vertical bar graph depicting the following data:
Fiscal year: 2004 (May 2004: Fiscal years 2004-2011);
Dollars in millions: $126 million.
Fiscal year: 2005 (July 2005: Fiscal years 2005-2018);
Dollars in millions: $249 million.
Fiscal year: 2007 (January 2007: Fiscal years 2007-2019);
Dollars in millions: $442 million.
Source: GAO analysis of Marine Corps data.
[End of figure]
Use of IT Acquisition Management Best Practices Maximizes Chances for
Success:
Acquisition best practices are tried and proven methods, processes,
techniques, and activities that organizations define and use to
minimize program risks and maximize the chances of a program's success.
Using best practices can result in better outcomes, including cost
savings, improved service and product quality, and a better return on
investment. For example, two software engineering analyses of nearly
200 systems acquisition projects indicate that teams using systems
acquisition best practices produced cost savings of at least 11 percent
over similar projects conducted by teams that did not employ the kind
of rigor and discipline embedded in these practices.[Footnote 15] In
addition, our research shows that best practices are a significant
factor in successful acquisition outcomes and increase the likelihood
that programs and projects will be executed within cost and schedule
estimates.[Footnote 16]
We and others have identified and promoted the use of a number of best
practices associated with acquiring IT systems.[Footnote 17] See table
3 for a description of several of these activities.
Table 3: Summary of Selected System Acquisition Best Practices:
Business practice: Architectural alignment; To ensure that the
acquisition is consistent with the organization's enterprise
architecture;
Description: Architectural alignment is the process for analyzing and
verifying that the proposed architecture of the system being acquired
is consistent with the enterprise architecture for the organization
acquiring the system. Such alignment is needed to ensure that acquired
systems can interoperate and are not unnecessarily duplicative of one
another.
Business practice: Economic justification; To ensure that system
investments have an adequate economic justification;
Description: Economic justification is the process for ensuring that
acquisition decisions are based on reliable analyses of the proposed
investment's likely costs versus benefits over its useful life, as well
as an analysis of the risks associated with actually realizing the
acquisition's forecasted benefits for its estimated costs. Economic
justification is not a one-time event but rather is performed
throughout an acquisition's life cycle in order to permit informed
investment decision making.
Business practice: Earned value management; To ensure that actual
progress is being monitored against expected progress;
Description: Earned value management is a tool that integrates the
technical, cost, and schedule parameters of a contract and measures
progress against them. During the planning phase, an integrated program
baseline is developed by time phasing budget resources for defined
work. As work is performed and measured against the baseline, the
corresponding budget value is "earned." Using this earned value metric,
cost and schedule variances, as well as cost and time to complete
estimates, can be determined and analyzed.
Business practice: Requirements management; To ensure that requirements
are traceable, verifiable, and controlled;
Description: Requirements management is the process for ensuring that
the requirements are traceable, verifiable, and controlled.
Traceability refers to the ability to follow a requirement from origin
to implementation and is critical to understanding the interconnections
and dependencies among the individual requirements, as well as the
impact when a requirement is changed. Requirements management begins
when the contractual requirements are documented and ends when system
responsibility is transferred to the support organization.
Business practice: Risk management; To ensure that risks are identified
and systematically mitigated;
Description: Risk management is the process for identifying potential
acquisition problems and taking appropriate steps to avoid their
becoming actual problems. Risk management occurs early and continuously
in the acquisition life cycle.
Business practice: System quality measurement; To ensure the maturity
and stability of system products;
Description: System quality measurement is the process for
understanding the maturity and stability of the system products being
developed, operated, and maintained so that problems can be identified
and addressed early, therefore limiting their overall impact on program
cost and schedule. One indicator of system quality is the volume and
significance of system defect reports and change proposals.
Source: GAO.
[End of table]
Prior GAO Reviews Have Identified IT Acquisition Management Weaknesses
in DOD's Business System Investments:
We have previously reported[Footnote 18] that DOD has not effectively
managed a number of business system investments. Among other things,
our reviews of individual system investments have identified weaknesses
in such areas as architectural alignment and informed investment
decision making, which are also the focus areas of the Fiscal Year 2005
National Defense Authorization Act[Footnote 19] business system
provisions. Our reviews have also identified weaknesses in other system
acquisition and investment management areas--such as EVM, economic
justification, requirements management, risk management, and test
management.
Most recently, for example, we reported that the Army's approach to
investing about $5 billion over the next several years in its General
Fund Enterprise Business System, Global Combat Support System-Army
Field/Tactical,[Footnote 20] and Logistics Modernization Program did
not include alignment with Army enterprise architecture or use a
portfolio-based business system investment review process.[Footnote 21]
Moreover, we reported that the Army did not have reliable analyses,
such as economic analyses, to support its management of these programs.
We concluded that until the Army adopts a business system investment
management approach that provides for reviewing groups of systems and
making enterprise decisions on how these groups will collectively
interoperate to provide a desired capability, it runs the risk of
investing significant resources in business systems that do not provide
the desired functionality and efficiency. Accordingly, we made
recommendations aimed at improving the department's efforts to achieve
total asset visibility and enhancing its efforts to improve its control
and accountability over business system investments. The department
agreed with our recommendations.
We also reported that DON had not, among other things, economically
justified its ongoing and planned investment in the Naval Tactical
Command Support System (NTCSS)[Footnote 22] and had not invested in
NTCSS within the context of a well-defined DOD or DON enterprise
architecture. In addition, we reported that DON had not effectively
performed key measurement, reporting, budgeting, and oversight
activities and had not adequately conducted requirements management and
testing activities. We concluded that, without this information, DON
could not determine whether NTCSS, as defined, and as being developed,
is the right solution to meet its strategic business and technological
needs. Accordingly, we recommended that the department develop the
analytical basis to determine if continued investment in the NTCSS
represents prudent use of limited resources and to strengthen
management of the program, conditional upon a decision to proceed with
further investment in the program. The department largely agreed with
these recommendations.
In addition, we reported that the Army had not defined and developed
its Transportation Coordinators' Automated Information for Movements
System II (TC-AIMS II)--a joint services system with the goal of
helping to manage the movement of forces and equipment within the
United States and abroad--in the context of a DOD enterprise
architecture.[Footnote 23] We also reported that the Army had not
economically justified the program on the basis of reliable estimates
of life cycle costs and benefits and had not effectively implemented
risk management. As a result, we concluded that the Army did not know
if its investment in TC-AIMS II, as planned, is warranted or represents
a prudent use of limited DOD resources. Accordingly, we recommended
that DOD, among other things, develop the analytical basis needed to
determine if continued investment in TC-AIMS II, as planned, represents
prudent use of limited defense resources. In response, the department
largely agreed with our recommendations and has since reduced the
program's scope by canceling planned investments.
Key DOD and Related IT Acquisition Management Controls Have Not Been
Fully Implemented on GCSS-MC:
DOD IT-related acquisition policies and guidance, along with other
relevant guidance, provide an acquisition management control framework
within which to manage business system programs like GCSS-MC. Effective
implementation of this framework can minimize program risks and better
ensure that system investments are defined in a way to optimally
support mission operations and performance, as well as deliver promised
system capabilities and benefits on time and within budget. Thus far,
GCSS-MC has not been managed in accordance with key aspects of this
framework, which has already contributed to more than 3 years in
program schedule delays and about $193 million in cost increases. These
IT acquisition management control weaknesses include:
* compliance with DOD's federated BEA not being sufficiently
demonstrated;
* expected costs not being reliably estimated;
* earned value management not being adequately implemented;
* system requirements not always being effectively managed, although
this has recently improved;
* key program risks not being effectively managed; and:
* key system quality measures not being used.
The reasons that these key practices have not been sufficiently
executed include limitations in the applicable DOD guidance and tools,
and not collecting relevant data, each of which is described in the
applicable sections of this report. By not effectively implementing
these key IT acquisition management controls, the program has already
experienced sizeable schedule and cost increases, and it is at
increased risk of (1) not being defined in a way that best meets
corporate mission needs and enhances performance and (2) costing more
and taking longer than necessary to complete.
GCSS-MC Compliance with DOD's Federated BEA Has Not Been Sufficiently
Demonstrated:
DOD and federal guidance[Footnote 24] recognize the importance of
investing in business systems within the context of an enterprise
architecture.[Footnote 25] Moreover, the 2005 National Defense
Authorization Act requires that defense business systems be compliant
with DOD's federated BEA.[Footnote 26] Our research and experience in
reviewing federal agencies show that not making investments within the
context of a well-defined enterprise architecture often results in
systems that are duplicative, are not well integrated, are
unnecessarily costly to interface and maintain, and do not optimally
support mission outcomes.[Footnote 27]
To its credit, the program office has followed DOD's BEA compliance
guidance.[Footnote 28] However, this guidance does not adequately
provide for addressing all relevant aspects of BEA compliance.
Moreover, DON's enterprise architecture, which is a major component of
DOD's federated BEA, as well as key aspects of DOD's corporate BEA,
have yet to be sufficiently defined to permit thorough compliance
determinations. In addition, current policies and guidance do not
require DON investments to comply with its enterprise architecture.
This means that the department does not have a sufficient basis for
knowing if GCSS-MC has been defined to optimize DON and DOD business
operations. Each of these architecture alignment limitations is
discussed as follows:
* The program's compliance assessments did not include all relevant
architecture products. In particular, the program did not assess
compliance with the BEA's technical standards profile, which outlines,
for example, the standards governing how systems physically communicate
with other systems and how they secure data from unauthorized access.
This is particularly important because systems, like GCSS-MC, need to
employ common standards in order to effectively and efficiently share
information with other systems. A case in point is GCSS-MC and the Navy
Enterprise Resource Planning program.[Footnote 29] Specifically, GCSS-
MC has identified 13 technical standards that are not in the BEA
technical standards profile, and Navy Enterprise Resource Planning has
identified 25 technical standards that are not in the profile. Of
these, some relate to networking protocols, which could limit
information sharing between these and other systems.
In addition, the program office did not assess compliance with the BEA
products that describe system characteristics. This is important
because doing so would create a body of information about programs that
could be used to identify common system components and services that
could potentially be shared by the programs, thus avoiding wasteful
duplication. For example, our analysis of GCSS-MC program documentation
shows that they contain such system functions as receiving goods,
taking physical inventories, and returning goods, which are also system
functions cited by the Navy Enterprise Resource Planning program.
However, because compliance with the BEA system products was not
assessed, the extent to which these functions are potentially
duplicative was not considered.
Furthermore, the program office did not assess compliance with BEA
system products that describe data exchanges among systems. As we
previously reported, establishing and using standard system interfaces
is a critical enabler to sharing data.[Footnote 30] For example, GCSS-
MC program documentation indicates that it is to exchange order and
status data with other systems. However, the program office has not
fully developed its architecture product describing these exchanges and
thus does not have the basis for understanding how its approach to
exchanging information differs from that of other systems that it is to
interface with. Compliance against each of these BEA products was not
assessed because DOD's compliance guidance does not provide for doing
so and, according to BTA and program officials, some BEA and program-
level architecture products are not sufficiently defined. According to
these officials, BTA plans to continue to define these products as the
BEA evolves.
* The compliance assessment was not used to identify potential areas of
duplication across programs, which DOD has stated is an explicit goal
of its federated BEA and associated investment review and decision-
making processes. More specifically, even though the compliance
guidance provides for assessing programs' compliance with the BEA
product that defines DOD operational activities, and GCSS-MC was
assessed for compliance with this product, the results were not used to
identify programs that support the same operational activities and
related business processes. Given that the federated BEA is intended to
identify and avoid not only duplications within DOD components, but
also between DOD components, it is important that such commonality be
addressed. For example, program-level architecture products for GCSS-MC
and Navy Enterprise Resource Planning, as well as two Air Force
programs (Defense Enterprise Accounting and Management System-Air Force
and the Air Force Expeditionary Combat Support System) show that each
supports at least six of the same BEA operational activities (e.g.,
conducting physical inventory, delivering property, and services) and
three of these four programs support at least 18 additional operational
activities (e.g., performing budgeting, managing receipt, and
acceptance). As a result, these programs may be investing in
duplicative functionality. Reasons for not doing so were that
compliance guidance does not provide for such analyses to be conducted,
and programs have not been granted access rights to use this
functionality.
* The program's compliance assessment did not address compliance
against the DON's enterprise architecture, which is one of the biggest
members of the federated BEA. This is particularly important given that
DOD's approach to fully satisfying the architecture requirements of the
2005 National Defense Authorization Act is to develop and use a
federated architecture in which component architectures are to provide
the additional details needed to supplement the thin layer of corporate
policies, rules, and standards included in the corporate BEA.[Footnote
31] As we recently reported, the DON's enterprise architecture is not
mature because, among other things, it is missing a sufficient
description of its current and future environments in terms of business
and information/data. However, certain aspects of an architecture
nevertheless exist and, according to DON, these aspects will be
leveraged in its efforts to develop a complete enterprise architecture.
For example, the FORCEnet architecture documents DON's technical
infrastructure. Therefore, opportunities exist for DON to assess its
programs in relation to these architecture products and to understand
where its programs are exposed to risks because products do not exist,
are not mature, or are at odds with other DON programs. According to
DOD officials, compliance with the DON architecture was not assessed
because DOD compliance policy is limited to compliance with the
corporate BEA, and the DON enterprise architecture has yet to be
sufficiently developed.
The program's compliance assessment was not validated by DOD or DON
investment oversight and decision-making authorities. More
specifically, neither the DOD IRBs nor the DBSMC, nor the BTA in
supporting both of these investment oversight and decision-making
authorities, reviewed the program's assessments. According to BTA
officials, under DOD's tiered approach to investment accountability,
these entities are not responsible for validating programs' compliance
assessments. Rather, this is a component responsibility, and thus they
rely on the military departments and defense agencies to validate the
assessments.
However, the DON Office of the CIO, which is responsible for
precertifying investments as compliant before they are reviewed by the
IRB, did not evaluate any of the programs' compliance assessments.
According to CIO officials, they rely on Functional Area Managers
[Footnote 32] to validate a program's compliance assessments. However,
no DON policy or guidance exists that describes how the Functional Area
Managers should conduct such validations.
Validation of program assessments is further complicated by the absence
of information captured in the assessment tool about what program
documentation or other source materials were used by the program office
in making its compliance determinations. Specifically, the tool is only
configured, and thus was only used, to capture the results of a
program's comparison of program architecture products to BEA products.
Thus, it was not used to capture the system products used in making
these determinations.
In addition, the program office did not develop certain program-level
architecture products that are needed to support and validate the
program's compliance assessment and assertions. According to the
compliance guidance, program-level architecture products, such as those
defining information exchanges and system data requirements are not
required to be used until after the system has been deployed. This is
important because waiting until the system is deployed is too late to
avoid the costly rework required to address areas of noncompliance.
Moreover, it is not consistent with other DOD guidance,[Footnote 33]
which states that program-level architecture products that describe,
for example, information exchanges, should be developed before a
program begins system development.
The limitations in existing BEA compliance-related policy and guidance,
the supporting compliance assessment tool, and the federated BEA, puts
programs like GCSS-MC at increased risk of being defined and
implemented in a way that does not sufficiently ensure interoperability
and avoid duplication and overlap. We currently have a review under way
for the Senate Armed Services Committee, Subcommittee on Readiness and
Management Support, which is examining multiple programs' compliance
with the federated BEA.
Investment in GCSS-MC Was Not Economically Justified Using Reliable
Estimates of Costs:
The investment in the first increment of GCSS-MC has not been
economically justified on the basis of reliable analyses of estimated
system costs over the life of the program. According to the program's
economic analysis, the first increment will have an estimated life
cycle cost of about $442 million and deliver an estimated $1.04 billion
in risk-adjusted estimated benefits during this same life cycle. This
equates to a net present value of about $688 million. While the most
recent cost estimate was derived using some effective estimating
practices, it did not make use of other practices that are essential to
having an accurate and credible estimate. As a result, the Marine Corps
does not have a sufficient basis for deciding whether GCSS-MC, as
defined, is the most cost-effective solution to meeting its mission
needs, and it does not have a reliable basis against which to measure
cost performance.
The Cost Estimate Was Not Derived Using Practices Necessary for an
Accurate and Credible Estimate:
A reliable cost estimate is critical to the success of any IT program,
as it provides the basis for informed investment decision making,
realistic budget formulation and program resourcing, meaningful
progress measurement, proactive course correction, and accountability
for results. According to the Office of Management and Budget (OMB),
[Footnote 34] programs must maintain current and well-documented cost
estimates, and these estimates must encompass the full life cycle of
the program. OMB states that generating reliable cost estimates is a
critical function necessary to support OMB's capital programming
process. Without reliable estimates, programs are at increased risk of
experiencing cost overruns, missed deadlines, and performance
shortfalls.
Our research has identified a number of practices for effective program
cost estimating. We have issued guidance that associates these
practices with four characteristics of a reliable cost estimate.
[Footnote 35] These four characteristic are specifically defined as
follows:
* Comprehensive: The cost estimates should include both government and
contractor costs over the program's full life cycle, from the inception
of the program through design, development, deployment, and operation
and maintenance, to retirement. They should also provide a level of
detail appropriate to ensure that cost elements are neither omitted nor
double counted and include documentation of all cost-influencing ground
rules and assumptions.
* Well-documented: The cost estimates should have clearly defined
purposes and be supported by documented descriptions of key program or
system characteristics (e.g., relationships with other systems,
performance parameters). Additionally, they should capture in writing
such things as the source data used and their significance, the
calculations performed and their results, and the rationale for
choosing a particular estimating method or reference. Moreover, this
information should be captured in such a way that the data used to
derive the estimate can be traced back to, and verified against, their
sources. The final cost estimate should be reviewed and accepted by
management on the basis of confidence in the estimating process and the
estimate produced by the process.
* Accurate: The cost estimates should provide for results that are
unbiased and should not be overly conservative or optimistic (i.e.,
should represent the most likely costs). In addition, the estimates
should be updated regularly to reflect material changes in the program,
and steps should be taken to minimize mathematical mistakes and their
significance. The estimates should also be grounded in a historical
record of cost estimating and actual experiences on comparable
programs.
* Credible: The cost estimates should discuss any limitations in the
analysis performed that are due to uncertainty or biases surrounding
data or assumptions. Further, the estimates' derivation should provide
for varying any major assumptions and recalculating outcomes based on
sensitivity analyses, and the estimates' associated risks and inherent
uncertainty should be disclosed. Also, the estimates should be verified
based on cross-checks using other estimating methods and by comparing
the results with independent cost estimates.
The $442 million life cycle cost estimate for the first increment
reflects many of the practices associated with a reliable cost
estimate, including all practices associated with being comprehensive
and well-documented, and several related to being accurate and
credible. (See table 4.) However, several important accuracy and
credibility practices were not satisfied.
Table 4: Summary of Cost Estimating Characteristics That the Cost
Estimate Satisfies:
Characteristic of reliable estimates: Comprehensive;
Satisfied?[A]: Yes.
Characteristic of reliable estimates: Well-documented;
Satisfied?[A]: Yes.
Characteristic of reliable estimates: Accurate;
Satisfied?[A]: Partially.
Characteristic of reliable estimates: Credible;
Satisfied?[A]: Partially.
Source: GAO analysis of Marine Corps data.
[A] "Yes" means that the program office provided documentation
demonstrating satisfaction of the criterion. "Partially" means that the
program office provided documentation demonstrating satisfaction of
part of the criterion. "No" means that the program office has yet to
provide documentation demonstrating satisfaction of the criterion.
[End of table]
The cost estimate is comprehensive because it includes both the
government and contractor costs specific to development, acquisition
(nondevelopment), implementation, and operations and support over the
program's 12-year life cycle. Moreover, the estimate clearly describes
how the various subelements are summed to produce the amounts for each
cost category, thereby ensuring that all pertinent costs are included,
and no costs are double counted. Lastly, cost-influencing ground rules
and assumptions, such as the program's schedule, labor rates, and
inflation rates are documented.
The cost estimate is also well-documented in that the purpose of the
cost estimate was clearly defined, and a technical baseline has been
documented that includes, among others things, the relationships with
other systems and planned performance parameters. Furthermore, the
calculations and results used to derive the estimate are documented,
including descriptions of the methodologies used and traceability back
to source data (e.g., vendor quotes, salary tables). Also, the cost
estimate was reviewed both by the Naval Center for Cost Analysis and
the Office of the Secretary of Defense, Director for Program Analysis
and Evaluation, which ensures a level of confidence in the estimating
process and the estimate produced.
However, the estimate lacks accuracy because not all important
practices related to this characteristic were satisfied. Specifically,
while the estimate is grounded in documented assumptions (e.g.,
hardware refreshment every 5 years), and periodically updated to
reflect changes to the program, it is not grounded in historical
experience with comparable programs. As stated in our guide, estimates
should be based on historical records of cost and schedule estimates
from comparable programs, and such historical data should be maintained
and used for evaluation purposes and future estimates on other
comparable programs. The importance of doing so is evident by the fact
that GCSS-MC's cost estimate has increased by about $193 million since
July 2005, which program officials attributed to, among other things,
schedule delays, software development complexity, and the lack of
historical data from similar ERP programs. While the program office did
leverage historical cost data from other ERP programs, including the
Navy's Enterprise Resource Planning Pilot programs and programs at the
Bureau of Prisons and the Department of Agriculture, program officials
told us that these programs' scopes were not comparable. For example,
none of the programs had to utilize a communication architecture as
complex as the Marine Corps, which officials cited as a significant
factor in the cost increases and a challenge in estimating costs.
The absence of analogous cost data for large-scale ERP programs is due
in part to the fact that DOD has not established a standardized cost
element structure for ERP programs that can be used to capture actual
cost data. According to officials with the Defense Cost and Resource
Center, such cost element structures are needed, along with a
requirement for programs to report on their costs, but approval and
resources have yet to be gained for either these structures or the
reporting of their costs. Until a standardized data structure exists,
programs like GCSS-MC will continue to lack a historical database
containing cost estimates and actual cost experiences of comparable ERP
programs. This means that the current and future GCSS-MC cost estimates
will lack sufficient accuracy for effective investment decision making
and performance measurement.
Compounding the estimate's limited accuracy are limitations in its
credibility. Specifically, while the estimate satisfies some of the key
practices for a credible cost estimate (e.g., confirming key cost
drivers, performing sensitivity analyses,[Footnote 36] having an
independent cost estimate prepared by the Naval Center for Cost
Analysis that was within 4 percent of the program's estimate, and
conducting a risk analysis that showed a range of estimated costs of
$411 million to $523 million), no risk analysis was performed to
determine the program schedule's risks and associated impact on the
cost estimate. As described earlier in this report, the program has
experienced about 3 years in schedule delays and recently experienced
delays in completing the solution design phase. Therefore, the
importance of conducting a schedule risk analysis and using the results
to assess the variability in the cost estimate is critical for ensuring
a credible cost estimate. Program officials agreed that the program's
schedule is aggressive and risky and that this risk was not assessed in
determining the cost estimate's variability. Without doing so, the
program's cost estimate is not credible, and thus the program is at
risk of cost overruns as a result of schedule delays.
Benefits Estimate Has Been Adjusted to Reflect Questionable Assumptions
and Limited Data:
Forecasting expected benefits over the life of a program is also a key
aspect of economically justifying an investment. OMB guidance[Footnote
37] advocates economically justifying investments on the basis of net
present value. If net present value is positive, then the corresponding
benefit-to-cost ratio will be greater than 1 (and vice versa).[Footnote
38] This guidance also advocates updating the analyses over the life of
the program to reflect material changes in expected benefits, costs,
and risks. Since estimates of benefits can be uncertain because of the
imprecision in both the underlying data and modeling assumptions used,
effects of this uncertainty should be analyzed and reported. By doing
this, informed investment decision making can occur through the life of
the program, and a baseline can be established against which to compare
the accrual of actual benefits from deployed system capabilities.
The original benefit estimate for the first increment was based on
questionable assumptions and insufficient data from comparable
programs. The most recent economic analysis, dated January 2007,
includes monetized, yearly benefit estimates for fiscal years 2010-2019
in three key areas--inventory reductions, reductions in inventory
carrying costs, and improvements in maintenance processes.
Collectively, these benefits totaled about $2.89 billion (not risk-
adjusted). However, these calculations were made using questionable
assumptions and limited data. For example,
* The total value of the Marine Corps inventory needed to calculate
inventory reductions and reductions in carrying costs could not be
determined because of limitations with existing logistic systems.
* The cost savings resulting from improvements in maintenance processes
were calculated based on assumptions from an ERP implementation in the
commercial sector that, according to program officials, is not
comparable in scope to GCSS-MC.
To account for the uncertainty inherent in the benefits estimate, the
program office performed a Monte Carlo simulation.[Footnote 39]
According to the program office, this risk analysis generated a
discounted and risk-adjusted benefits estimate of $1.04 billion. As a
result of the $1.85 billion adjustment to estimated benefits, the
program office has a more realistic benefit baseline against which to
compare the accrual of actual benefits from deployed system
capabilities.
EVM Has Not Been Adequately Implemented:
The program office has elected to implement EVM, which is a proven
means for measuring program progress and thereby identifying potential
cost overruns and schedule delays early, when they can be minimized. In
doing so, it has adopted a tailored EVM approach that focuses on
schedule. However, this schedule-focused approach has not been
effectively implemented because it is based on a baseline schedule that
was not derived using key schedule estimating practices. According to
program officials, the schedule was driven by an aggressive program
completion date established in response to direction from oversight
entities to complete the program as soon as possible. As a result, they
said that following these practices would have delayed this completion
date. Regardless, this means that the schedule baseline is not
reliable, and progress will likely not track to the schedule.
A Tailored EVM Approach Is Being Used to Measure Program Progress:
The program office has adopted a tailored approach to performing EVM
because of the contract type being used. As noted earlier, the contract
types associated with GCSS-MC integration and implementation vary, and
include, for example, firm-fixed-price contracts[Footnote 40] and time-
and-materials contracts. Under a firm-fixed-price contract, the price
is not subject to any adjustment on the basis of the contractor's cost
experience in performing the contract. For a time-and-materials
contract, supplies or services are acquired on the basis of (1) an
undefined number of direct labor hours that are paid at specified fixed
hourly rates and (2) actual cost for materials.
According to DOD guidance,[Footnote 41] EVM is generally not encouraged
for firm-fixed-price, level of effort, and time-and-material contracts.
[Footnote 42] In these situations, the guidance states that programs
can use a tailored EVM approach in which an integrated master schedule
(IMS) is exclusively used to provide visibility into program
performance.
DON has chosen to implement this tailored EVM approach on GCSS-MC. In
doing so, it is measuring progress against schedule commitments, and
not cost commitments, using an IMS for each program phase. According to
program officials, the IMS describes and guides the execution of
program activities. Regardless of the approach used, effective
implementation depends on having a reliable IMS.
Schedule Baseline Was Not Derived in Accordance with Key Estimating
Practices:
The success of any program depends in part on having a reliable
schedule specifying when the program's set of work activities will
occur, how long they will take, and how they are related to one
another. As such, the schedule not only provides a road map for the
systematic execution of a program, but it also provides the means by
which to gauge progress, identify and address potential problems, and
promote accountability. Our research has identified nine practices
associated with effective schedule estimating.[Footnote 43] These
practices are (1) capturing key activities, (2) sequencing key
activities, (3) assigning resources to key activities, (4) integrating
key activities horizontally and vertically, (5) establishing the
duration of key activities, (6) establishing the critical path for key
activities, (7) identifying "float time"[Footnote 44] between key
activities, (8) distributing reserves to high-risk activities, and (9)
performing a schedule risk analysis.
The current IMS for the solution design and transition-to-build phase
of the first increment was developed using some of these practices.
However, it does not reflect several practices that are fundamental to
having a schedule baseline that provides a sufficiently reliable basis
for measuring progress and forecasting slippages. To the program
office's credit, its IMS captures and sequences key activities required
to complete the project, integrates the tasks horizontally, and
identifies the program's critical path. However, the program office is
not monitoring the actual durations of scheduled activities so that it
can address the impact of any deviations on later scheduled activities.
Moreover, the schedule does not adequately identify the resources
needed to complete the tasks and is not integrated vertically, meaning
that multiple teams executing different aspects of the program cannot
effectively work to the same master schedule. Further, the IMS does not
adequately mitigate schedule risk by identifying float time between key
activities, introducing schedule reserve for high-risk activities, or
including the results of a schedule risk analysis. See table 5 for the
results of our analyses relative to each of the nine practices.
Table 5: IMS Satisfaction of Nine Schedule Estimating Key Practices:
Practice: Capturing key activities;
Explanation: The schedule should reflect all key activities (e.g.,
steps, events, outcomes) as defined in the program's work breakdown
structure, to include activities to be performed by both the government
and its contractors;
Satisfied?[A]: Yes;
GAO analysis: The program's schedule reflects both government and
contractor activities, such as building and testing of the software
components, as well as key milestones for measuring progress.
Practice: Sequencing key activities;
Explanation: The schedule should be planned so that it can meet
critical program dates. To meet this objective, key activities need to
be logically sequenced in the order that they are to be carried out. In
particular, activities that must finish prior to the start of other
activities (i.e., predecessor activities), as well as activities that
cannot begin until other activities are completed (i.e., successor
activities) should be identified. By doing so, interdependencies among
activities that collectively lead to the accomplishment of events or
milestones can be established and used as a basis for guiding work and
measuring progress;
Satisfied?[A]: Yes;
GAO analysis: The schedule includes the sequencing of key activities,
meaning that it includes both the predecessor and successor activities
and thus establishes interdependencies among the activities that form
the basis for guiding work and measuring progress.
Practice: Establishing the duration of key activities;
Explanation: The schedule should realistically reflect how long each
activity will take to execute. In determining the duration of each
activity, the same rationale, historical data, and assumptions used for
cost estimating should be used for schedule estimating. Further, these
durations should be as short as possible, and they should have specific
start and end dates. Excessively long periods needed to execute an
activity should prompt further decomposition of the activity so that
shorter execution durations will result. The schedule should be
continually monitored to determine when forecasted completion dates
differ from the planned dates, which can be used to determine whether
schedule variances will affect downstream work;
Satisfied?[A]: Partially;
GAO analysis: The schedule establishes the durations of key activities
based on government and contractor opinions, as well as historical data
from the contractor's experience. However, the program office does not
monitor the actual start and completion dates of work activities so
that the impact of deviations on downstream scheduled work can be
proactively addressed. Program officials agreed that they have not been
tracking actual start and end dates, but stated that they intend to do
so for future phases.
Practice: Assigning resources to key activities;
Explanation: The schedule should reflect what resources (e.g., labor,
material, and overhead) are needed to do the work, whether all required
resources will be available when they are needed, and whether any
funding or time constraints exist;
Satisfied?[A]: Partially;
GAO analysis: The schedule allocated some, but not all, resources to
all key activities. For example, it did not allocate labor hours and
materials. Program officials stated that these resources are assigned
to more detailed activities at the team level. However, they have yet
to provide adequate documentation to show that this is occurring.
Practice: Integrating key activities horizontally and vertically;
Explanation: The schedule should be horizontally integrated, meaning
that it should link the products and outcomes associated with already
sequenced activities. These links are commonly referred to as
"handoffs" and serve to verify that activities are arranged in the
right order to achieve aggregated products or outcomes. The schedule
should also be vertically integrated, meaning that traceability exists
among varying levels of activities and supporting tasks and subtasks.
Such mapping or alignment among levels enables different groups to work
to the same master schedule;
Satisfied?[A]: Partially;
GAO analysis: The schedule is horizontally integrated, meaning that the
activities across the multiple teams are arranged in the right order to
achieve aggregated products or outcomes. However, the schedule is not
vertically integrated, meaning that traceability does not exist among
varying levels of activities. Program officials stated that team-level
schedules can be traced vertically to the IMS, but they have not
provided adequate documentation to show that this is occurring.
Practice: Establishing the critical path for key activities;
Explanation: Using scheduling software, the critical path--the longest
duration path through the sequenced list of key activities--should be
identified. The establishment of a program's critical path is necessary
for examining the effects of any activity slipping along this path.
Potential problems that might occur along or near the critical path
should also be identified and reflected in the scheduling of the time
for high-risk activities (see next);
Satisfied?[A]: Yes;
GAO analysis: The program's critical path has been defined using
scheduling software and includes, among other things, the preparation
for testing activities and the execution of six test scenarios.
Practice: Identifying float time between key activities;
Explanation: The schedule should identify float time--the time that a
predecessor activity can slip before the delay affects successor
activities--so that schedule flexibility can be determined. As a
general rule, activities along the critical path typically have the
least amount of float time;
Satisfied?[A]: No;
GAO analysis: The program office could not identify the amount of float
time allocated to key activities to account for potential problems that
might occur along or near the critical path. Therefore, if the schedule
for an activity near the critical path were to slip, it is likely that
the delay would impact the critical path.
Practice: Distributing reserves to high-risk activities;
Explanation: The baseline schedule should include a buffer or a reserve
of extra time. Schedule reserve for contingencies should be calculated
by performing a schedule risk analysis (see next). As a general rule,
the reserve should be applied to high-risk activities, which are
typically found along the critical path;
Satisfied?[A]: No;
GAO analysis: The program office did not allocate schedule reserve for
high-risk activities on the critical path. Schedule reserve should be
calculated by performing a schedule risk analysis, and allocating
additional reserve for those activities identified as high risk.
Without this reserve, any delays on activities on the critical path
increase the risk of delays to the scheduled completion date.
Practice: Performing a schedule risk analysis;
Explanation: A schedule risk analysis should be performed using
statistical techniques to predict the level of confidence in meeting a
program's completion date. This analysis focuses not only on critical
path activities but also on activities near the critical path, since
they can potentially affect program status;
Satisfied?[A]: No;
GAO analysis: The program office did not perform a schedule risk
analysis to determine the level of confidence in meeting the program's
activities and its completion date.
Sources: GAO analysis of Marine Corps data.
[A] "Yes" means that the program office provided documentation
demonstrating satisfaction of the criterion. "Partially" means that the
program office provided documentation demonstrating satisfaction of
part of the criterion. "No" means that the program office has yet to
provide documentation demonstrating satisfaction of the criterion.
[End of table]
According to program officials, they intend to begin monitoring actual
activity start and completion dates so that they can proactively adjust
later scheduled activities that are affected by deviations. However,
they do not plan to perform the three practices related to
understanding and managing schedule risk because doing so would likely
extend the program's completion date, and they set this date to be
responsive to direction from DOD and DON oversight entities to complete
the program as soon as possible. In our view, not performing these
practices does not allow the inherent risks in meeting this imposed
completion date to be proactively understood and addressed. The
consequence of omitting these practices is a schedule that does not
provide a reliable basis for performing EVM.
Requirements Management Weakness Was Recently Corrected:
Well-defined and managed requirements are recognized by DOD guidance as
essential and can be viewed as a cornerstone of effective system
acquisition. One aspect of effective requirements management is
requirements traceability.[Footnote 45] By tracing requirements both
backward from system requirements to higher level business or
operational requirements and forward to system design specifications
and test plans, the chances of the deployed product satisfying
requirements are increased, and the ability to understand the impact of
any requirement changes, and thus make informed decision about such
changes, is enhanced.
The program office recently strengthened its requirements traceability.
In November 2007, and again in February 2008, the program office was
unable to demonstrate for us that it could adequately trace its 1,375
system requirements to both design specifications and test
documentation. Specifically, the program office was at that time using
a tool called DOORS®, which if implemented properly, allows each
requirement to be linked from its most conceptual definition to its
most detailed definition, as well as to design specifications and test
cases. In effect, the tool maintains the linkages among requirement
documents, design documents, and test cases even if requirements
change. However, the system integration contractor was not using the
tool. Instead the contractor was submitting its 244 work products,
[Footnote 46] accompanied by spreadsheets that linked each work product
to one or more system requirements and test cases. The program office
then had to verify and validate the spreadsheets and import and link
each work product to the corresponding requirement and test case in
DOORS. Because of the sheer number of requirements and work products
and its potential to impact cost, schedule, and performance, the
program designated this approach as a medium risk. It later closed the
risk because the proposed mitigation strategy failed to mitigate it,
and it was realized as a high-priority program issue (i.e., problem).
According to program officials, this requirements traceability approach
resulted in time-consuming delays in approving the design work products
and importing and establishing links between these products and the
requirements in DOORS, in part because the work products were not
accompanied by complete spreadsheets that established the traceability.
As a result, about 30 percent of the contractor's work products had yet
to be validated, approved, and linked to requirements when the design
phase was originally scheduled to be complete. Officials stated that
the contractor was not required to use DOORS because it was not
experienced with this tool and becoming proficient with it would have
required time and resources, thereby increasing both the program's cost
and schedule. Ironically, however, not investing the time and resources
to address the limitations in the program's traceability approach
contributed to recent delays in completing the solution design
activities, and additional resources had to be invested to address its
requirements traceability problems.
The program office now reports that it can trace requirements backward
and forward. In April 2008, we verified this by tracing 60 out of 61
randomly sampled requirements backward to system requirements and
forward to approved design specifications and test plans. Program
officials explained that the reason that we could not trace the one
requirement was that the related work products had not yet been
approved. In addition, they stated that there were additional work
products that had yet to be finalized and traced.
Without adequate traceability, the risk of a system not performing as
intended and requiring expensive rework is increased. To address its
requirements traceability weakness, program officials told us that they
now intend to require the contractor to use DOORS during the next phase
of the program (build and test). If implemented effectively, the new
process should address previous requirements traceability weaknesses
and thereby avoid a repeat of past problems.
Not All Program Risks Have Been Adequately Managed:
Proactively managing program risks is a key acquisition management
control and, if done properly, can greatly increase the chances of
programs delivering promised capabilities and benefits on time and
within budget. To the program office's credit, it has defined a risk
management process that meets relevant guidance. However, it has not
effectively implemented the process for all identified risks. As a
result, these risks have become actual program problems that have
impacted the program's cost, schedule, and performance commitments.
DOD acquisition management guidance,[Footnote 47] as well as other
relevant guidance,[Footnote 48] advocates identifying facts and
circumstances that can increase the probability of an acquisition's
failing to meet cost, schedule, and performance commitments and then
taking steps to reduce the probability of their occurrence and impact.
In brief, effective risk management consists of: (1) establishing a
written plan for managing risks; (2) designating responsibility for
risk management activities; (3) encouraging project-wide participation
in the identification and mitigation of risks; (4) defining and
implementing a process that provides for the identification, analysis,
and mitigation of risks; and (5) examining the status of identified
risks in program milestone reviews.
The program office has developed a written plan for managing risks, and
established a process that together provide for the above cited risk
management practices, and it has followed many key aspects of its plan
and process. For example,
* The Program Manager has been assigned overall responsibility for
managing risks. Also, individuals have been assigned ownership of each
risk, to include conducting risk analyses, implementing mitigation
strategies, and working with the risk support team.[Footnote 49]
* The plan and process encourage project-wide participation in the
identification and mitigation of risks by allowing program staff to
submit a risk for inclusion in a risk database[Footnote 50] and take
ownership of the risk and the strategy for mitigating it. In addition,
stakeholders can bring potential risks to the Program Manager's
attention through interviews, where potential risks are considered and
evaluated.
* The program office has thus far identified and categorized individual
risks. As of December 2007, the risk database contained 27 active
risks--2 high, 15 medium, and 10 low.[Footnote 51]
* Program risks are considered during program milestone reviews.
Specifically, our review of documentation for the Design Readiness
Review,[Footnote 52] a key decision point during the system development
and demonstration phase leading up to a Milestone C decision, showed
that key risks were discussed. Furthermore, the Program Manager reviews
program risks' status through a risk watch list[Footnote 53] and
bimonthly risk briefings.
However, the program office has not consistently followed other aspects
of its process. For example, it did not perform key practices for
identifying and managing schedule risks, such as conducting a schedule
risk assessment and building in reserve time to its schedule. In
addition, mitigation steps for several key risks were either not
performed in accordance with the risk management strategy, or risks
that were closed as having been mitigated were later found to be actual
program issues (i.e., problems).
For 25 medium risks in the closed risk database, as of February 2008, 4
were closed because mitigation steps were not performed in accordance
with the strategy and the risks ultimately became actual issues.
Examples from these medium risks are as follows:
* In one case, the mitigation strategy was for the contractor to
deliver certain design documents that were traced to system
requirements and to do so before beginning the solution build phase.
The design documents, however, were not received in accordance with the
mitigation strategy. Specifically, program officials told us that the
design documents contained inaccuracies or misinterpretations of the
requirements and were not completed on time because of the lack of
resources to correct these problems. As a result, the program
experienced delays in completing its solution design activities.
* In another case, the mitigation strategy included creating the
documentation needed to execute the contract for monitoring the build
phase activities. However, the mitigation steps were not performed due
to, among other things, delays in approving the contractual approach.
As a result, the risk became a high-priority issue in February 2008.
According to a program issue report, the lack of a contract to monitor
system development progress may result in unnecessary rework and thus
additional program cost overruns, schedule delays, and performance
shortfalls.
Four of the same 25 medium risks were retired because key mitigation
steps for each one were implemented, but the strategies proved
ineffective, and the risks became actual program issues. Included in
these 4 risks were the following:
* In one case, the program office closed a risk regarding data exchange
with another DON system because key mitigation steps to establish
exchange requirements were fully implemented. However, in February
2008, a high-priority issue was identified regarding the exchange of
data with this system. According to program officials, the risk was
mitigated to the fullest extent possible and closed based on the
understanding that continued evaluation of data exchange requirements
would be needed. However, because the risk was retired, this evaluation
did not occur.
* In another case, a requirements management risk was closed on the
basis of having implemented mitigation steps, which involved
establishing a requirements management process, including having
complete requirements traceability spreadsheets. However, although
several of the mitigation steps were not fully implemented, the risk
was closed on the basis of what program officials described as an
understanding reached with the contractor regarding the requirements
management process. Several months later, a high-priority issue
concerning requirements traceability was identified because the program
office discovered that the contractor was not adhering to the
understanding.
Unless risk mitigation strategies are monitored to ensure that they are
fully implemented and that they produce the intended outcomes, and
additional mitigation steps are taken when they are not, the program
office will continue to be challenged in preventing risks from
developing into actual cost, schedule, and performance problems.
Important Aspects of System Quality and Program Maturity Are Not Being
Measured:
Effective management of programs like GCSS-MC depends in part on the
ability to measure the quality of the system being acquired and
implemented. Two measures of system quality are trends in (1) the
number of unresolved severe system defects and (2) the number of
unaddressed high-priority system change requests.
GCSS-MC documentation recognizes the importance of monitoring such
trends. Moreover, the program office has established processes for (1)
collecting and tracking data on the status of program issues, including
problems discovered during early test events, and (2) capturing data on
the status of requests for changes to the system. However, its
processes do not provide the full complement of data that are needed to
generate a reliable and meaningful picture of trends in these areas. In
particular, data on problems and change request priority levels and
closure dates are either not captured or not consistently maintained.
Further, program office oversight of contractor-identified issues or
defects is limited. Program officials acknowledged these data
limitations, but they stated that oversight of contractor-identified
issues is not their responsibility. Without tracking trends in key
indicators, the program office cannot adequately understand and report
to DOD decision makers whether GCSS-MC's quality and stability are
moving in the right direction.
Data to Understand Trends in Program Problems Are Limited:
Program guidance and related best practices[Footnote 54] encourage
trend analysis and the reporting of system defects and program problems
as measures or indicators of system quality and program maturity. As we
have previously reported,[Footnote 55] these indicators include trends
in the number of unresolved problems according to their significance or
priority.
To the program office's credit, it collects and tracks what it calls
program issues, which are problems identified by program office staff
or the system integrator that are process, procedure, or management
related. These issues are contained in the program's Issues-Risk
Management Information System (I-RMIS). Among other things, each issue
in I-RMIS is to have an opened and closed date and an assigned priority
level of high, medium, or low. In addition, the integration contractor
tracks issues that its staff identifies related to such areas as system
test defects. These issues are contained in the contractor's Marine
Corps Issue Tracking System (MCITS). Each issue in MCITS is to have a
date when it was opened and is to be assigned a priority on a scale of
1-5. According to program officials, the priority levels are based on
guidance from the Institute of Electrical and Electronics Engineers
(IEEE). (See table 6 for a description of each priority level.)
Table 6: MCITS Priority Levels:
Priority: 1;
Description:
* Prevents the accomplishment of an essential capability;
* Jeopardizes safety, security, or other requirement designated
critical.
Priority: 2;
Description:
* Adversely affects the accomplishment of an essential capability, and
no work-around solution is known;
* Adversely affects technical, cost, or schedule risks to the project
or to life cycle support of the system, and no work-around solution is
known.
Priority: 3;
Description:
* Adversely affects the accomplishment of an essential capability, but
a work-around solution is known;
* Adversely affects technical, cost, or schedule risks to the project
or to life cycle support of the system, but a work-around solution is
known.
Priority: 4;
Description:
* Results in user/operator inconvenience or annoyance but does not
affect a required operational or mission-essential capability;
* Results in inconvenience or annoyance for development or maintenance
personnel but does not prevent the accomplishment of the
responsibilities of those personnel.
Priority: 5;
Description:
* Any other effect.
Sources: GCSS-MC and IEEE.
[End of table]
However, neither I-RMIS nor MCITS contain all the data needed to
reliably produce key measures or indicators of system quality and
program maturity. Examples of these limitations are as follows:
* For I-RMIS, the program office has not established a standard
definition of the priority levels used. Rather, according to program
officials, each issue owner is allowed to assign a priority based on
the owner's definition of what high, medium, and low mean. By not using
standard priority definitions for categorizing issues, the program
office cannot ensure that it has an accurate and useful understanding
of the problems it is facing at any given time, and it will not know if
it is addressing the highest priority issues first.
* For MCITS, the integration contractor does not track closure dates
for all issues. For example, as of April 2008, over 30 percent of the
closed issues did not have closure dates. This is important because it
limits the contractor's ability to understand trends in the number of
high-priority issues that are unresolved. Program officials
acknowledged the need to have closure dates for all closed issues and
stated that they intend to correct this. If it is not corrected, the
program office will not be able to create a reliable measure of system
quality and program maturity.
Compounding the above limitations in MCITS data is the program office's
decision not to use contractor-generated reports that are based on
MCITS data. Specifically, reports summarizing MCITS issues are posted
to a SharePoint[Footnote 56] site for the program office to review.
However, program officials stated that they do not review these reports
because the MCITS issues are not their responsibility, but the
contractor's. However, without tracking and monitoring contractor-
identified issues, which include such things as having the right skill-
sets and having the resources to track and monitor issues captured in
separate databases, the program office is missing an opportunity to
understand whether proactive action is needed to address emerging
quality shortfalls in a timely manner.
Data to Understand Trends in Changes to the System Are Limited:
Program guidance and related best practices[Footnote 57] encourage
trend reporting of change requests as measures or indicators of system
stability and quality. These indicators include trends in the number
and priority of approved changes to the system's baseline functional
and performance capabilities that have yet to be resolved.
To its credit, the program office collects and tracks changes to the
system, which can range from minor or administrative changes to more
significant changes that propose or impact important system
functionality. These changes can be identified by either the program
office or the contractor, and they are captured in a master change
request spreadsheet. Further, the changes are to be prioritized
according to the level described in table 7, and the dates that change
requests are opened and closed are to be recorded.
Table 7: Change Request Priorities:
Priority: 1;
Description:
* Prevents the accomplishment of an essential capability;
* Jeopardizes safety, security, or other requirement designated
critical.
Priority: 2;
Description:
* Adversely affects the accomplishment of an essential capability, and
no work-around solution is known;
* Adversely affects technical, cost, or schedule risks to the project
or to life cycle support of the system, and no work-around solution is
known.
Priority: 3;
Description:
* Adversely affects the accomplishment of an essential capability, but
a work-around solution is known;
* Adversely affects technical, cost, or schedule risks to the project
or to life cycle support of the system, but a work-around solution is
known.
Priority: 4;
Description:
* Results in user/operator inconvenience or annoyance but does not
affect a required operational or mission-essential capability;
* Results in inconvenience or annoyance for development or maintenance
personnel but does not prevent the accomplishment of the
responsibilities of those personnel.
Priority: 5;
Description:
* Any other effect.
Sources: GCSS-MC and IEEE.
[End of table]
However, the change request master spreadsheet does not contain the
data needed to reliably produce key measures or indicators of system
stability and quality. Examples of these limitations are as follows:
* The program office has not prioritized proposed changes or managed
these changes according to their priorities. For example, of the 572
change requests as of April 2008, 171 were assigned a priority level,
and 401 were not. Of these 171, 132 were categorized as priority 1.
Since then, the program office has temporarily recategorized the 401
change requests to priority 3 until each one's priority can be
evaluated. The program office has yet to establish a time frame for
doing so.
* The dates that change requests are resolved are not captured in the
master spreadsheet. Rather, program officials said that these dates are
in the program's IMS and are shown there as target implementation
dates. While the IMS does include the dates changes will be
implemented, these dates are not actual dates, and they are not used to
establish trends in unresolved change requests.
Without the full complement of data needed to monitor and measure
change requests, the program office cannot know and disclose to DOD
decision makers whether the quality and stability of the system are
moving in the right direction.
Conclusions:
DOD's success in delivering large-scale business systems, such as GCSS-
MC, is in large part determined by the extent to which it employs the
kind of rigorous and disciplined IT management controls that are
reflected in DOD policies and related guidance. While implementing
these controls does not guarantee a successful program, it does
minimize a program's exposure to risk and thus the likelihood that it
will fall short of expectations. In the case of GCSS-MC, living up to
expectations is important because the program is large, complex, and
critical to supporting the department's warfighting mission.
The department has not effectively implemented a number of essential IT
management controls on GCSS-MC, which has already contributed to
significant cost overruns and schedule delays, and has increased the
program's risk going forward of not delivering a cost-effective system
solution and not meeting future cost, schedule, capability, and benefit
commitments. Moreover, GCSS-MC could be duplicating the functionality
of related systems and may be challenged in interoperating with these
systems because compliance with key aspects of DOD's federated BEA has
not been demonstrated. Also, the program's estimated return on
investment, and thus the economic basis for pursing the proposed system
solution, is uncertain because of limitations in how the program's cost
estimate was derived, raising questions as to whether the nature and
level of future investment in the program needs to be adjusted. In
addition, the program's schedule was not derived using several key
schedule estimating practices, which impacts the integrity of the cost
estimate and precludes effective implementation of EVM. Without
effective EVM, the program cannot reliably gauge progress of the work
being performed so that shortfalls can be known and addressed early,
when they require less time and fewer resources to overcome. Another
related indicator of progress, trends in system problems and change
requests, also cannot be gauged because the data needed to do so are
not being collected. Collectively, these weaknesses have already helped
to push back the completion of the program's first increment by more
than 3 years and added about $193 million in costs, and they are
introducing a number of risks that, if not effectively managed, could
further impact the program. However, whether these risks will be
effectively managed is uncertain because the program has not always
followed its defined risk management process and, as a result, has
allowed yesterday's potential problems to become today's actual cost,
schedule, and performance problems.
While the program office is primarily responsible for ensuring that
effective IT management controls are implemented on GCSS-MC, other
oversight and stakeholder organizations share some responsibility. In
particular, even though the program office has not demonstrated its
alignment with the federated BEA, it nevertheless followed established
DOD architecture compliance guidance and used the related compliance
assessment tool in assessing and asserting its compliance. The root
cause for not demonstrating compliance thus is not traceable to the
program office, but rather is due to, among other things, the
compliance guidance and tool being limited, and the program's oversight
entities not validating the compliance assessment and assertion. Also,
even though the program's cost estimate was not informed by the cost
experiences of other ERP programs of the same scope, the program office
is not to blame because the root cause for this is that the Defense
Cost and Resource Center has not maintained a standardized cost element
structure for its ERP programs and a historical database of ERP program
costs for program's like GCSS-MC to use. In contrast, other weaknesses
are within the program office's control, as evidenced by its positive
actions to address the requirements traceability shortcomings that we
brought to its attention during of the course of our work and its well-
defined risk management process.
All told, this means that addressing the GCSS-MC IT management control
weaknesses require the combined efforts of the various DOD
organizations that share responsibility for defining, justifying,
managing, and overseeing the program. By doing so, the department can
better assure itself that GCSS-MC will optimally support its mission
operations and performance goals and will deliver promised capabilities
and benefits, on time and within budget.
Recommendations for Executive Action:
To ensure that each GCSS-MC system increment is economically justified
on the basis of a full and reliable understanding of costs, benefits,
and risks, we recommend that the Secretary of Defense direct the
Secretary of the Navy to ensure that investment in the next acquisition
phase of the program's first increment is conditional upon fully
disclosing to program oversight and approval entities the steps under
way or planned to address each of the risks discussed in this report,
including the risk of not being architecturally compliant and being
duplicative of related programs, not producing expected mission
benefits commensurate with reliably estimated costs, not effectively
implementing EVM, not mitigating known program risks, and not knowing
whether the system is becoming more or less mature and stable. We
further recommend that investment in all future GCSS-MC increments be
limited if the management control weaknesses that are the source of
these risks, and which are discussed in this report, have not been
fully addressed.
To address each of the IT management control weaknesses discussed in
this report, we are also making a number of additional recommendations.
However, we are not making recommendations for the architecture
compliance weaknesses discussed in this report because we have a
broader review of DON program compliance to the BEA and DON enterprise
architecture that will be issued shortly and will contain appropriate
recommendations.
To improve the accuracy of the GCSS-MC cost estimate, as well as other
cost estimates for the department's ERP programs, we recommend that the
Secretary of Defense direct the appropriate organization within DOD to
collaborate with relevant organizations to standardize the cost element
structure for the department's ERP programs and to use this standard
structure to maintain cost data for its ERP programs, including GCSS-
MC, and to use this cost data in developing future cost estimates.
To improve the credibility of the GCSS-MC cost estimate, we recommend
that the Secretary of Defense direct the Secretary of the Navy, through
the appropriate chain of command, to ensure that the program's current
economic analysis is adjusted to reflect the risks associated with it
not reflecting cost data for comparable ERP programs, and otherwise not
having been derived according to other key cost estimating practices,
and that future updates to the GCSS-MC economic analysis similarly do
so.
To enhance GCSS-MC's use of EVM, we recommend that the Secretary of
Defense direct the Secretary of the Navy, through the appropriate chain
of command, to ensure that the program office (1) monitors the actual
start and completion dates of work activities performed so that the
impact of deviations on downstream scheduled work can be proactively
addressed; (2) allocates resources, such as labor hours and material,
to all key activities on the schedule; (3) integrates key activities
and supporting tasks and subtasks; (4) identifies and allocates the
amount of float time needed for key activities to account for potential
problems that might occur along or near the schedule's critical path;
(5) performs a schedule risk analysis to determine the level of
confidence in meeting the program's activities and completion date; (6)
allocates schedule reserve for high-risk activities on the critical
path; and (7) discloses the inherent risks and limitations associated
with any future use of the program's EVM reports until the schedule has
been risk-adjusted.
To improve GCSS-MC management of program risks, we recommend that the
Secretary of Defense direct the Secretary of the Navy, through the
appropriate chain of command, to ensure that the program office (1)
adds each of the risks discussed in this report to its active inventory
of risks, (2) tracks and evaluates the implementation of mitigation
plans for all risks, (3) discloses to appropriate program oversight and
approval authorities whether mitigation plans have been fully executed
and have produced the intended outcome(s), and (4) only closes a risk
if its mitigation plan has been fully executed and produced the
intended outcome(s).
To strengthen GCSS-MC system quality measurement, we recommend that the
Secretary of Defense direct the Secretary of the Navy, through the
appropriate chain of command, to ensure that the program office (1)
collects the data needed to develop trends in unresolved system defects
and change requests according to their priority and severity and (2)
discloses these trends to appropriate program oversight and approval
authorities.
Agency Comments and Our Evaluation:
In written comments on a draft of this report, signed by the Deputy
Under Secretary of Defense (Business Transformation) and reprinted in
appendix II, the department stated that it concurred with two of our
recommendations and partially concurred with the remaining five. In
general, the department partially concurred because it said that
efforts were either under way or planned that will address some of the
weaknesses that these recommendations are aimed at correcting. For
example, the department stated that GCSS-MC will begin to use a
recently developed risk assessment tool that is expected to assist
programs in identifying and mitigating internal and external risks.
Further, it said that these risks will be reported to appropriate
department decision makers. We support the efforts that DOD described
in its comments because they are generally consistent with the intent
of our recommendations and believe that if they are fully and properly
implemented, they will go a long way in addressing the management
control weaknesses that our recommendations are aimed at correcting. In
addition, we have made a slight modification to one of these five
recommendations to provide the department with greater flexibility in
determining which organizations should provide for the recommendation's
implementation.
We are sending copies of this report to interested congressional
committees; the Director, Office of Management and Budget; the
Congressional Budget Office; the Secretary of Defense; and the
Department of Defense Office of the Inspector General. We also will
make copies available to others upon request. In addition, the report
will be available at no charge on the GAO Web site at [hyperlink,
http://www.gao.gov].
If you or your staff have any questions about this report, please
contact me at (202) 512-3439 or hiter@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. GAO staff who made major contributions to
this report are listed in appendix III.
Signed by:
Randolph C. Hite:
Director, Information Technology Architecture and Systems Issues:
[End of section]
Appendix I: Objective, Scope, and Methodology:
Our objective was to determine whether the Department of the Navy is
effectively implementing information technology management controls on
the Global Combat Support System-Marine Corps (GCSS-MC). To accomplish
this, we focused on the first increment of GCSS-MC relative to the
following management areas: architectural alignment, economic
justification, earned value management, requirements management, risk
management, and system quality measurement. In doing so, we analyzed a
range of program documentation, such as the acquisition strategy,
program management plan, and Acquisition Program Baseline, and
interviewed cognizant program officials.
To determine whether GCSS-MC was aligned with the Department of
Defense's (DOD) federated business enterprise architecture (BEA), we
reviewed the program's BEA compliance assessments and system
architecture products, as well as versions 4.0, 4.1, and 5.0 of the BEA
and compared them with the BEA compliance requirements described in the
Fiscal Year 2005 National Defense Authorization Act[Footnote 58] and
DOD's BEA compliance guidance and evaluated the extent to which the
compliance assessments addressed all relevant BEA products. We also
determined the extent to which the program-level architecture
documentation supported the BEA compliance assessments. We obtained
documentation, such as the BEA compliance assessments from the GCSS-MC
and Navy Enterprise Resource Planning programs, as well as the Air
Force's Defense Enterprise Accounting and Management System and Air
Force Expeditionary Combat Support System programs. We then compared
these assessments to identify potential redundancies or opportunities
for reuse and determined if the compliance assessments examined
duplication across programs and if the tool that supports these
assessments is being used to identify such duplication. In doing so, we
interviewed program officials and officials from the Department of the
Navy, Office of the Chief Information Officer, and reviewed recent GAO
reports to determine the extent to which the programs were assessed for
compliance against the Department of the Navy enterprise architecture.
We also interviewed program officials and officials from the Business
Transformation Agency and the Department of the Navy, including the
logistics Functional Area Manager, and obtained guidance documentation
from these officials to determine the extent to which the compliance
assessments were subject to oversight or validation.
To determine whether the program had economically justified its
investment in GCSS-MC, we reviewed the latest economic analysis to
determine the basis for the cost and benefit estimates. This included
evaluating the analysis against Office of Management and Budget
guidance and GAO's Cost Assessment Guide.[Footnote 59] In doing so, we
interviewed cognizant program officials, including the Program Manager
and cost analysis team, regarding their respective roles,
responsibilities, and actual efforts in developing and/or reviewing the
economic analysis. We also interviewed officials at the Office of
Program Analysis and Evaluation and Naval Center for Cost Analysis as
to their respective roles, responsibilities, and actual efforts in
developing and/or reviewing the economic analysis.
To determine the extent to which the program had effectively
implemented earned value management, we reviewed relevant
documentation, such the contractor's monthly status reports,
Acquisition Program Baselines, and schedule estimates and compared them
with DOD policies and guidance.[Footnote 60] We also reviewed the
program's schedule estimates and compared them with relevant best
practices[Footnote 61] to determine the extent to which they reflect
key estimating practices that are fundamental to having a reliable
schedule. In doing so, we interviewed cognizant program officials to
discuss their use of best practices in creating the program's current
schedule.
To determine the extent to which the program implemented requirements
management, we reviewed relevant program documentation, such as the
baseline list of requirements and system specifications and evaluated
them against relevant best practices[Footnote 62] to determine the
extent to which the program has effectively managed the system's
requirements and maintained traceability backward to high-level
business operation requirements and system requirements, and forward to
system design specifications, and test plans. To determine the extent
to which the requirements were traceable, we randomly selected 61
program requirements and traced them both backward and forward. This
sample was designed with a 5 percent tolerable error rate at the 95
percent level of confidence, so that, if we found 0 problems in our
sample, we could conclude statistically that the error rate was less
than 5 percent. Based upon the weight of all other factors included in
our evaluation, our verification of 60 out of 61 requirements was
sufficient to demonstrate traceability. In addition, we interviewed
program officials involved in the requirements management process to
discuss their roles and responsibilities for managing requirements.
To determine the extent to which the program implemented risk
management, we reviewed relevant risk management documentation, such as
risk plans and risk database reports demonstrating the status of the
program's major risks and compared the program office's activities with
DOD acquisition management guidance[Footnote 63] and related best
practices.[Footnote 64] We also reviewed the program's mitigation
process with respect to key risks, including 25 medium risks in the
retired risk database that were actively addressed by the program
office, to determine the extent to which these risks were effectively
managed. In doing so, we interviewed cognizant program officials
responsible, such as the Program Manager, Risk Manager, and subject
matter experts to discuss their roles and responsibilities and obtain
clarification on the program's approach to managing risks associated
with acquiring and implementing GCSS-MC.
To determine the extent to which the program is collecting the data and
monitoring trends in the number of unresolved system defects and the
number of unaddressed change requests, we reviewed program
documentation such as the testing strategy, configuration management
policy, test defect reports, change request logs, and issue data logs.
We compared the program's data collection and analysis practices
relative to these areas to program guidance and best practices[Footnote
65] to determine the extent to which the program is measuring important
aspects of system quality. We also interviewed program officials such
as system developers, relevant program management staff, and change
control managers to discuss their roles and responsibilities for system
quality measurement.
We conducted our work at DOD offices and contractor facilities in the
Washington, D.C., metropolitan area, and Triangle, Va., from June 2007
to July 2008, in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objective. We
believe that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objective.
[End of section]
Appendix II: Comments from the Department of Defense:
Office Of The Under Secretary Of Defense:
Acquisition Technology And Logistics:
3000 Defense Pentagon:
Washington, DC 20301-3000:
July 21, 2008:
Mr. Randolph C. Hite:
Director, Information Technology Architecture and Systems Issues:
U.S. Government Accountability Office:
441 G Street, N.W.
Washington, DC 20548:
Dear Mr. Hite:
This is the Department of Defense (DoD) response to the GAO draft
report GAO-08-822, "DOD Business Systems Modernization: Key Marine
Corps System Acquisition Needs to be Better Justified, Defined and
Managed," dated June 9, 2008 (GAO Code 310648). Detailed comments on
the report recommendations are enclosed.
DoD concurred with two recommendations and partially concurred with the
remaining five. Overall, the Department has already taken steps to
address some of GAO's findings. including updating guidance for
Business Enterprise Architecture (BEA) 5.0 compliance. Further, the
Department's Business Capability Lifecycle (BCL) is expected to provide
stakeholders at the program-, Component-, and governance-levels with
additional visibility into program risks via the Enterprise Risk
Assessment Methodology (ERAM) as well as further visibility into
existing management control issues. BCL will also support the
collection of cost data for Enterprise Resource Planning (ERP) for
programs' future cost estimation purposes.
We welcome the GAO's insight on the Department's progress with its
business transformation efforts and continue to value our partnership.
Sincerely,
Signed by:
Paul A. Brinkley:
Deputy Under Secretary of Defense (Business Transformation):
Enclosure: As stated:
GAO DRAFT REPORT DATED JUNE 9, 2008:
GAO-08-822 (GAO CODE 310648):
DOD Business Systems Modernization: Key Marine Corps System Acquisition
Needs To Be Better Justified, Defined, And Managed:
Department Of Defense Comments To The GAO Recommendations:
Recommendation 1: The GAO recommends that the Secretary of Defense
direct the Secretary of the Navy to ensure that investment in the next
acquisition phase of the program's first increment is conditional upon
fully disclosing to program oversight and approval entities the steps
underway or planned to address each of the risks discussed in this
report, including the risk of not being architecturally compliant and
being duplicative of related programs, not producing expected mission
benefits commensurate with reliably estimated costs, not effectively
implementing earned value management (EVM), not mitigating known
program risks, and not knowing whether the system is becoming more or
less mature and stable. (Page 54/GAO Draft Report)
DOD Response: Partially Concur.
The Department is already taking steps to promote greater visibility
into program risks through its implementation of the Business
Capability Lifecycle (BCL). Per a May 18, 2007 memo from the Under
Secretary of Defense for Acquisition, Technology and Logistics
(AT&L)/Vice Chair of the Defense Business Systems Management Committee
(DBSMC), the Global Combat Support Systems - Marine Corps (GCSS-MC)
program is one of the programs that will begin to use the Enterprise
Risk Assessment Methodology (ERAM) for the duration of its lifecycle,
which will assist the program in identifying and mitigating internal
and external risks. ERAM findings are shared with appropriate team
members and decision makers within the Component as well as Investment
Review Board (IRB) and DBSMC leadership, and programs utilize ERAM
findings to create risk mitigation plans as needed. Given that future
development of the GCSS-MC capabilities will be achieved via an
incremental approach as part of the Department's BCE process, it is
envisioned that any emerging risk issues will be identified and
addressed.
Additionally, the GCSS-MC Program Office is promoting greater
visibility of risks at the program level by developing and using a
database to track emerging risks for mitigation in future increments,
and the program manager is kept apprised of the status of these risks
through a watch list and regular briefings by the risk owners.
The program is also taking several steps to ensure that the program is
architecturally compliant in accordance with current DoD guidance and
policies. As the Business Enterprise Architecture (BEA) evolves, the
GCSS-MC program has continued to update system maps as required as part
of the acquisition review process. Additionally, as part of the Marine
Corps' overall Logistics Modernization effort, a Logistics Operational
Architecture (LOG OA) was developed to guide future development of
Logistics IT systems used by the Marine Air Ground Task Forces (MAGTFs)
and the supporting establishment organizations. This architecture has
been mapped to the BEA. the USTRANSCOM Deployment and Distribution
Enterprise Architecture and efforts are ongoing to develop a Joint Army-
Marine Corps Integrated Architecture in support of logistics
operations. GCSS-MC, as the centerpiece of the Logistics Modernization
effort, has been aligned to the LOG OA to provide a basis for
identifying capability gaps and overlaps in architecture that are not
being used to develop future capability requirements.
The Business Transformation Agency (BTA) will also continue to revise
the BEA compliance guidance and further define program-level
architecture products to provide the Components with clearer
requirements. For example, the latest issuance of the BEA compliance
guidance in May 2008 (attached) updated the BEA compliance requirements
to enable Components to assert to the BEA 5.0 data synonym and
attributes that were developed in support of Enterprise Data
Initiatives, and to enable the IRBs to enforce program compliance to a
more defined set of requirements that promote interoperability and data
standardization. The new guidance document will better assist the
Components as they map their programs to BEA 5.0 requirements in future
increments.
Recommendation 2: The GAO recommends that the Secretary of Defense
direct the Secretary of the Navy to ensure that investment in all
future Global Combat Support Systems-Marine Corps (GCSS-MC) increments
be limited if the management control weaknesses that arc the source of
these risks, and which are discussed in this report, have not been
fully addressed. (Page 54/GAO Draft Report)
DOD Response: Partially concur.
The Department is already taking steps to address some of the
management control issues GAO noted in the report. As previously
discussed, the BEA compliance guidance was recently updated to provide
Components with additional clarity on requirements for asserting
compliance to BEA 5.0. In addition, the implementation of BCL is
expected to provide the program offices, Components, and IRB/DBSMC
leadership with enhanced visibility of risks. This visibility will
allow the Department to examine the root causes of identified risks and
take action as needed to make improvements in its management controls
as business transformation efforts continue.
Going forward, the GCSS-MC Program Office will address the correction
of all identified weaknesses prior to seeking investment decisions for
future increments in accordance with all existing Department of the
Navy and Department of Defense guidance and policies, including IRB
policy and Guidance.
Recommendation 3: The GAO recommends that the Secretary of Defense
direct the Director of the Office of Program Analysis and Evaluation to
ensure the Defense Cost and Resource Center, in collaboration with
other relevant organizations, standardize the cost element structure
for the department's Enterprise Resource Planning (ERP) programs and to
use this standard structure to maintain cost data for its ERP programs,
including GCSS-MC, and to use this cost data in developing future cost
estimates. (Page 55/GAO Draft Report)
DOD Response: Partially Concur.
The Department concurs with GAO's recommendation that actual cost data
for ERP programs should be maintained for use in developing future cost
estimates. Assembling this data would be best achieved by taking steps
to standardize the Work Breakdown Structure (WBS) for ERPs. The BTA
will support the Office of AT&L/Acquisition Resources and Analysis
(ARA), the organization responsible for issuing DoD's handbook on work
breakdown structures for defense materiel items (MIL-HDBK-881 A), in
developing the standard WBS for ERP systems.
In addition, the Department intends to track and maintain cost data for
ERPs through BCL's Integrated Management Information Environment
(IMIE). Since BCL unifies reporting requirements of the Defense
Acquisition System (DAS) 5000-series policies, the Chairman of the
Joint Chiefs of Staff Instruction (CJCSI) 3170, and the IRB Concept of
Operations (CONOPS) for business systems, the required information is
presented to the IRB/DBSMC governance structure, including the
program's economic analysis. As ERPs undergo the BCL process and more
business cases are submitted to the IRBs, cost data will be captured in
IMIE for use in developing future cost estimates, and in developing
economic analysis guidance for both acquisition and Clinger-Cohen Act
compliance decisions.
Recommendation 4: The GAO recommends that the Secretary of Defense
direct the Secretary of the Navy, through the appropriate chain of
command, to ensure that the program's current economic analysis is
adjusted to reflect the risks associated with it not reflecting cost
data for comparable ERP programs, and otherwise not having been derived
according to other key costing estimating practices, and that future
updates to the GCSSMC economic analysis similarly do so. (Page 55/GAO
Draft Report)
DOD Response: Partially concur
To date, economic analyses for GCSS-MC have leveraged historical cost
data from ERP programs; however this is not analogous historical cost
data for large-scale ERP programs with similar architectural
complexities. Per DoD's response to Recommendation 3, the Department
intends to track and maintain cost data for ERPs through the BCL's
IMIE. which will assist programs in developing future economic
analyses, and a standard WBS for ERP systems will be developed by
AT&L/ARA and BTA.
Through the Naval Center for Cost Analysis (NCCA), the Department of
Navy conducted a risk analysis of the GCSS-MC estimate, which included
a schedule component. This information was provided as part of the data
collected through NCCA, and cited in the Department of the Navy Cost
Analysis Improvement Group (DON CAIG) memo.
Moving forward, the program will also include risk analysis, as a point
comparison to that developed through the DON CAIG. The program is
currently preparing a cost estimate in support of the GCSS-MC Block 1
Milestone C and will ensure that these and future costs are adjusted to
reflect risks associated with the lack of comparable historical data.
Recommendation 5: The GAO recommends that the Secretary of Defense
direct the Secretary of the Navy, through the appropriate chain of
command, to ensure that the program office: (1) monitors the actual
start and completion dates of work activities performed so that the
impact of deviations on downstream scheduled work can he proactively
addressed; (2) allocates resources, such as labor hours and material,
to all key activities on the schedule; (3) integrates key activities
and supporting tasks and sub-tasks; (4) identities and allocates the
amount of float time needed for key activities to account for potential
problems that might occur along or near the schedule's critical path;
(5) performs a schedule risk analysis to determine the level of
confidence in meeting the program's activities and completion date; (6)
allocates schedule reserve for high risk activities on the critical
path; and (7) discloses the inherent risks and limitations associated
with any future use of the program's EVM reports until the schedule has
been risk-adjusted. (Page 55/GAO Draft Report)
DOD Response: Concur.
The Department's EVM policy recognizes that the preparation of an
integrated master schedule (IMS) is a best practice, and requires the
preparation of an IMS whenever EVM compliance is required. GCSS-MC is
currently rebaselining its own integrated master schedule (IMS) to
adjust for schedule risk. The rebaselined GCSS-MC IMS will be used to:
* Provide a baseline for monitoring and reporting planned versus actual
start and completion dates of work activities performed, so that the
impact of deviations on downstream scheduled work can be proactively
addressed;
* Allocate resources to activities;
* Integrate key activities and supporting tasks and sub-tasks;
* Identify and allocate the amount of float time needed for key
activities;
* Allocate schedule reserve for high risk activities on the critical
path.
As the schedule is rebaselined, GCSS-MC will perform a schedule risk
analysis to determine the level of confidence in meeting the program's
activities and completion dates. The program will disclose the inherent
risk and limitations associated with the program's EVM reports until
the schedule has been rebaselined and risk adjusted.
Recommendation 6: The GAO recommends that the Secretary of Defense
direct the Secretary of the Navy, through the appropriate chain of
command, to ensure that the program office: (1) adds each of the risks
discussed in this report to its active inventory of risks; (2) tracks
and evaluates the implementation of mitigation plans for all risks; (3)
discloses to appropriate program oversight and approval authorities
whether mitigation plans have been fully executed and have produced the
intended outcome(s); and (4) only closes a risk if its mitigation plan
has been fully executed and produced the intended outcome(s). (Page
55/GAO Draft Report)
DOD Response: Partially concur.
GCSS-MC has addressed the majority of these risk factors through the
240-plus risks in the risk database, although not necessarily in the
general wording of the GAO report, but instead has addressed them under
a broader range of risk topics in details. The Program Office will,
however, re-examine the risks in the database to ensure that the risks
in this report are covered. In addition, program risks are discussed
during program milestone reviews and regularly reviewed by the program
manager.
Recommendation 7: The GAO recommends that the Secretary of Defense
direct the Secretary of the Navy, through the appropriate chain of
command, to ensure that the program office: (1) collects the data
needed to develop trends in unresolved system defects and change
requests according to their priority and severity; and (2) discloses
these trends to appropriate program oversight and approval authorities.
DOD Response: Concur.
The GCSS-MC program office is currently working with an independent
software measurement analysis company to collect the data needed to
develop trends in unresolved system defects and change requests
according to their priority and severity levels. Results will be
reported to appropriate program oversight authorities on a monthly
basis.
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Randolph C. Hite, (202) 512-3439, or hiter@gao.gov:
Staff Acknowledgments:
In addition to the individual named above, key contributors to this
report were Neelaxi Lakhmani, Assistant Director; Monica Anatalio;
Harold Brumm; Neil Doherty; Cheryl Dottermusch; Nancy Glover; Mustafa
Hassan; Michael Holland; Ethan Iczkovitz; Anh Le; Josh Leiling; Emily
Longcore; Lee McCracken; Madhav Panwar; Karen Richey; Melissa
Schermerhorn; Karl Seifert; Sushmita Srikanth; Jonathan Ticehurst;
Christy Tyson; and Adam Vodraska.
[End of section]
Footnotes:
[1] Business systems are information systems, including financial and
nonfinancial systems, that support DOD business operations, such as
civilian personnel, finance, health, logistics, military personnel,
procurement, and transportation.
[2] GAO, High-Risk Series: An Update, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-07-310] (Washington, D.C.:
January 2007).
[3] DOD defines logistics as the science of planning and carrying out
the movement and maintenance of forces. Logistics includes the aspects
of military operations that deal with: (1) design and development,
acquisition, storage, movement, distribution, maintenance, evacuation,
and disposition of materiel; (2) movement, evacuation, and
hospitalization of personnel; (3) acquisition or construction,
maintenance, operation, and disposition of facilities; and (4)
acquisition or furnishing of services.
[4] GCSS-MC was formally designated a Major Automated Information
System Acquisition program in July 2004, which is a program or
initiative that is so designated by the Assistant Secretary of Defense
(Networks and Information Integration)/Chief Information Officer or
that is estimated to require program costs in any single year in excess
of $32 million (fiscal year 2000 constant dollars), total program costs
in excess of $126 million (fiscal year 2000 constant dollars), or total
life cycle costs in excess of $378 million (fiscal year 2000 constant
dollars).
[5] A garrison location is a home station, usually in the continental
United States, for a unit that is not deployed.
[6] An ERP solution is commercial off-the-shelf software package
consisting of multiple, integrated functional modules that perform a
variety of business related tasks, such as payroll, general ledger
accounting, and supply chain management.
[7] The current life cycle cost estimate is from fiscal year 2007
through fiscal year 2019, in base year 2007 dollars, and excludes costs
associated with supporting and maintaining legacy systems during GCSS-
MC development, totaling $8.2 million, and does not reflect program
costs of $61.4 million from fiscal year 2004 through fiscal year 2006
that are considered sunk costs. According to the Office of Management
and Budget, sunk costs are those incurred in the past that will not be
affected by any present or future decision and should be ignored when
determining whether a new investment is worthwhile.
[8] DAS is a framework-based approach that is intended to translate
mission needs and requirements into stable, affordable, and well-
managed acquisition programs.
[9] E-business Suite is a Web-based software system consisting of
various software (packaged software, applications server, Web server,
database server and administrative software) and hardware (servers,
switches, storage, and ancillary equipment) to support the computing
environment.
[10] Time-and-materials contracts provide for acquiring supplies or
services on the basis of (1) direct labor hours at specified fixed
hourly rates that include wages, overhead, general and administrative
expenses, and profit and (2) actual cost for materials.
[11] Fixed-price contracts with award fees include a fixed priced
(including normal profit) and an additional, separate award fee amount.
The fixed price is paid for satisfactory performance; the award fee, if
any, is earned based on periodic evaluation by the government.
[12] FOC means that the system has been deployed in all intended
locations.
[13] According to the May 10, 2004, analysis of alternatives, this
estimate was a "rough order of magnitude" for research and development,
procurement and operations and support from fiscal years 2004 through
2011.
[14] According to the July 15, 2005, economic analysis, program costs
are estimated from fiscal years 2005 through 2018, in base year 2005
dollars, and exclude $9.6 million associated with supporting and
maintaining legacy systems during GCSS-MC development and $11.9 million
in fiscal year 2004 sunk costs.
[15] Donald E. Harter, Mayuram S. Krishnan, and Sandra A. Slaughter,
"Effects of Process Maturity on Quality, Cycle Time, and Effort in
Software Product Development," Management Science, 46, no. 4, 2000; and
Bradford K. Clark, "Quantifying the Effects of Process Improvement on
Effort," IEEE Software (November/December 2000).
[16] GAO, Information Technology: DOD's Acquisition Policies and
Guidance Need to Incorporate Additional Best Practices and Controls,
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-722] (Washington,
D.C.: July 30, 2004).
[17] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-722].
[18] See, for example, GAO, DOD Business Transformation: Lack of an
Integrated Strategy Puts the Army's Asset Visibility System Investments
at Risk, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-860]
(Washington, D.C.: July 27, 2007); GAO, Information Technology: DOD
Needs to Ensure That Navy Marine Corps Intranet Program Is Meeting
Goals and Satisfying Customers, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-07-51] (Washington, D.C.: Dec. 8, 2006); GAO, Defense
Travel System: Reported Savings Questionable and Implementation
Challenges Remain, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-
980] (Washington, D.C.: Sept. 26, 2006); GAO, DOD Systems
Modernization: Uncertain Joint Use and Marginal Expected Value of
Military Asset Deployment System Warrant Reassessment of Planned
Investment, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-171]
(Washington, D.C.: Dec. 15, 2005); and GAO, DOD Systems Modernization:
Planned Investment in the Navy Tactical Command Support System Needs to
Be Reassessed, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-
215] (Washington, D.C.: Dec. 5, 2005).
[19] Ronald W. Reagan National Defense Authorization Act for Fiscal
Year 2005, Pub. L. No. 108-375, § 332 (2004) (codified at 10 U.S.C. §§
186 and 2,222).
[20] Field/tactical refers to Army units that are deployable to
locations around the world, such as Iraq or Afghanistan.
[21] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-860].
[22] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-215].
[23] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-171].
[24] Department of Defense Architecture Framework, Version 1.0, Volume
1 (February 2004); GAO, Information Technology: A Framework for
Assessing and Improving Enterprise Architecture Management (Version
1.1), [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-584G]
(Washington, D.C.: April 2003); Chief Information Officer Council, A
Practical Guide to Federal Enterprise Architecture, Version 1.0
(February 2001); and Institute of Electrical and Electronics Engineers
(IEEE), Standard for Recommended Practice for Architectural Description
of Software-Intensive Systems 1471-2000 (Sept. 21, 2000).
[25] A well-defined enterprise architecture provides a clear and
comprehensive picture of an entity, whether it is an organization
(e.g., a federal department) or a functional or mission area that cuts
across more than one organization (e.g., personnel management). This
picture consists of snapshots of both the enterprise's current or "As
Is" environment and its target or "To Be" environment, as well as a
capital investment road map for transitioning from the current to the
target environment. These snapshots consist of integrated "views,"
which are one or more architecture products that describe, for example,
the enterprise's business processes and rules; information needs and
flows among functions, supporting systems, services, and applications;
and data and technical standards and structures.
[26] DOD has adopted a federated approach for developing its business
mission area enterprise architecture, which includes the corporate BEA
representing the thin layer of DOD-wide corporate architectural
policies, capabilities, rules, and standards; component architectures
(e.g., DON enterprise architecture); and program architectures (e.g.,
GCSS-MC architecture).
[27] See, for example, GAO, Information Technology: FBI Is Taking Steps
to Develop an Enterprise Architecture, but Much Remains to Be
Accomplished, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-05-363]
(Washington, D.C.: Sept. 9, 2005); GAO, Homeland Security: Efforts
Under Way to Develop Enterprise Architecture, but Much Work Remains,
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-777] (Washington,
D.C.: Aug. 6, 2004); GAO, Information Technology: Architecture Needed
to Guide NASA's Financial Management Modernization, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-04-43] (Washington, D.C.: Nov.
21, 2003); GAO, DOD Business Systems Modernization: Important Progress
Made to Develop Business Enterprise Architecture, but Much Work
Remains, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-1018]
(Washington, D.C.: Sept. 19, 2003); GAO, Information Technology: DLA
Should Strengthen Business Systems Modernization Architecture and
Investment Activities, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-01-631] (Washington, D.C.: June 29, 2001); and GAO,
Information Technology: INS Needs to Better Manage the Development of
Its Enterprise Architecture, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO/AIMD-00-212] (Washington, D.C.: Aug. 1, 2000).
[28] DOD, Business Enterprise Architecture Compliance Guidance (April
10, 2006).
[29] Navy Enterprise Resource Planning was initiated in 2003 to
modernize and standardize certain Navy business operations, such as
financial, acquisition, workforce management, supply, and maintenance.
Moreover, according to program officials, both Navy Enterprise Resource
Planning and GCSS-MC are under the leadership of Navy's Program
Executive Office for Enterprise Information Systems, which is
responsible for developing, acquiring and deploying seamless enterprise-
wide IT systems.
[30] GAO, DOD Business Systems Modernization: Progress in Establishing
Corporate Management Controls Needs to Be Replicated Within Military
Departments, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-705].
(Washington, D.C.: May 15, 2008).
[31] As we recently reported, while the corporate BEA includes
corporate policies, capabilities, rules, and standards, it is still
evolving and will continue to add additional details. See GAO-08-705.
[32] Functional Area Managers are responsible for reducing the number
of DON applications within specific portfolios, such as logistics and
financial management, and reviewing program-level compliance assertions
before they are submitted to the DON CIO.
[33] Department of Defense, Business Transformation Guidance. Version
1.1. (July 2007).
[34] Office of Management and Budget, Circular No. A-11, Preparation,
Submission, and Execution of the Budget (Washington, D.C.: Executive
Office of the President, June 2006); Circular No. A-130 Revised,
Management of Federal Information Resources (Washington, D.C.:
Executive Office of the President, Nov. 28, 2000); and Capital
Programming Guide: Supplement to Circular A-11, Part 7, Preparation,
Submission, and Execution of the Budget (Washington, D.C.: Executive
Office of the President, June 2006).
[35] GAO, Cost Assessment Guide: Best Practices for Estimating and
Managing Program Costs, Exposure Draft, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP] (Washington, D.C.:
July 2007).
[36] Sensitivity analysis reveals how the cost estimate is affected by
a change in a single assumption or cost driver at a time while holding
all other variables constant.
[37] Office of Management and Budget, Circular No. A-94, Guidelines and
Discount Rates for Benefit-Cost Analysis of Federal Programs (Oct. 29,
1992).
[38] A benefit-to-cost ratio is calculated by dividing the present
value of benefits by the present value of costs, while net present
value is the present value of benefits minus the present value of
costs. Present value is the worth of a future stream of costs or
benefits in terms of money paid or received immediately. Prevailing
interest rates provide the basis for converting future amounts into
equivalent present amounts. These interest rates serve as discount
rates in the discounting process--in the calculation of present values.
[39] A Monte Carlo simulation allows the model's parameters to vary
simultaneously according to their associated probability distribution.
The result is a set of estimated probabilities of achieving alternative
outcomes, given the uncertainty in the underlying parameters.
[40] A firm-fixed-price contract provides for a price that is not
subject to any adjustment on the basis of the contractor's cost
experience in performing the contract. This contract type places upon
the contractor maximum risk and full responsibility for all costs and
resulting profit or loss. It provides maximum incentive for the
contractor to control costs and perform effectively and imposes a
minimum administrative burden upon the contracting parties.
[41] Defense Contract Management Agency, Department of Defense Earned
Value Management Implementation Guide, (Washington, D.C.: October
2006). See also DOD Memorandum: Revision to DOD Earned Value Management
Policy (Washington, D.C.: Mar. 7, 2005).
[42] According to DOD guidance, EVM is generally not encouraged for
firm-fixed-price and time-and-material contracts, except in certain
situations, as determined by the complexity of the contracted effort
and the inherent risks involved.
[43] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP].
[44] Float time is the amount of time an activity can slip before
affecting the critical path.
[45] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004).
Software Engineering Institute, Software Acquisition Capability
Maturity Model® version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA:
March 2002).
[46] Work products are the contractor's standardized business
procedures and related test cases, as well as customized design
specifications and test cases for requirements that cannot be met with
standardized products, such as for special reports and external
interfaces.
[47] Department of Defense, Risk Management Guide for DOD Acquisition,
6th Edition, Version 1.0, [hyperlink,
http://www.acq.osd.mil/sse/ed/docs/2006-RM-Guide-4Aug06-final-
version.pdf] (accessed March 13, 2008).
[48] Software Engineering Institute, CMMI for Acquisition, Version 1.2,
CMU/SEI-2007-TR-017 (Pittsburgh, PA: November 2007).
[49] The risk team consists of (1) a senior risk advisor who provides
risk-related consultation to management, (2) a risk manager who
provides general process and quality assurance support for risk
management, and (3) a risk database administrator who issues risk
reports and maintains a risk database.
[50] The database includes those active risks that continue to require
attention and closed risks that have either been addressed or have
become actual problems. The database also provides information such as
the risk owner, current risk level, likelihood of occurrence, and
potential impact to the program's cost, schedule, and performance
commitments.
[51] Risk levels of high, medium, low are assigned using quantitative
measurements of the probability of the risk occurring and the potential
impact to the program's cost, schedule, and performance. Based on that
assessment, a risk level is assigned to represent the risk's
significance. High risks represent the greatest significance, medium
risks represent moderate significance, and low risks represent the
least significant risks.
[52] This review determines whether the system's design is mature and
stable.
[53] The Program Manager's watch list is a report that captures the
status, mitigation plans, and trends for all high risks and those
medium risks that need to be elevated to senior managers.
[54] GAO, Year 2000 Computing Crisis: A Testing Guide, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-10.1.21] (Washington, D.C.:
November 1998); and IEEE Std 12207-2008, Systems and software
engineering-Software life cycle processes (Piscataway, NJ: 2008).
[55] See, for example, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-06-215].
[56] An integrated software suite used to facilitate collaboration,
document management, and access to information between the government
and contractor.
[57] IEEE Std 12207-2008, Systems and software engineering-Software
life cycle processes (Piscataway, NJ: 2008).
[58] Ronald W. Reagan National Defense Authorization Act for Fiscal
Year 2005, Pub. L. No. 108-375, § 332 (2004) (codified at 10 U.S.C. §§
186 and 2,222).
[59] Office of Management and Budget, Circular No. A-94, Guidelines and
Discount Rates for Benefit-Cost Analysis of Federal Program (Oct. 29,
1992); GAO, Cost Assessment Guide: Best Practices for Estimating and
Managing Program Costs, Exposure Draft, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP] (Washington, D.C.:
July 2007).
[60] Defense Contract Management Agency, Department of Defense Earned
Value Management Implementation Guide, (Washington, D.C.: October
2006). See also DOD Memorandum: Revision to DOD Earned Value Management
Policy, (Washington, D.C.: Mar. 7, 2005).
[61] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP].
[62] Software Engineering Institute, Software Acquisition Capability
Maturity Model® version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA:
March 2002).
[63] Department of Defense, Risk Management Guide for DOD Acquisition,
6th Edition, Version 1.0, [hyperlink,
http://www.acq.osd.mil/sse/ed/docs/2006-RM-Guide-4Aug06-final-
version.pdf] (accessed March 13, 2008).
[64] Software Engineering Institute, CMMI for Acquisition, Version 1.2,
CMU/SEI-2007-TR-017 (Pittsburgh, PA: November 2007).
[65] GAO, Year 2000 Computing Crisis: A Testing Guide, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-10.1.21] (Washington, D.C.:
November 1998); and IEEE Std 12207-2008, Systems and software
engineering-Software life cycle processes (Piscataway, NJ: 2008).
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
