DOD Systems Modernization
Planned Investment in the Naval Tactical Command Support System Needs to be Reassessed Gao ID: GAO-06-215 December 5, 2005Because it is important that the Department of Defense (DOD) adheres to disciplined information technology (IT) acquisition processes to successfully modernize its business systems, GAO was asked to determine whether the Naval Tactical Command Support System (NTCSS) is being managed according to important aspects of DOD's acquisition policies and guidance, as well as other relevant acquisition management best practices. NTCSS was started in 1995 to help Navy personnel effectively manage ship, submarine, and aircraft support activities. To date, about $1 billion has been spent to partially deploy NTCSS to about one-half its intended ashore and afloat sites.
The Department of the Navy has not managed its NTCSS program in accordance with key aspects of the department's policies and related guidance, including federal and recognized best practice guidance. Collectively, these policies and guidance are intended to reasonably ensure that investment in a given IT system represents the right solution to fill a mission need and, if it is, that acquisition and deployment of the system are handled in a manner that maximizes the chances of delivering defined system capabilities on time and within budget. In the case of NTCSS, neither of these outcomes is being realized. The Navy has not economically justified its ongoing and planned investment in NTCSS. Specifically, it (1) has not reliably estimated future costs and benefits and (2) has not ensured that independent reviews of its economic justification were performed to determine its reliability. The Navy has not invested in NTCSS within the context of a well-defined DOD or Navy enterprise architecture, which is necessary to guide and constrain NTCSS in a way that promotes interoperability and reduces redundancy with related and dependent systems. The Navy has not effectively performed key measurement, reporting, budgeting, and oversight activities. In particular, earned value management, which is a means for determining and disclosing actual performance against budget and schedule estimates, has not been implemented effectively, and oversight entities have not had the visibility into the program needed to affect its direction. The Navy has not adequately conducted requirements management and testing activities. For example, requirements were neither prioritized nor traced to related documentation to ensure that the system delivers capabilities that meet user needs. This contributed to failures in developmental testing that have prevented the latest component of NTCSS from passing operational testing twice over the last 4 years. Reasons the Navy cited for not following policies and guidance ranged from their not being applicable to the NTCSS program, to lack of time available to apply them, to plans for strengthening system practices not being applied retroactively. Nevertheless, the Navy has begun taking steps and is considering other steps intended to address some of the above problems. Until program management improves, NTCSS will remain a risky program.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-06-215, DOD Systems Modernization: Planned Investment in the Naval Tactical Command Support System Needs to be Reassessed
This is the accessible text file for GAO report number GAO-06-215
entitled 'DOD Systems Modernization: Planned Investment in the Naval
Tactical Command Support System Needs to Be Reassessed' which was
released on December 6, 2005.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Subcommittee on Readiness and Management Support,
Committee on Armed Services, U.S. Senate:
December 2005:
DOD Systems Modernization:
Planned Investment in the Naval Tactical Command Support System Needs
to Be Reassessed:
GAO-06-215:
GAO Highlights:
Highlights of GAO-06-215, a report to the Subcommittee on Readiness and
Management Support, Committee on Armed Services, U.S. Senate:
Why GAO Did This Study:
Because it is important that the Department of Defense (DOD) adheres to
disciplined information technology (IT) acquisition processes to
successfully modernize its business systems, GAO was asked to determine
whether the Naval Tactical Command Support System (NTCSS) is being
managed according to important aspects of DOD‘s acquisition policies
and guidance, as well as other relevant acquisition management best
practices. NTCSS was started in 1995 to help Navy personnel effectively
manage ship, submarine, and aircraft support activities. To date, about
$1 billion has been spent to partially deploy NTCSS to about one-half
its intended ashore and afloat sites.
What GAO Found:
The Department of the Navy has not managed its NTCSS program in
accordance with key aspects of the department‘s policies and related
guidance, including federal and recognized best practice guidance.
Collectively, these policies and guidance are intended to reasonably
ensure that investment in a given IT system represents the right
solution to fill a mission need and, if it is, that acquisition and
deployment of the system are handled in a manner that maximizes the
chances of delivering defined system capabilities on time and within
budget. In the case of NTCSS, neither of these outcomes is being
realized. Specifically,
* The Navy has not economically justified its ongoing and planned
investment in NTCSS. Specifically, it (1) has not reliably estimated
future costs and benefits and (2) has not ensured that independent
reviews of its economic justification were performed to determine its
reliability.
* The Navy has not invested in NTCSS within the context of a well-
defined DOD or Navy enterprise architecture, which is necessary to
guide and constrain NTCSS in a way that promotes interoperability and
reduces redundancy with related and dependent systems.
* The Navy has not effectively performed key measurement, reporting,
budgeting, and oversight activities. In particular, earned value
management, which is a means for determining and disclosing actual
performance against budget and schedule estimates, has not been
implemented effectively, and oversight entities have not had the
visibility into the program needed to affect its direction.
* The Navy has not adequately conducted requirements management and
testing activities. For example, requirements were neither prioritized
nor traced to related documentation to ensure that the system delivers
capabilities that meet user needs. This contributed to failures in
developmental testing that have prevented the latest component of NTCSS
from passing operational testing twice over the last 4 years.
Reasons the Navy cited for not following policies and guidance ranged
from their not being applicable to the NTCSS program, to lack of time
available to apply them, to plans for strengthening system practices
not being applied retroactively. Nevertheless, the Navy has begun
taking steps and is considering other steps intended to address some of
the above problems. Until program management improves, NTCSS will
remain a risky program.
What GAO Recommends:
GAO is making recommendations to the Secretary of Defense to develop
the analytical basis to determine if continued investment in NTCSS
represents prudent use of limited resources. GAO is also making
recommendations to strengthen management of the program, conditional
upon a decision to proceed with further investment in the program. DOD
either fully or partially concurred with the recommendations. It also
stated that while some of GAO‘s findings are valid, the overall
findings understated and misrepresented the program‘s level of
discipline and conformance with applicable guidance and direction.
www.gao.gov/cgi-bin/getrpt?GAO-06-215.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Randolph C. Hite at (202)
512-3439 or hiter@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
NTCSS Has Not Been Managed in Accordance with DOD and Other Relevant
System Acquisition and Development Guidance:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendixes:
Appendix I: Objective, Scope, and Methodology:
Appendix II: Trouble Reports and Change Proposals Assessment:
Trouble Reports:
Change Proposals:
Appendix III: Earned Value Management Assessment:
Appendix IV: Comments from the Department of Defense:
Appendix V: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Legacy Systems and Applications:
Table 2: Optimized Applications Developed During Stage 2 of the NTCSS
Program:
Table 3: Modernized Applications Developed During Stage 3 of the NTCSS
Program:
Table 4: Applications in Operation as of April 2005:
Table 5: NTCSS Budget from FY 1995 through FY 2005:
Table 6: NTCSS Oversight Roles and Responsibilities:
Table 7: NTCSS Management and Stakeholder Roles and Responsibilities:
Table 8: Navy Satisfaction of Cost Estimating Criteria:
Table 9: Navy Satisfaction of OMB Economic Analysis Criteria:
Table 10: Threshold Amounts in NTCSS Acquisition Program Baselines:
Table 11: NTCSS Trouble Report and Change Proposal Priorities:
Table 12: Navy Satisfaction of EVM Criteria:
Figures:
Figure 1: Total Number of Open NTCSS and OOMA Priority 1, 2, and 3
Trouble Reports
Figure 2: Open NTCSS Priority 1 and 2 Trouble Reports
Figure 3: Open NTCSS Priority 3 Trouble Reports
Figure 4: Open OOMA Priority 1 and 2 Trouble Reports
Figure 5: Open OOMA Priority 3 Trouble Reports
Figure 6: Total Number of Open NTCSS and OOMA Priority 1, 2, and 3
Change Proposals
Figure 7: Open NTCSS Priority 1 and 2 Change Proposals
Figure 8: Open NTCSS Priority 3 Change Proposals
Figure 9: Open OOMA Priority 1 and 2 Change Proposals
Figure 10: Open OOMA Priority 3 Change Proposals:
Abbreviations:
CDA: central design agency:
CIO: Chief Information Officer:
DOD: Department of Defense:
ERP: enterprise resource planning:
EVM: earned value management:
IT: information technology:
MRMS: Maintenance Resource Management System:
NALCOMIS: Naval Aviation Logistics Command Management Information
System:
OMB: Office of Management and Budget:
OOMA: Optimized Organizational Maintenance Activity:
NTCSS: Naval Tactical Command Support System:
PA&E: Office of Program Analysis and Evaluation:
RIT: Rapid Improvement Team:
SEI: Software Engineering Institute:
SNAP: Shipboard Non-Tactical Automated Data Processing Program:
SW-CMM: Software Capability Maturity Model:
Letter December 5, 2005:
The Honorable John Ensign:
Chairman:
The Honorable Daniel K. Akaka:
Ranking Minority Member:
Subcommittee on Readiness and Management Support:
Committee on Armed Services:
United States Senate:
Because it is so important that the Department of Defense (DOD) adhere
to disciplined information technology (IT) acquisition processes in
order to successfully modernize its business systems, you requested
that we determine whether the department is following its own revised
policies and guidance for acquiring systems,[Footnote 1] which it
issued in May 2003. As part of our response to your request, we agreed
to review the Naval Tactical Command Support System (NTCSS) program.
NTCSS was started in 1995 and is intended to help Navy personnel
effectively manage ships, submarines, and aircraft support activities.
The Navy expects to spend $348 million on NTCSS between fiscal years
2006 and 2009, for a total of approximately $1.45 billion since program
inception.
As agreed, our objective was to determine whether NTCSS is being
managed according to important aspects of DOD's acquisition policies
and guidance, as well as other relevant acquisition management best
practices. We focused on the program's (1) economic justification; (2)
architectural alignment; (3) project management, including progress
measurement, progress reporting, funding disclosure, and oversight
activities; and (4) system development, including requirements
management and testing. For requirements management and testing, we
focused on the NTCSS application that is currently being developed,
known as the Optimized Organizational Maintenance Activity (OOMA).
We conducted our review from September 2004 through November 2005 in
accordance with generally accepted government auditing standards. For
details on our objective, scope, and methodology, see appendix I.
Results in Brief:
The Navy has not managed its NTCSS program in accordance with key
aspects of the department's system acquisition policies and related
guidance, including federal and recognized best practice guidance.
Collectively, these policies and guidance are intended to reasonably
ensure that investment in a given IT system represents the right
solution to fill a mission need--and if it is, that acquisition and
deployment of the system are handled in a manner that maximizes the
chances of delivering defined system capabilities on time and within
budget. In the case of NTCSS, neither of these outcomes is being
realized. As a result, the Navy does not currently have a sufficient
basis for determining whether NTCSS is the right systems solution for
its aircraft, ship, and submarine tactical command support needs, and
it has not pursued the proposed solution in the right way, meaning in a
fashion that increases chances of delivering defined capabilities on
time and within budget. Key areas in which the Navy did not follow
relevant policies and guidance are described here.
* The Navy has not economically justified its ongoing and planned
investment in NTCSS on the basis of reliable estimates of future costs
and benefits. The most recent economic justification's cost estimates
were not reliably derived, and return on investment was not properly
calculated. In addition, independent reviews of the economic
justification to determine its reliability did not occur, and the Navy
has not measured whether already deployed and operating components of
the system are producing expected value.
* The Navy has not invested in NTCSS within the context of a well-
defined enterprise architecture, which is an institutional blueprint to
guide and constrain program investment decisions in a way that promotes
interoperability and reduces redundancy among related and dependent
systems. As we recently reported,[Footnote 2] DOD's business enterprise
architecture does not contain sufficient context (depth and scope of
operational and technical requirements) to effectively guide and
constrain business transformation and system modernization efforts.
Further, the Navy does not yet have a defined architecture, although it
plans to develop one. Investing in systems, in the absence of an
enterprise architecture, requires explicit recognition and deliberate
consideration of the inherent risks to ensure fully informed investment
decision making.
* The Navy has not effectively performed key measurement, reporting,
and oversight activities. In particular, earned value management, which
is a means for determining and disclosing actual performance against
budget and schedule estimates, and revising estimates based on
performance to date has not been implemented effectively. Also,
complete and current reporting of NTCSS progress and problems in
meeting cost, schedule, and performance goals has not occurred, leaving
oversight entities without the information needed to mitigate risks,
address problems, and take corrective action. In addition, NTCSS
budgets have not reflected the proper category of appropriated funds
associated with system development efforts. Further, oversight
entities' roles and responsibilities have not been fully discharged.
* The Navy has not adequately conducted requirements management and
testing activities. For the NTCSS application that is currently under
development, the Navy has not adequately managed requirements, as
evidenced by the absence of requirements traceability to system design
specifications and testing documents, and the lack of prioritization of
the requirements. The lack of requirements traceability and other
issues have in turn contributed to problems with developmental testing,
including the failure of these tests to identify problems that
subsequently prevented the system from passing operational testing
twice over the last 4 years. Based on the Navy's data, the recent trend
in key indicators of system maturity, such as the number and nature of
reported systems problems and change proposals, shows that problems
with NTCSS persist and that these problems could involve costly and
timely rework to address.[Footnote 3]
Reasons the Navy cited for not following policies and guidance included
questioning their applicability to the NTCSS program, having
insufficient time in which to apply them, and believing that plans to
adopt them were not meant to be applied retroactively. In some cases,
the Navy did not acknowledge that any deviations from policies and
guidance had occurred, but in these cases, it has yet to provide us
with documentation demonstrating that it did adhere to them.
Collectively, this means that after investing 10 years and $1 billion
on NTCSS, it is unclear whether the Navy's planned future investment in
the program is warranted. Even if key uncertainties are addressed and
it can be demonstrated that NTCSS is the right solution, then the
manner in which NTCSS is being defined, developed, tested, measured,
and overseen is also of concern. Accordingly, we are making
recommendations to the Secretary of Defense aimed at developing the
basis needed to determine whether continued investment in NTCSS is a
prudent use of limited departmental resources. We are also making
recommendations to strengthen management of the program, conditional
upon a decision to proceed with further investment in the NTCSS
program.
The Office of the Assistant Secretary of Defense for Networks and
Information Integration provided written comments on a draft of the
report. In its comments, DOD concurred with two of the recommendations
and partially concurred with the remaining five. DOD also stated that
while some of our findings are valid, our overall findings
significantly understated and misrepresented the program's level of
discipline and conformance with applicable guidance and direction. We
do not agree. Our report cites numerous instances, supported by
analyses, where the Navy did not comply with either DOD acquisition
policies and guidelines or industry best practices. DOD's comments are
reprinted in their entirety in appendix IV of this report, along with
our detailed responses to each.
Background:
The Navy's primary mission is to organize, train, maintain, and equip
combat-ready naval forces capable of winning the global war on terror
and any other armed conflict, deterring aggression by would-be foes,
preserving freedom of the seas, and promoting peace and security. To
support this mission, the Navy performs a variety of interrelated and
interdependent business functions such as logistics and financial
management. The Navy requested, for fiscal year 2005, about $3.5
billion to operate, maintain, and modernize its business systems and
related IT infrastructure that support these business functions. This
request represents about 27 percent of the $13 billion that DOD
requested for all of its business systems for fiscal year 2005. Of the
4,150 business systems that DOD reports in its current inventory, the
Navy accounts for 2,353, or about 57 percent, of the total.
In 1995, we designated DOD's business systems modernization efforts as
a high-risk program and continue to designate it as such today[Footnote
4] for several reasons, including the department's challenges in
implementing effective IT investment management structures and
processes, developing and implementing an enterprise architecture, and
implementing effective IT system acquisition and development processes.
NTCSS Genesis and Status Overview:
In the early 1990s, the Navy employed a variety of IT systems to
support the management of information, personnel, materials, and funds
required to maintain and operate ships, submarines, and aircraft. Three
core systems--each managed by a separate program office--consisting of
nine major applications, provided this support: (1) the Shipboard Non-
Tactical Automated Data Processing Program (SNAP), managed by the Space
and Naval Warfare Systems Command; (2) the Naval Aviation Logistics
Command Management Information System (NALCOMIS), managed by the Naval
Air Systems Command; and (3) the Maintenance Resource Management System
(MRMS), managed by the Naval Sea Systems Command. See table 1 for a
description of these three legacy systems and a list of their
respective applications.
Table 1: Legacy Systems and Applications:
Legacy system: SNAP systems; SNAP I; SNAP II;
Description: Manages systems for maintenance, supply, and financial
operations at the organizational and intermediate levels.[A]; Manages
medical and dental services, pay and personnel administration, food
service, retail sales and service, training programs, technical data
storage and retrieval, support and test equipment, and other mission
support-related areas at the organizational level; SNAP I was developed
for the Navy's larger ships, marine aviation logistics squadrons,[B]
training sites, and selected activities ashore; SNAP II provides the
same functionality as SNAP I, but it was developed for use on smaller
ships and submarines. SNAP II was also modified to use microcomputers
as the computing platforms when it is deployed on ships with
constricted physical space; this version is known as MicroSNAP;
Application: SNAP I:
* Shipboard Uniform Automated Data Processing System;
* Organizational Maintenance Management System;
* Administration Data Management I; SNAP II:
* Supply and Financial Management;
* Organizational Maintenance Management System II Maintenance Data
System;
* Administration Data Management II.
Legacy system: NALCOMIS;
Description: Supports day-to-day aircraft maintenance and related
material maintenance functionality both at sea and ashore; Provides the
initial maintenance response when a problem is reported--including
aircraft component troubleshooting, servicing, inspection, and removal
and replacement at the organizational level; Supports, at the
intermediate maintenance level, the repair of components after
defective parts have been removed from an aircraft and sent to a
central location to be refurbished;
Application:
* NALCOMIS Organizational Maintenance Activity;
* NALCOMIS Intermediate Maintenance Activity.
Legacy system: MRMS;
Description: Supports intermediate-level ship and submarine maintenance
at ashore facilities by providing management information such as
planning, scheduling, workload forecasting, work progression,
production control, productivity analysis, and resource management;
Application:
* Maintenance Resource Management System.
Source: Navy.
[A] The "organizational" level is the first stage of aircraft
maintenance activity that is performed on individual planes and
involves the upkeep and servicing of the aircraft at the location where
it is deployed, such as a ship. Components or parts that cannot be
repaired at the organizational level are removed from the plane and
sent to a central location for repair. This second stage of maintenance
is known as the "intermediate" level, and it normally occurs on land.
If the defective part cannot be fixed at the intermediate level, it is
then sent to a third stage of maintenance, known as the "depot" level,
which is not in the scope of the NTCSS program.
[B] Marine aviation logistics squadrons are groups of planes that are
land-based but that can be deployed on an aircraft carrier for a
specific mission. When the mission is completed, these planes return to
their land base.
[End of table]
In 1992, we recommended that the Navy merge the management of all
shipboard nontactical programs under a single command that would have
authority and control over funding and development.[Footnote 5] In
1993, the Navy developed a strategy to do so. In 1994, the Navy also
identified a number of problems with the three legacy systems.
Specifically, the Navy determined that (1) the individual systems did
not consistently handle increasing workloads and provide the
flexibility to meet changing operational demands; (2) the systems'
software architectures were ineffective and inefficient; (3) the
hardware was outdated, slow, expensive to maintain, and nonstandard;
and (4) the systems could not support modernized applications.
To address these concerns, the Navy initiated the NTCSS program in 1995
to enhance the combat readiness of ships, submarines, and aircraft. To
accomplish this, NTCSS was to provide unit commanding officers and
crews with information about, for example, maintenance activities,
parts inventories, finances, technical manuals and drawings, and
personnel. According to the Navy, it spent approximately $1.1 billion
for NTCSS from its inception through fiscal year 2005 and expects to
spend another $348 million between fiscal years 2006 and 2009, for a
total of approximately $1.45 billion.
The Navy defined a three-stage acquisition process for NTCSS.
Stage 1: Purpose was to replace hardware in order to establish a common
infrastructure across all force-level ships, unit-level ships, aviation
squadrons, Naval air stations, marine aviation logistics squadrons, and
other maintenance activities--both at sea and ashore.[Footnote 6]
During this stage, software and business processes were not to be
changed. This phase was begun in 1994 under the legacy SNAP and
NALCOMIS programs and, according to program officials, it is
fundamentally complete--although technology refresh or replacement
activities are still occurring.
Stage 2: Purpose was to provide the functionality of the legacy systems
software with more efficient, more easily maintained software and to
eliminate functional overlap among the systems. This stage was to
involve software technology modernization but no changes in software
functionality or business processes. Existing legacy systems used flat
files and hierarchical databases, which were to be converted to
relational databases, and the existing application code was to be
rewritten using modern software languages. A common hardware and
systems software environment was also to be implemented, and
functionality found in eight of the nine legacy applications was to be
consolidated and rewritten as four new NTCSS applications. Development
of these four applications began in 1995 and was reportedly completed
in 2000. This stage was known as NTCSS Optimization. See table 2 for a
description of the functionality of these new applications.
Stage 3: Purpose was to improve NTCSS's functionality by implementing
business process improvements. According to Navy officials, this stage
is known as NTCSS Modernization and, to date, includes two efforts: (1)
replace the last legacy application and (2) create a Web-enabled
version of the three unit-level Optimized NTCSS applications that were
developed under Stage 2. See table 3 for a description of the
functionality of these business process improvements.
Table 2: Optimized Applications Developed During Stage 2 of the NTCSS
Program:
NTCSS Optimized applications: Relational Supply;
Description: Supports supply chain management, inventory management,
and financial management processes; Provides Navy personnel with access
to the supply support functions they perform most often--ordering,
receiving, and issuing necessary supplies and materials; maintaining
financial records; and reconciling supply, inventory, and financial
records with the Navy's shore infrastructure;
Status: Operational, as of September 1998, on large force-level ships,
smaller unit-level ships, and at air stations and marine aviation
logistics squadrons.[A].
NTCSS Optimized applications: Organizational Maintenance Management
System--Next Generation;
Description: Assists shipboard personnel in planning, scheduling,
reporting, and tracking maintenance and related logistics support
actions; Maintains online lists of maintenance actions to be performed,
parts required to maintain shipboard equipment, and parts carried
onboard ship to support maintenance actions; Interfaces with Relational
Supply to requisition parts that are not onboard;
Status: Operational, as of September 1998, primarily on large force-
level ships and smaller unit-level ships.
NTCSS Optimized applications: Relational Administration Data
Management;
Description: Automates the management of personnel awards and
decorations, work assignments, and berthing assignments;
Status: Operational, as of April 2000, on large force-level ships,
smaller unit-level ships, and at air stations and marine aviation
logistics squadrons.
NTCSS Optimized applications: Optimized Intermediate Maintenance
Activities;
Description: Provides online intermediate-level aviation maintenance,
configuration, and logistics management support; Interfaces with other
major integrated logistics support systems within the Naval aviation
community;
Status: Operational, as of April 2000, at force-level ships and at air
stations and marine aviation logistics squadrons.
Source: Navy.
[A] Relational Supply is also in use at additional sites that are not a
part of the NTCSS program.
[End of table]
Table 3: Modernized Applications Developed During Stage 3 of the NTCSS
Program:
NTCSS modernized applications: Optimized Organizational Maintenance
Activity (OOMA);
Description: Is to support day-to-day maintenance management tools for
aviation squadrons and other organizational-level maintenance
activities; Is to provide the foundation for achieving a completely
automated maintenance environment, such as a single point of data
entry, automated and assisted pilot and maintenance debrief, online
diagnostics, structural life prognostics,[A] interactive electronic
technical manuals, and forecasting and tracking of maintenance
schedules;
Status: Initiated in 1999, withdrawn from operational testing[B] in
April 2001 when it became clear that it would fail. Failed operational
testing again in May 2004. Scheduled for third operational test in the
third quarter of fiscal year 2006; Fielded at 77 sites as of June 2005.
NTCSS modernized applications: eNTCSS;
Description: Was to provide a Web-enabled version of NTCSS, and allow
users to access the three unit-level Optimized applications from any
workstation on a ship's local area network via a standard Web browser
and to execute work activities in a Web-server environment;
Status: Initiated in 2001. Cancelled in April 2004; Fielded on one
submarine and scheduled to be fielded on one more; Is to be replaced
with the Optimized applications, but a date has yet to be determined.
Source: Navy.
[A] According to the U.S. Marine Corps Logistics Directorate,
structural life prognostics is defined as the ability to reliably
predict the remaining useful life of mechanical or structural
components, within an actionable time period, and within acceptable
confidence limits.
[B] According to the DOD Defense Acquisition Guidebook, the primary
purpose of operational test and evaluation is for representative users
to evaluate systems in a realistic environment in order to determine
whether these systems are operationally effective and suitable for
their intended use before production or deployment.
[End of table]
As of April 2005, legacy applications were still in use at 51 percent
of the Navy's 659 sites. These 659 sites either have legacy, Optimized,
or modernized applications. Table 4 shows the distribution of the
legacy, Optimized, and modernized applications.
Table 4: Applications in Operation as of April 2005:
Legacy applications:
Applications: SNAP I[A, B];
Number of sites: 10.
Applications: SNAP II[A, B, C];
Number of sites: 68.
Applications: MicroSNAP;
Number of sites: 32.
Applications: NALCOMIS Organizational Maintenance Activity[D];
Number of sites: 214.
Applications: NALCOMIS Intermediate Maintenance Activity[B];
Number of sites: 10.
Applications: Maintenance Resource Management System[E];
Number of sites: 2.
Applications: Subtotal;
Number of sites: 336;
Percentage of total: 51.
Optimized applications[F]:
Applications: Relational Supply[C], Organizational Maintenance
Management System-Next Generation, Relational Administration Data
Management, Optimized Intermediate Maintenance Activities.
Applications: Subtotal;
Number of sites: 229;
Percentage of total: 35.
Modernized applications:
Applications: Optimized Organizational Maintenance Activity;
Number of sites: 93.
Applications: eNTCSS;
Number of sites: 1.
Applications: Subtotal;
Number of sites: 94;
Percentage of total: 14.
Applications: Total;
Number of sites: 659;
Percentage of total: 100.
Source: Navy.
[A] SNAP I and SNAP II are each composed of three different legacy
applications (see table 1).
[B] The Navy plans to decommission some of the ships that use these
applications and upgrade the remaining ships to NTCSS Optimized
applications.
[C] This application also is in use at additional sites that are not a
part of the NTCSS program.
[D] The functionality included in this application is to be replaced in
the future by Optimized Organizational Maintenance Activity.
[E] The Navy plans to incorporate this functionality into
Organizational Maintenance Management System-Next Generation at a
future date.
[F] These four applications are deployed as a single software package
at all 229 sites.
[End of table]
According to Navy officials, about $1.1 billion was spent on NTCSS
between 1995 and 2005. This includes about $1 billion on NTCSS
Optimized applications[Footnote 7] and $91 million on OOMA and eNTCSS.
Table 5 shows NTCSS's budget totals from the time the program began in
fiscal year 1995 through fiscal year 2005.
Table 5: NTCSS Budget from FY 1995 through FY 2005:
Dollars in thousands:
NTCSS Optimized;
FY 95: 83,537;
FY 96: 69,794;
FY 97: 69,075;
FY 98: 123,469;
FY 99: 119,822;
FY 00: 91,053;
FY 01: 95,322;
FY 02: 95,549;
FY 03: 82,708;
FY 04: 108,087;
FY 05: 71,926;
Total: 1,010,342.
OOMA;
FY 96: 920;
FY 97: 700;
FY 98: 983;
FY 99: 4,724;
FY 00: 16,527;
FY 01: 20,854;
FY 02: 14,920;
FY 03: 3,981;
FY 04: 2,871;
FY 05: 13,291;
Total: 79,771.
eNTCSS;
FY 01: 5,000;
FY 02: 5,309;
FY 03: 985;
Total: 11,294.
Total;
FY 95: 83,537;
FY 96: 70,714;
FY 97: 69,775;
FY 98: 124,452;
FY 99: 124,546;
FY 00: 107,580;
FY 01: 121,176;
FY 02: 115,778;
FY 03: 87,674;
FY 04: 110,958;
FY 05: 85,217;
Total: 1,101,407.
Source: Navy.
[End of table]
NTCSS Oversight and Management Roles and Responsibilities:
A number of Navy and DOD organizations are involved in overseeing and
managing the NTCSS program. Table 6 lists the organizations involved in
NTCSS oversight and their respective roles and responsibilities.
Table 6: NTCSS Oversight Roles and Responsibilities:
Oversight entity: Deputy Assistant Secretary of the Navy for Command,
Control, Communication, Computers and Intelligence, and Space;
Roles and responsibilities: Currently serves as the milestone decision
authority. Assigned overall responsibility for the NTCSS program;
approves the program to proceed through its acquisition cycle on the
basis of a review of key documents, such as an acquisition plan, an
independently evaluated life cycle cost-and-benefits estimate,
Acquisition Program Baseline documents, and Defense Acquisition
Executive Summary reports.
Oversight entity: Program Executive Office for Command, Control,
Communication, Computers and Intelligence, and Space; Space and Naval
Warfare Systems Command;
Roles and responsibilities: Serves as the program executive office.
Assigned overall responsibility for NTCSS program oversight; reviews
the component cost analysis, acquisition strategy, and Acquisition
Program Baseline prior to approval by the milestone decision authority.
Oversight entity: Department of Navy Chief Information Officer;
Roles and responsibilities: Reviews the acquisition program during the
department's planning, programming, budgeting, and execution processes
to ensure that the program's goals are achievable and executable;
ensures conformance to appropriation law, financial management
regulations, and Navy, DOD, and federal IT policies in several areas
(e.g., security, architecture, and investment management); works
closely with the program office during milestone review assessments.
Oversight entity: Assistant Secretary of the Navy, Research Development
and Acquisition, Chief Engineer;
Roles and responsibilities: Ensures system compliance with
architectural standards and promotes interoperability of the Navy's
systems.
Oversight entity: Office of the Secretary of Defense, Office of the
Director for Program Analysis and Evaluation;
Roles and responsibilities: Verifies and validates the reliability of
cost and benefit estimates found in economic analyses and provides its
results to the milestone decision authority.
Oversight entity: Naval Cost Analysis Division;
Roles and responsibilities: Performs independent cost estimates,
maintains cost analysis tools, and focuses on cost analysis policy and
oversight.
Oversight entity: Executive Steering Committee; Members are
representatives from: Office of the Chief of Naval Operations for
Material Readiness and Logistics Operations (Chairman); Commander in
Chief, U.S. Atlantic Fleet; Commander in Chief, U.S. Pacific Fleet;
Commandant of the Marine Corps; and; Program Executive Office for
Command, Control, Communication, Computers and Intelligence, and Space;
Roles and responsibilities: Establishes priorities for NTCSS
development and implementation and for defining long-term architectural
goals; meets after regularly scheduled NTCSS meetings (e.g.,
Requirements Integrated Product Team meetings and Forum meetings).[A].
Source: Navy.
[A] The Requirements Integrated Product Team is chartered to collect
and analyze users' requirements, input these requirements into the
NTCSS requirements management process, and provide recommendations to
the program office on these requirements. The Forum brings together
stakeholders and acquisition and development personnel to (1) discuss
issues and requirements related to current and future system readiness,
(2) develop specific action items and recommendations that will result
in improved program products and services to the Fleet, and (3)
facilitate key decisions by senior program leadership at Executive
Steering Committee meetings.
[End of table]
There have been three milestone decision authorities for NTCSS since
the program was begun. Initially, the milestone decision authority was
in the Office of the Assistant Secretary of Defense for Networks and
Information Integration/Chief Information Officer. In July 1999, this
authority was delegated to the Assistant Secretary of the Navy for
Research, Development, and Acquisition, who then delegated oversight
authority to Deputy Assistant Secretary of the Navy for Command,
Control, Communication, Computers and Intelligence, and Space in March
2000.
Table 7 lists the organizations involved in NTCSS management and their
respective roles and responsibilities.
Table 7: NTCSS Management and Stakeholder Roles and Responsibilities:
Entity: Program Manager, Warfare; Space and Naval Warfare Systems
Command;
Roles and responsibilities: Serves as the program office. Assigned
responsibility for day-to-day program management of NTCSS and, as such,
is the single point of accountability for managing the program's
objectives through development, production, and sustainment. Manages
cost, schedule, and performance reporting. Prepares and updates the
acquisition strategy, component cost analysis, and acquisition program
baselines. Coordinates all testing activities in coordination with
requirements.
Entity: Space and Naval Warfare Systems Command, Systems Center
Norfolk;
Roles and responsibilities: Serves as the central design agency.
Assigned responsibility for software development, including application
design, development, and testing activities. Responsible for managing
trouble reports and change proposals.[A] Manages Space and Naval
Warfare Systems Command, Systems Center Norfolk Detachment San Diego,
which installs the initial NTCSS systems on ships, submarines, and at
land sites and performs subsequent on-site software maintenance.
Entity: Space and Naval Warfare Systems Command, Systems Center
Charleston;
Roles and responsibilities: Serves as the in-service engineering
activity. Provides engineering support and installs and integrates
hardware.
Entity: Office of the Chief of Naval Operations for Material Readiness
and Logistics Operations;
Roles and responsibilities: Serves as the program and resource sponsor.
Balances user requirements with available resources. Works with users
to ensure that operational and functional requirements are prioritized
correctly and are supported. Addresses various issues pertaining to
Navy policy, requirements, resources, and schedules.
Entity: Functional Managers; Includes representatives from: Naval Sea
Systems Command; Naval Supply Systems Command; Naval Air Systems
Command; and; Commander in Chief, U.S. Atlantic Fleet;
Roles and responsibilities: Represent the system users. Participate in
the process of establishing functional requirements for input into the
change management and system design processes. Prepare test plans and
test analysis reports to support functional certification of software.
Source: Navy.
[A] Navy officials provided data regarding trouble reports and change
proposals for the Optimized and modernized NTCSS applications. For
details see appendix II.
[End of table]
NTCSS Participation in DOD's Rapid Improvement Team Pilot:
In 2001, the DOD Chief Information Officer and the Undersecretary of
Defense for Acquisition, Technology, and Logistics chartered a pilot
project aimed at saving time by significantly reducing the reporting
and oversight requirements. The ultimate goal was to enable the
acquisition process to deliver mission-effective IT systems within 18
months. Known as the Rapid Improvement Team (RIT) for IT Acquisition
Management Transformation, the pilot was to cover a 2-year period from
January 1, 2002, through December 31, 2003. Nine programs from the
military services participated in the pilot. NTCSS was selected to
participate in the pilot by its milestone decision authority due to its
longevity and because of its perceived low risk, stability, and
compliance with IT management best practices. It was also believed that
little system development remained to be done. NTCSS began
participating in the RIT pilot in October 2002.
The RIT pilot relieved the program office of the normal acquisition
process activities, such as preplanned, formal milestone decision
reviews or briefings, and it granted the program office the authority
to pass key milestones once it determined that established requirements
had been met. This streamlined approach was considered possible because
all information related to these requirements was to be continually
updated and available to oversight organizations and stakeholders via a
RIT Web site. More specifically, the program office was to update the
Web site monthly via a set of electronic forms with the kind of data
that were traditionally found in DOD oversight documents. The program
office was also to use the Web site to input key acquisition documents
(e.g., acquisition plans, economic analyses, requirements documents and
test plans) in an electronic library. In turn, the milestone decision
authority and other oversight organizations were to review these data
on at least a monthly basis and directly retrieve any acquisition
documents to be reviewed from the library. No response from the
milestone decision authority would indicate implicit approval of the
program data. Although the formal RIT pilot ended in December 2003,
program officials told us that they continued to operate using the RIT
pilot's procedures and continued to update program information on the
Web site through December 2004.
According to a memorandum issued by the Office of the Assistant
Secretary of Defense for Networks and Information Integration/Chief
Information Officer and the Undersecretary of Defense for Acquisition,
Technology, and Logistics, the principal output of the pilot would be a
blueprint for IT acquisition that is transferable to other systems. A
report summarizing the results of the entire RIT pilot program was
published in April 2005.[Footnote 8] This report concluded that (1) by
instituting risk-based governance, the milestone decision authority can
be assigned to an organization subordinate to the Office of the
Secretary of Defense without adding unacceptable risk to the investment
process and (2) the success of risk-based governance and cycle time
reduction is predicated on the adoption of net-centricity[Footnote 9]
by the business community.
Prior Review Identified Strengths and Weaknesses in DOD's Acquisition
Policies and Guidance:
In July 2004, we reported[Footnote 10] that DOD's revised systems
acquisition policies and guidance incorporated many best practices for
acquiring business systems, such as (1) justifying system investments
economically, on the basis of costs, benefits, and risks, and (2)
continually measuring an acquisition's performance, cost, and schedule
against approved baselines. However, the revised policies and guidance
did not incorporate a number of other best practices, particularly
those associated with acquiring commercial component-based business
systems, and DOD did not have documented plans for incorporating these
additional best practices into its policies. We also reported that the
department's revised acquisition policies did not include sufficient
controls to ensure that military services and defense agencies would
appropriately follow these practices. We concluded that, until these
additional best practices were incorporated into DOD's acquisition
policies and guidance, there was increased risk that system
acquisitions would not deliver planned capabilities and benefits on
time and within budget and increased risk that an organization will not
adopt and use best practices that were defined. Accordingly, we made 14
recommendations to the Secretary of Defense that were aimed at
strengthening DOD's acquisition policy and guidance by including
additional IT systems acquisition best practices and controls for
ensuring that these best practices were followed. DOD agreed with most
of our recommendations and has since issued additional system
acquisition guidance.[Footnote 11]
NTCSS Has Not Been Managed in Accordance with DOD and Other Relevant
System Acquisition and Development Guidance:
DOD system acquisition and development policies and guidance, along
with other federal and best practices guidance, provide an effective
framework within which to manage IT business system programs and
investments, like NTCSS. Proper implementation of this framework can
minimize program risks and better ensure that system investments
deliver promised capabilities and benefits on time and within budget.
The Navy has not managed NTCSS in accordance with many key aspects of
these policies and guidance. For example, the Navy has not economically
justified its investment in NTCSS on the basis of cost and benefits. It
has not invested in NTCSS within the context of a well-defined
enterprise architecture. Further, the Navy has not effectively
performed key measurement, reporting, and oversight activities, and has
not adequately conducted requirements management and testing
activities. Reasons the Navy cited for not following policies and
guidance included questioning their applicability to the NTCSS program,
having insufficient time in which to apply them, and believing that
plans to adopt them were not meant to be applied retroactively. In some
cases, the Navy did not acknowledge that any deviations from policies
and guidance had occurred but, in these cases, it has yet to provide us
with documentation demonstrating that it did adhere to them. As a
result, the Navy does not currently have a sufficient basis for
determining whether NTCSS is the right system solution for its tactical
command support needs, and it has not pursued the proposed solution in
a way that increases the likelihood of delivering defined capabilities
on time and within budget.
The Navy Has Not Economically Justified Investment in NTCSS on the
Basis of Costs and Benefits:
The decision to invest in any system should be based on reliable
analyses of estimated system costs and expected benefits over the life
of the program. DOD policy requires such analyses, and other relevant
acquisition management practices provide guidance on how these analyses
should be prepared. However, the current economic analysis for the
NTCSS program does not meet this guidance. Additionally, the analysis
was not independently reviewed in accordance with DOD guidance.
Finally, contrary to DOD policy and relevant acquisition management
practices, the Navy has not demonstrated that NTCSS Optimized
applications are producing expected benefits. Without such reliable
analyses, an organization cannot adequately know that a given system
investment is justified.
The Latest NTCSS Cost Estimate Was Not Derived Reliably:
According to DOD guidance,[Footnote 12] the cost estimates used to
economically justify an investment should be reasonable, traceable, and
based on realistic assumptions. Our research shows that a reliable cost
estimate should meet nine specific criteria developed by Carnegie
Mellon University's Software Engineering Institute (SEI),[Footnote 13]
such as appropriately sizing the task being estimated and identifying
and explaining estimate assumptions.
In March 2004, the NTCSS program office prepared its fifth NTCSS
economic analysis. This analysis examined the costs associated with
three alternative NTCSS hardware, software, operating system and data
base management configurations, and was to be used to inform decisions
about system development and implementation. The analysis did include
estimated costs for each alternative. However, it did not include
measurable, quantifiable benefits for each alternative. Rather, it
included only qualitative benefits. Further, the cost estimates used in
this analysis did not meet six of the nine criteria associated with
reliable cost estimates. For example, while the estimate's purpose was
stated in writing, the system life cycle used was 6 years rather than
the 10 years recommended. Also, documentation showing that the costs
were based on data from the program's demonstrated accomplishments has
yet to be provided to us, and the assumptions used to create the cost
estimate were not identified and explained. See table 8 for the results
of our analyses relative to each of the nine criteria.
Table 8: Navy Satisfaction of Cost Estimating Criteria:
Criterion: The objectives of the program are stated in writing;
Explanation: The objectives of the program should be clearly and
concisely stated for the cost estimator to use;
Criterion met[A]: Yes;
GAO analysis: The objective of the program was clearly stated.
Criterion: The life cycle to which the estimate applies is clearly
defined;
Explanation: The life cycle should be clearly defined to ensure that
the full cost of the program--that is, all direct and indirect costs
for planning, procurement, operations and maintenance, and disposal--
are captured. For investments such as NTCSS, the life cycle should
cover 10 years past full operational capability of the system.[B];
Criterion met[A]: No;
GAO analysis: The life cycle was not clearly defined to ensure that the
full cost of the program is included. The life cycle defined was 6
years past full operational capability, instead of the full 10 years
defined in the DOD guidance.
Criterion: The task has been appropriately sized;
Explanation: An appropriate sizing metric should be used in the
development of the estimate, such as the amount of software to be
developed and the amount of software to be revised;
Criterion met[A]: Yes;
GAO analysis: The method used in the model lends itself to being
appropriately sized.
Criterion: The estimated cost and schedule are consistent with
demonstrated accomplishments on other projects;
Explanation: Estimates should be validated by relating them back to
demonstrated and documented performance on completed projects;
Criterion met[A]: No;
GAO analysis: No documentation was provided to show the use of
historical data to produce the estimate.
Criterion: A written summary of parameter values and their rationales
accompany the estimate;
Explanation: If a parametric equation was used to generate the
estimate, the parameters that feed the equation should be provided,
along with an explanation of why they were chosen;
Criterion met[A]: No;
GAO analysis: The model used undocumented values as the source of the
estimate for multiple elements.
Criterion: Assumptions have been identified and explained;
Explanation: Accurate assumptions regarding issues such as schedule,
quantity, technology, development processes, manufacturing techniques,
software language, etc., should be understood and documented;
Criterion met[A]: No;
GAO analysis: Any assumptions used in the model were not identified.
Criterion: A structured process, such as a template or format, has been
used to ensure that key factors have not been overlooked;
Explanation: A work breakdown structure or similar structure that
organizes, defines, and graphically displays the individual work units
to be performed should be used. The structure should be revised over
time as more information becomes known about the work to be performed;
Criterion met[A]: Yes;
GAO analysis: A work breakdown structure was provided and included all
the standards elements.
Criterion: Uncertainties in parameter values have been identified and
quantified;
Explanation: For all major cost drivers, an uncertainty analysis should
be performed to recognize and reflect the risk associated with the cost
estimate;
Criterion met[A]: No;
GAO analysis: No risk analysis was documented in the estimate.
Criterion: If more than one cost model or estimating approach has been
used, any differences in the results have been analyzed and explained;
Explanation: The primary methodology or cost model results should be
compared with any secondary methodology (for example, cross-checks) to
ensure consistency;
Criterion met[A]: No;
GAO analysis: No secondary model was discussed in the estimate
documentation.
Sources: SEI criteria, DOD guidance, and GAO analysis of Navy data.
[A] "Yes" means that the program provided documentation demonstrating
satisfaction of the criterion. "Partially" means that the program
provided documentation demonstrating satisfaction of part of the
criterion. "No" means that the program has yet to provide documentation
demonstrating satisfaction of the criterion.
[B] DOD, DOD Automated Information System (AIS) Economic Analysis (EA)
Guide, May 1, 1995.
[End of table]
Program officials told us that they did not develop the 2004 cost
estimate in accordance with all of the SEI cost estimating criteria
because they had only a month to complete the economic analysis. By not
following practices associated with reliable estimates, the Navy has
decided on a course of action that is not based on one of the key
ingredients to sound and prudent decision making--a reliable estimate
of system life cycle costs. Among other things, this means that the
investment decision made by the Navy has not been adequately justified
and, that to the extent that program budgets were based on cost
estimates, the likelihood of funding shortfalls and inadequate funding
reserves is increased.
The Latest NTCSS Economic Analysis Did Not Meet Key Federal Guidance:
According to Office of Management and Budget (OMB) guidance,[Footnote
14] economic analyses should meet certain criteria to be considered
reasonable, such as comparing alternatives on the basis of net present
value and conducting an uncertainty analysis of costs and benefits.
The latest NTCSS economic analysis, prepared in March 2004, identified
potential costs and benefits from three alternative NTCSS hardware,
software, operating system, and data base management configurations.
However, the analysis provided only monetized costs for each
alternative. It did not provide monetized benefits. Further, the
analysis did not meet five of eight OMB criteria. For example, the
alternatives were not compared on the basis of their net present
values, an appropriate interest rate was not used to discount the net
present values, and the uncertainty associated with the cost estimates
was not disclosed and used in the analysis. See table 9 for the results
of our analyses relative to each of the eight criteria.
Table 9: Navy Satisfaction of OMB Economic Analysis Criteria:
Criterion: The economic analysis clearly explained why the investment
was needed;
Explanation: The economic analysis should clearly explain the reason
why the investment is needed, i.e., why the status quo alternative is
unacceptable;
Criterion met[A]: Yes;
GAO analysis: The economic analysis explained why the status quo
alternative was not viable.
Criterion: At least two alternatives to the status quo were considered;
Explanation: At least two meaningful alternatives to the status quo
should be examined to help ensure that the alternative chosen was not
preselected;
Criterion met[A]: Yes;
GAO analysis: Three alternatives to the status quo were considered.
Criterion: The general rationale for the inclusion of each alternative
was discussed;
Explanation: The general rationale for the inclusion of each
alternative should be discussed to enable reviewers of the analysis to
gain an understanding of the context for the selection of one
alternative over the others;
Criterion met[A]: Yes;
GAO analysis: The rationale for each alternative was discussed.
Criterion: The quality of the cost estimate for each alternative was
reasonable;
Explanation: The quality of the cost estimate of each alternative
should be complete and reasonable for a net present value to be
accurate. One measure of a cost estimate's reasonableness is its
satisfaction of earlier cited SEI criteria;
Criterion met[A]: No;
GAO analysis: The cost estimates were not complete and did not meet a
majority of the SEI criteria.
Criterion: The quality of the benefits to be realized from each
alternative was reasonable;
Explanation: The quality of the benefit estimate of each alternative
should be complete and reasonable for a net present value to be
calculable and accurate;
Criterion met[A]: No;
GAO analysis: Monetized estimates of benefits were not provided, and no
explanation was given as to why these estimates were not provided.
Criterion: Alternatives were compared on the basis of net present
value;
Explanation: The net present value should be calculated because it
consistently results in the selection of the alternative with the
greatest benefit net of cost;
Criterion met[A]: No;
GAO analysis: The economic analysis stated that all costs and benefits
were expressed in undiscounted constant fiscal year 2004 dollars;
however, monetized benefits were not reported in the economic analysis.
As a result, the net present value was not calculated.
Criterion: The proper discount rate used for calculating each
alternative's overall net present value should be used;
Explanation: OMB Circular A-94 is the general guidance for conducting
cost-benefit analyses for federal government programs and provides
specific guidance on the discount rates to be used in evaluating those
programs whose benefits and costs are distributed over time;
Criterion met[A]: No;
GAO analysis: Since all dollar amounts are expressed in undiscounted
constant fiscal year 2004 dollars, the discount rate used in the
economic analysis is, by default, zero. The discount rates provided by
OMB Circular No. A-94 are all positive (i.e., greater than zero).
Criterion: An uncertainty analysis of costs and benefits was included;
Explanation: Estimates of benefits and costs are typically uncertain
because of imprecision in both underlying data and modeling
assumptions. Because such uncertainty is basic to virtually any cost-
benefit analysis, its effects should be analyzed and reported;
Criterion met[A]: No;
GAO analysis: No uncertainty analysis for the overall reported costs
was included.
Sources: OMB guidance and GAO analysis of Navy data.
[A] "Yes" means that the program provided documentation demonstrating
satisfaction of the criterion. "Partially" means that the program
provided documentation demonstrating satisfaction of part of the
criterion. "No" means that the program has yet to provide documentation
demonstrating satisfaction of the criterion.
[End of table]
Program officials told us that they did not adhere to the OMB criteria
because they had only a month to complete the economic analysis and,
therefore, did not have the time necessary to comply with it. By not
following established OMB guidance, the reliability of the latest NTCSS
economic analysis is questionable. This further increases the risk that
the Navy is following a course of action that will not produce the
expected return on investment.
The Latest NTCSS Economic Analysis Was Not Independently Reviewed:
DOD guidance[Footnote 15] states that economic analyses and cost
estimates should be independently reviewed and assessed. In this
regard, the Office of Program Analysis and Evaluation, located in the
Office of the Secretary of Defense, is responsible for verifying and
validating the reliability of economic analyses and providing the
results to the milestone decision authority; the Naval Cost Analysis
Division is responsible for preparing independent cost estimates.
However, neither of these offices reviewed the most recent economic
analysis for NTCSS. An official from the Office of Program Analysis and
Evaluation told us that this office did not review the 2004 economic
analysis because, once NTCSS entered the RIT Pilot, the program office
no longer provided documentation needed to review the analysis.
Officials from the Naval Cost Analysis Division also stated that they
did not review the estimates in this economic analysis. According to
officials from this office, they are only required to review cost
estimates that are prepared for milestone reviews, and staffing
limitations do not permit them to review all cost estimates.
By not having the economic analysis reviewed by independent parties,
the Navy has no independent verification that the estimates of life
cycle costs and benefits are reasonable and traceable, that the cost
estimates are built on realistic program and schedule assumptions, or
that the return on investment calculation is valid. This casts further
doubt on the reliability of the economic analysis the Navy has used to
justify its ongoing investment in NTCSS.
The Navy Has Yet to Measure Whether Actual Benefits Have Accrued from
Deployed NTCSS Capabilities:
The Clinger-Cohen Act of 1996 and OMB guidance[Footnote 16] emphasize
the need to develop information to ensure that IT projects are actually
contributing to tangible, observable improvements in mission
performance. DOD guidance[Footnote 17] also requires that analyses be
conducted to validate estimated benefits and measure the extent to
which desired outcomes have been achieved. To this end, agencies should
define and collect metrics to determine whether expected benefits are
being achieved and modify subsequent applications and investments to
reflect the lessons learned.
However, the Navy has yet to measure whether NTCSS Optimized
applications are actually producing expected benefits commensurate with
actual costs. For example, in 1999 the Navy projected that deploying
the NTCSS Optimized applications would result in reduced costs
associated with NTCSS maintenance, training, and other support
activities. However, the Navy does not know the extent to which NTCSS
Optimized applications are meeting these expectations--even though
these applications have been deployed to 229 user sites since 1998--
because metrics to demonstrate that these expectations have been met
have not been defined and collected.
Program officials and officials representing the milestone decision
authority stated that the Navy is not required to measure actual
accrual of benefits because DOD guidance to do so was not yet in effect
when the NTCSS Optimized applications were deployed, and there was no
explicit requirement to apply this guidance retroactively. Program
officials also stated that it will not be possible to measure actual
return-on-investment for the already deployed NTCSS Optimized
applications until the entire NTCSS system is deployed and operational.
Similarly, an official with the milestone decision authority stated
that actual NTCSS return-on-investment has not yet been measured.
Because it is not measuring whether cost and benefit projections are
being met, the Navy lacks important information that it will need to
inform future economic analyses and investment decisions.
The Navy Recently Decided to Prepare a Benefits Assessment:
In February 2005, officials from the Office of the Chief of Naval
Operations for Material Readiness and Logistics Operations[Footnote 18]
and representatives from key user organizations questioned whether
NTCSS can cost effectively meet users' future needs. Initially this
office tasked the program office to develop a new economic analysis to
determine whether to continue investing in NTCSS or in some other
system solution, such as the Navy enterprise resource planning (ERP)
program.[Footnote 19] In November 2005, officials from the Office of
the Chief of Naval Operations for Material Readiness and Logistics
Operations stated that they were no longer planning to develop a new
economic analysis but planning to conduct a benefits assessment to
evaluate changing NTCSS to some solution to enable the system to
perform future ashore activities. These officials acknowledged that
this assessment will be less than the initially planned economic
analysis in that it will exclude any analysis of costs and alternative
solutions. However, they also acknowledged that DOD policy and guidance
does not address benefits assessments as a recognized acquisition
program document. They stated that this assessment will be prepared for
inclusion in the 2006 budget submission.
Without knowing the extent to which NTCSS Optimized applications are
meeting cost and benefit expectations, the Navy is not in a position to
make informed, and thus justified, decisions on whether and how to
proceed with the program. Such a situation introduces a serious risk
that the Navy will not be able to demonstrate whether NTCSS is cost-
effective until it has already spent hundreds of millions of dollars
more on the NTCSS Optimized applications and OOMA.
The Navy Has Not Defined and Developed NTCSS within the Context of an
Enterprise Architecture:
DOD policy and guidance,[Footnote 20] as well as federal and best
practice guidance,[Footnote 21] recognize the importance of investing
in IT business systems within the context of an enterprise
architecture. Our research and experience in reviewing federal agencies
shows that not doing so often results in systems that are duplicative,
not well integrated, unnecessarily costly to interface and maintain,
and do not optimally support mission outcomes.[Footnote 22] NTCSS has
not been defined and developed in the context of a DOD or Navy
enterprise architecture because a well-defined version of either has
not existed to guide and constrain the program, and meaningful analysis
showing how NTCSS aligns to evolving DOD and Navy architecture efforts
was not produced. This means that the Navy does not have a sufficient
basis for knowing if NTCSS, as defined, properly fits within the
context of future DOD and Navy business operational and technological
environments.
More specifically, a well-defined enterprise architecture provides a
clear and comprehensive picture of an entity, whether it is an
organization (e.g., a federal department) or a functional or mission
area that cuts across more than one organization (e.g., personnel
management). This picture consists of snapshots of both the
enterprise's current or "As Is" environment and its target or "To Be"
environment, as well as a capital investment road map for transitioning
from the current to the target environment. These snapshots consist of
integrated "views," which are one or more architecture products that
describe, for example, the enterprise's business processes and rules;
information needs and flows among functions; supporting systems,
services, and applications; and data and technical standards and
structures. GAO has promoted the use of architectures to guide and
constrain systems modernization, recognizing them as a crucial means to
a challenging goal: agency operational structures that are optimally
defined in both the business and technological environments.
DOD has long operated without a well-defined enterprise architecture
for its business environment. In 2001, we first reported that DOD did
not have such an architecture and recommended that it develop one to
guide and constrain IT business systems, like NTCSS.[Footnote 23] Over
the next 4 years, we reported that DOD's architecture development
efforts were not resulting in the kind of business enterprise
architecture that could effectively guide and constrain business system
investments,[Footnote 24] largely because the department did not have
in place the architecture management structures and processes described
in federal guidance. In particular, we most recently reported in July
2005[Footnote 25] that despite spending about $318 million producing
eight versions of its architecture, DOD's latest version still did not
have, for example, a clearly defined purpose that could be linked to
the department's goals and objectives and a description of the "As Is"
environment and a transition plan. Further, we reported that the
description of the "To Be" environment was still missing important
content (depth and scope) relative to, for example, the actual systems
to be developed or acquired to support future business operations and
the physical infrastructure (e.g., hardware and software) that would be
needed to support the business systems. Over the last several years, we
have also reported that DOD's efforts for determining whether ongoing
investments were aligned to its evolving architecture were not
documented and independently verifiable.[Footnote 26] On September 28,
2005, DOD issued the next version of its business enterprise
architecture,[Footnote 27] which we are required to review, along with
other things such as the department's efforts to review certain
investments' alignment with the architecture, pursuant to the Fiscal
Year 2005 National Defense Authorization Act.[Footnote 28]
The Navy has also not had an enterprise architecture to guide and
constrain its IT system investments. For example, in February 2002 and
November 2003, we reported that while the Navy was developing an
enterprise architecture, the architecture products were not complete
and they were not, for example, under configuration
management.[Footnote 29] Since that time, the Navy has yet to develop
an enterprise architecture. In response to our request for the latest
version of its architecture, the Assistant Secretary of the Navy,
Research Development and Acquisition, Chief Engineer, provided us
documentation that describes high-level principles or goals that the
Navy wants to achieve, such as systems interoperability. However, most
of the critical products that an enterprise architecture should include
were not provided, such as (1) a data dictionary, which is a repository
of standard data definitions for applications; (2) a logical database
model that provides the data structures that support information flows
and that provides the basis for developing the schemas for designing,
building, and maintaining the existing physical databases; and (3) an
analysis of the gaps between the baseline and target architecture for
business processes, information/data, and services/application systems
to define missing and needed capabilities. According to the Deputy
Assistant Secretary of the Navy for Command, Control, Communication,
Computers and Intelligence, and Space, the Navy does not have an
enterprise architecture. However, these officials stated that the Navy
recognizes the importance of developing and using one and is taking
steps to do so. They did not have a time frame as to when this would be
accomplished, however.
In addition, NTCSS program officials told us that the system has been
assessed against DOD's business enterprise architecture, and based on
this assessment, the system is aligned. However, our analysis of the
alignment documentation showed while NTCSS could be mapped to several
enterprise architecture elements (e.g., strategic goals and
organizational roles), it was not mapped to other important elements
(e.g., technical standards and data model). Moreover, as previously
discussed, the version of the enterprise architecture used to assess
alignment lacked utility and did not provide a sufficient basis for
making informed investment decisions.
These officials stated that they have not yet assessed the system
against the Navy's architecture because (1) the architecture has yet to
be sufficiently developed and (2) compliance with this architecture may
not be required.
Without having a well-defined architecture to set the institutional
context within which a given investment like NTCSS must fit and taking
proactive and verifiable steps to understand the extent to which the
system as it is defined fits within this context, misalignments can
occur that can introduce redundancies and incompatibilities and that
can produce inefficiencies and require costly and time consuming rework
to fix. In the case of NTCSS, this could be a problem because of the
Navy's ongoing investment in its ERP program.[Footnote 30] As we
recently reported,[Footnote 31] this program is intended to provide
functionality in such areas as supply and workforce management for
ashore activities, which is functionality similar to that of NTCSS for
afloat activities. However, both programs have proceeded without a
common, institutional frame of reference (i.e., enterprise
architecture) that can be used to effectively manage their
relationships and dependencies. Our research and experience in
reviewing federal agencies shows that managing such relationships on a
program to program basis is untenable and has proven unsuccessful. This
is why the inherent risks associated with investing in systems in the
absence of a well-defined architecture need to be explicitly disclosed
and deliberately evaluated in order to make a well-informed investment
decision.
Key Program Management and Oversight Activities Have Not Been
Effectively Performed:
Key aspects of effective program management include reliable progress
measurement and reporting, appropriate budgeting, and meaningful
oversight. DOD policy requires such activities, and DOD and other
industry best practices provide guidance on how these activities should
be conducted. However, these activities have not been effectively
performed on the NTCSS program. Specifically, the Navy has not
adequately measured progress against planned cost and scheduled work
commitments, fulfilled defined reporting requirements, properly
budgeted for expenditures, and conducted meaningful program oversight.
As a result, opportunities for proactive program intervention and
actions to address risks and problems were missed, allowing the program
to proceed largely unchecked.
The Navy is Not Adequately Measuring Progress Against Planned Cost and
Scheduled Work Commitments:
Measuring and reporting progress against cost and schedule commitments
is a vital element of effective program management. DOD policy and
guidance recognize this by requiring the use of earned value
management, and describing how it is to be performed. The NTCSS program
has elected to use earned value management; however, it is not doing so
effectively. As a result, the program, as well as Navy and DOD
oversight authorities, have not had access to the kind of reliable and
timely information they need to make informed decisions.
DOD Has Adopted Industry Standards for Earned Value Management:
According to DOD policy and guidance,[Footnote 32] program offices
should obtain data from contractors and central design agencies on work
progress, and these data should relate cost, schedule, and technical
accomplishments. Moreover, the guidance states that these data should
be valid, timely, and auditable. The tool that many DOD entities,
including the NTCSS's program office and its central design agency, use
to obtain and report these data is known as earned value management
(EVM). Through EVM, program offices and others can determine a
contractor's or central design agency's ability to perform work within
cost and schedule estimates. It does so by examining variances between
the actual cost and time to perform work tasks and the
budgeted/estimated cost and time to perform the tasks.
In 1996, DOD adopted industry guidance[Footnote 33] that identifies 32
criteria that a reliable EVM system should meet. The 32 criteria are
organized into five categories: organization, planning and budgeting,
accounting, analysis and management reports, and revisions and data
maintenance (see app. III for the 32 criteria). As we previously
reported,[Footnote 34] EVM offers many benefits when done properly. In
particular, it is a means to measure performance and serves as an early
warning system for deviations from plans. It therefore enables a
program office to mitigate the risk of cost and schedule overruns.
NTCSS Has Not Effectively Implemented EVM:
The EVM system that NTCSS has implemented to measure program
performance does not provide the kind of reliable and timely data
needed to effectively identify and mitigate risks. According to the
NTCSS central design agency's self-assessment of its earned value
management system, 17 of the 32 industry best practice criteria are not
being satisfied by the EVM system it has implemented. For example, the
central design agency reported that the system cannot (1) establish and
maintain a budget baseline against which program performance can be
measured over time, (2) identify management reserves in case of
contingencies, (3) record all indirect costs[Footnote 35] that will be
allocated to the work, (4) summarize data elements and associated
variances through the work breakdown structure to support management
needs, and (5) develop revised estimates of cost at completion based on
performance to date.
Beyond this self-assessment, our review showed that 29 of the 32
criteria were not satisfied. For example, the system does not (1)
provide for the integration of planning, scheduling, budgeting, work
authorization, and cost accumulation management process; (2) identify
physical products, milestones, technical performance goals, or other
indicators used to measure progress; (3) reconcile current budgets to
prior budgets in terms of changes to the authorized work and internal
replanning; and (4) control retroactive changes to records. See
appendix III for the Navy's complete self-assessment and our full
analysis of the extent to which the 32 criteria are satisfied.
Officials with the program office and the central design agency stated
that although they chose to use EVM, they are not required by DOD
policy to do so and, therefore, do not have to comply with the 32
criteria. These officials stated that one reason they are not required
to use it is because the program office and the central design agency
are part of the same organization (the Space and Naval Warfare Systems
Command) and thus a formal contract or written agreement between them
does not exist. They also stated that although the program as a whole
exceeds dollar thresholds for which EVM is required,[Footnote 36] they
have chosen to break the program into smaller projects managed on a
fiscal year basis, and none of these projects individually exceeds
either the new or old DOD policy thresholds that would require the use
of EVM.
We do not agree that the absence of a contractual relationship or the
decomposition of the program into small, fiscal year-based projects is
a valid reason for not effectively implementing EVM. DOD and OMB
guidance require that the Navy base programmatic decisions on reliable
analyses of estimated system's costs and expected benefits over the
life of the program. The program office chose to use EVM as a means to
satisfy these requirements and to measure progress and identify
potential problems early, so that they could be effectively addressed.
To accomplish this, EVM must be performed correctly. By not
implementing it correctly on NTCSS, the Navy is losing an opportunity
to gain the kind of visibility into program progress needed to identify
problems and risks early and better ensure program success. Moreover,
by tracking individual projects on a yearly basis the program office
cannot adequately understand the status of the NTCSS program as a
whole, which hinders its ability to accurately forecast program costs
at completion and provide realistic schedule projections. In short,
without reliable, timely, and auditable EVM data, the program office
cannot adequately manage technical, cost, and schedule risks and
problems.
Two NTCSS Projects Illustrate How EVM Has Been Poorly Implemented:
Two of the individual NTCSS projects for which EVM activities were
reportedly being performed are (1) 2004 OOMA software development and
(2) 2004 NTCSS hardware installation and integration (for both OOMA and
Optimized NTCSS). For the OOMA software project, EVM was performed by
the central design agency and for the NTCSS hardware project it was
performed by the Space and Naval Warfare Systems Command Systems
Center, Charleston. On both projects, we found several examples of
ineffective EVM implementation, including the following:
* An integrated baseline review was not conducted for either of the
projects. According to DOD guidance and best practices, an integrated
baseline review should be conducted as needed throughout the life of a
program to ensure that the baseline for tracking cost, technical, and
schedule status reflects (1) all tasks in the statement of work, (2)
adequate resources in terms of staff and materials to complete the
tasks, and (3) integration of the tasks into a well-defined schedule.
Further, program managers are to use cost performance reports that have
been validated by an integrated baseline review. Without verifying the
baseline, monthly cost performance reporting, which is to track against
a set budget and schedule, does not have sufficient meaning or
validity.
* The estimate at completion for the 2004 OOMA software project, which
is a forecast value expressed in dollars representing the final
projected costs of the project when all work is completed, showed a
negative cost for a 6-month period (November 2003 to April 2004). When
EVM is properly implemented, this amount should include all work
completed and always be a positive number. The negative estimate at
completion for this project would mean that the central design agency
had incurred a savings rather than spending money, even though during
that time more than $1.7 million had been spent.
* The schedule performance index for the OOMA software project, which
is to reflect the critical relationship between the actual work
performed versus the costs expended to accomplish the work, showed
program performance during a time when the program office stated no
work was being performed. Specifically, the reports showed the schedule
performance fluctuating between $0.21 worth of work performed for every
dollar spent to more than $3.75 worth of work performed for every
dollar spent during a time that the program office claims all work was
halted. Perfect performance would indicate schedule indices equal to
1.0 at best (i.e., for every dollar spent there was 100 percent of the
schedule achieved).
* The estimate at completion for the OOMA hardware installation project
showed that almost $1 million in installation costs had been removed
from the total sunk costs, but no reason for doing so was provided in
the cost performance report.
* The cost and schedule indices for the OOMA hardware installation
project showed improbably high program performance during a time when
the installation schedules and installation budget had been drastically
cut because OOMA software failed operational testing. Specifically, the
reports between March 2004 and July 2004 showed the current cost
performance fluctuating between $0.07 worth of work performed for every
dollar spent to $8.48 worth of work performed for every dollar spent.
Navy officials cited several reasons for these shortcomings. For the
software project, program officials stated that prior to the
operational testing of OOMA in 2003, the central design agency's
implementation of EVM was primitive at best and that the resulting data
were not usable. They also stated that after the project failed
operational testing, they did not see the value in rebaselining the
project and thus all EVM analysis was halted. They did, however,
continue to invest in OOMA. For the hardware installation project, a
Charleston Center official responsible for developing the installation
reports stated that there were problems with collecting actual costs
because the individuals responsible for doing the work were covered by
other contracts, and there was no way to ensure that the costs were
being reported consistently. Regarding the approximately $1 million in
installation costs that were removed from the total sunk costs, this
official stated that these costs were erroneously charged to this
project and were thus removed because they were not part of the
original plan.
Ineffective implementation of EVM, as occurred on these two projects,
precludes NTCSS program officials from having reliable and timely
information about actual program status and does not provide these
officials with a sound basis for making informed program decisions.
The Navy Has Not Adequately Reported NTCSS's Progress and Problems:
One essential aspect of effective program management is complete and
current reporting by the program office to oversight organizations
responsible for making decisions regarding the program's future. DOD
policy recognizes this, stating that the program office is accountable
for providing credible schedule, performance, and cost reporting
information to the milestone decision authority. Officials from the
NTCSS milestone decision authority told us that they relied on the
program office to fully disclose progress against, and deviations from,
program cost, schedule, and performance goals. However, the program
office has not reported consistently or reliably on the program's
progress and, as a result, has not fully disclosed program status to
Navy and DOD oversight authorities who are responsible for making
proper investment decisions.
Navy Reporting Requirements for NTCSS Have Changed over the Last
Several Years:
Since program inception, NTCSS requirements for reporting cost,
schedule, and performance information have changed. Prior to October
2002, the program office was required to comply with applicable DOD
acquisition policies and guidance.[Footnote 37] This guidance generally
required the program office to provide oversight organizations with the
following three key reports:
* The Acquisition Program Baseline, which describes the program's cost,
schedule, and performance goals. This baseline document is to be
developed when the program is initiated, and it is to be updated for
each milestone review. Within 90 days of a program breach,[Footnote 38]
unless the program is back within its baseline goals, a new Acquisition
Program Baseline is to be prepared by the program office and approved
by the milestone decision authority.
* The Program Deviation Report, which is to be prepared when the
program office identifies deviations from the approved Acquisition
Program Baseline goals. More specifically, when the program office has
reason to believe that a program breach will occur, it is to
immediately notify the milestone decision authority. Within 30 days,
the program office is to inform the milestone decision authority of the
reason for the deviation and the actions it considers necessary to
bring the program back within baseline goals.
* The Defense Acquisition Executive Summary, which is prepared to
inform the milestone decision authority on the program's progress
against cost, schedule, and performance goals reflected in the
Acquisition Program Baseline. Prepared quarterly, the summary is
designed to provide an early warning to the DOD Chief Information
Officer (CIO) and the milestone decision authority by identifying
existing and potential program problems and describing mitigating
actions that have been taken.
Between October 2002 and December 2004, the reporting requirements for
the program changed.[Footnote 39] As previously discussed, NTCSS was
selected by its milestone decision authority to participate in the RIT
pilot, which was aimed at saving time in the acquisition management
process by reducing traditional DOD reporting and oversight
requirements, while still adhering to DOD acquisition guidance. Under
the RIT pilot, the program office was required to prepare the following
two monthly electronic reports:
* The Monthly Acquisition Program Review, which was to assess the
current health of the program on a monthly basis in such areas as cost
and schedule performance, testing, funding, and contracting. This
report was broken into eight parts. According to the program office,
the main part for NTCSS was the Program Manager Assessment.
* The Smart Chart, which was to address risks for different projects
within the program, including a description of the risk, actions taken
to address the risk, and recommendations for further actions. The Smart
Chart was also to contain any updates to the Acquisition Program
Baseline.
In short, the RIT reporting was to provide the same information
reported via the traditional acquisition baseline and the summary
report, but it was to be more frequent (monthly versus quarterly) and
use a different format (electronic versus paper). In addition, under
the RIT pilot, certain acquisition documents, such as acquisition
plans, economic analyses, requirements documents, and test plans, were
to be posted to the RIT Web site's electronic library rather than sent
in hard copy to the program's stakeholders.
In December 2004, the program office and the milestone decision
authority agreed to discontinue use of the RIT pilot procedures. In
January 2005, the reporting requirements reverted to the acquisition
policies and procedures as prescribed in the updated DOD 5000 series.
Currently, the program office is required to prepare the summary report
quarterly and the acquisition baseline as needed. Also, in January
2005, the Navy required the program office to begin making entries into
the Dashboard. The Dashboard, like the summary report, is prepared by
the program office on a quarterly basis for the milestone decision
authority and is to provide an assessment of the program in such areas
as cost, schedule, and performance characteristics.
The Navy Has Not Satisfied All NTCSS Reporting Requirements:
The program office did not comply with the reporting requirements that
were in effect during the 27 months of the RIT pilot. Specifically:
* The Smart Chart was not updated for 19 of the 27 months.
Specifically, the data were updated eight times between October 2002
and November 2003; the data were not updated after November 2003.
* The Program Manager Assessment was not updated for 11 of the 27
months. In addition, the updates were not always made in a timely
manner. For the 16 months that were updated, 7 were done after the
month had ended, and most of these updates were a month late.
* Of the 15 essential acquisition documents that the program office
committed to entering in the RIT electronic library, 10 were not
entered. For example, the most recent economic analysis and the test
and evaluation master plan for OOMA were not entered.
* The Program Deviation Report and Acquisition Program Baseline were
not prepared in a timely manner. Specifically, in April 2004, the
acquisition of eNTCSS was cancelled and, in May 2004, OOMA did not pass
operational testing--two events that caused the related cost and
schedule thresholds in the Acquisition Program Baseline to be breached.
While program officials had notified the milestone decision authority
of these events via (1) e-mail, (2) entries into the Program Manager
Assessment on the RIT Web site, and (3) briefings, the program office
did not prepare a Program Deviation Report until about 15 months later.
Moreover, this deviation report addressed only the OOMA failure, not
the cancellation of eNTCSS and reprogramming of unexpended eNTCSS
funding. In addition, program officials have yet to provide us with a
new Acquisition Program Baseline to reflect the program breach or
documentation showing that this revised baseline has been approved by
the milestone decision authority.
For the DOD and Navy reporting requirements in effect since January
2005, the Navy has satisfied some, but not all, of the reporting
requirements. For example, the program office has prepared the
Dashboard reports quarterly as required. However, it has not prepared
the Defense Acquisition Executive Summary quarterly as required; the
first report was not prepared until June 2005--6 months after the
requirement resumed and the report was due.
Program officials provided various reasons why the required program
reporting has not occurred. In the case of the Smart Charts and the
Program Manager Assessment reports, a contractor supporting the
Assistant Program Manager stated that the data may have been entered
into the Web site but not properly saved. Regarding the posting of
documents into the electronic library, an official from the milestone
decision authority stated that there was no documentation from the
Office of the Assistant Secretary of Defense for Networks and
Information Integration/Chief Information Officer that directed which,
if any, acquisition documents were to be entered into the RIT Web site.
Similarly, a contractor supporting the Assistant Program Manager stated
that the folders in the electronic library were established by the Army
and thus the Navy was not required to use them. However, our review of
documentation provided by the program office shows that it clearly
states which specific documents should be included in the electronic
library. Regarding the delay in preparation of the Program Deviation
Report and subsequent Acquisition Program Baseline revision, a
contractor supporting the Assistant Program Manager stated that a new
baseline should have been prepared sooner, but that this reporting was
delayed due to the uncertainty of which reporting methods to use after
the end of the formal RIT pilot.
Officials representing the milestone decision authority stated that
they relied on program office reporting on program status and progress,
and that they expected the program office to inform them if the program
exceeded its cost, schedule, and performance thresholds. Without
adequate reporting, oversight officials were not positioned to
effectively execute their roles and responsibilities.
The Navy Has Not Properly Budgeted for NTCSS:
In September 1999, the Navy Comptroller issued guidance directing
program offices to review their budgets and identify efforts that were
being improperly funded and to take the steps necessary to realign
these funds to "Research, Development, Test and Evaluation" as quickly
as possible. Further, DOD Financial Management Regulation[Footnote 40]
requires that IT development, test, and evaluation requirements
generally be funded in the "Research, Development, Test and Evaluation"
appropriations. More specifically it states that, "The Research,
Development, Test and Evaluation funds should be used to develop major
upgrades increasing the performance envelope of existing systems,
purchase test articles, and conduct developmental testing and/or
initial operational test and evaluation prior to system acceptance."
Similarly, Navy financial management policy[Footnote 41] states that,
"All costs associated with software development/modification efforts
that provide a new capability or expand the capability of the current
software program (i.e., expand the performance envelope) are funded in
the Research, Development, Test and Evaluation appropriation."[Footnote
42]
However, this has not occurred. Since 1997, the program office has not
identified "Research, Development, Test and Evaluation" funds in five
of its seven Acquisition Program Baseline documents, three of which
were prepared after the guidance was issued by the Comptroller of the
Navy. Instead, the Navy funded these activities primarily out of the
"Operations and Maintenance," "Other Procurement," and "Ship
Construction" appropriations. (See table 10.)
Table 10: Threshold Amounts in NTCSS Acquisition Program Baselines:
Dollars in thousands.
Revision 0;
Date prepared: March 1997;
Operations and maintenance: $182,986;
Other procurement: $199,636;
Ship construction: $11,683;
Research, development, test and evaluation: $0.
Revision 1;
Date prepared: March 1998;
Operations and maintenance: $257,542;
Other procurement: $303,565;
Ship construction: $23,836;
Research, development, test and evaluation: $3,026.
Revision 2;
Date prepared: December 1998;
Operations and maintenance: $223,370;
Other procurement: $285,550;
Ship construction: $18,220;
Research, development, test and evaluation: $0.
Revision 3;
Date prepared: January 2001;
Operations and maintenance: $276,100;
Other procurement: $382,000;
Ship construction: $27,300;
Research, development, test and evaluation: $0.
Revision 4;
Date prepared: January 2003;
Operations and maintenance: $276,100;
Other procurement: $382,000;
Ship construction: $27,300;
Research, development, test and evaluation: $0.
Revision 5;
Date prepared: July 2003;
Operations and maintenance: $276,100;
Other procurement: $382,000;
Ship construction: $27,300;
Research, development, test and evaluation: $0.
Revision 6;
Date prepared: January 2004;
Operations and maintenance: $376,400;
Other procurement: $346,600;
Ship construction: $25,700;
Research, development, test and evaluation: $29,800.
Source: Navy.
[End of table]
Program officials agreed that they have funded NTCSS development
activities, such as those associated with OOMA, out of the "Operation
and Maintenance" appropriation rather than the "Research, Development,
Test and Evaluation" appropriation. A contractor supporting the
Assistant Program Manager stated that, although they were aware of the
Comptroller of the Navy's budget guidance, the program office chose not
to comply because program officials believed in 1999 that the OOMA
application, which had been under development for 3 years, would pass
developmental testing and operational testing by 2001. As a result,
program officials determined that the effort required to reprogram
funding from the "Operation and Maintenance" appropriation into the
"Research, Development, Test and Evaluation" appropriation was not
warranted. Further, the official stated that although OOMA did not pass
operational testing in 2001, the program office did not fund OOMA with
"Research, Development, Test and Evaluation" funds until 2004 because
it continued to consider OOMA as being close to becoming operational.
The lack of proper budgeting for "Research, Development, Test and
Evaluation" funding has given oversight authorities the misleading
impression that NTCSS development activities were completed and that
the system was fully operational. Specifically, officials from the
Office of the Assistant Secretary of Defense for Networks and
Information Integration/Chief Information Officer, which was the
original NTCSS milestone decision authority, stated that since most of
the "Research, Development, Test and Evaluation" funding appeared to
have been spent, they concluded that the development portion of NTCSS
was essentially complete. As a result, these officials stated that they
had considered taking NTCSS off of the list of programs subject to
oversight reviews. However, after 9 years and over $79 million in
expenditures, the OOMA application still has not passed operational
testing and thus is still in development.
Navy Oversight of NTCSS Has Not Been Adequate:
DOD and Navy policies task a number of organizations with oversight of
IT system acquisition and development programs. For example, DOD policy
states that a milestone decision authority has overall program
responsibility. In addition, the Navy Chief Information Officer is
responsible for reviewing programs at certain points in the acquisition
cycle. Finally, the NTCSS Executive Steering Committee is responsible
for monitoring the near-term development and evolution of the NTCSS
program. However, effective oversight by these entities has not
occurred. As a result, opportunities to address long-standing program
weaknesses have been missed, and the program has been allowed to
proceed virtually unchecked.
The Milestone Decision Authority Has Not Adequately Overseen the
Program:
DOD acquisition policy[Footnote 43] states that a milestone decision
authority is the designated individual with overall responsibility for
a program and is to ensure accountability and maximize credibility in
program cost, schedule, and performance reporting. In this role, the
milestone decision authority is responsible for reviewing the program
throughout its acquisition life cycle, including: (1) whenever the
program reaches a milestone decision point; (2) whenever cost,
schedule, or performance goals are baselined or must be changed; and
(3) periodically through review of management information such as that
found in the Defense Acquisition Executive Summary reports.
However, the Navy milestone decision authority[Footnote 44] has not
conducted such reviews. Specifically:
* The NTCSS program has not reached a milestone decision point in over
5 years. The last such milestone was in April 2000 when the final two
NTCSS Optimized applications became operational. The next scheduled
milestone was to be in 2001, but because OOMA operational testing was
stopped and has yet to be successfully completed, a milestone decision
point has yet to occur. As a result, there has not been a triggering
event that would cause the milestone decision authority to formally
review the program or any of its projects. We discussed the state of
NTCSS in March 2005 with the milestone decision authority's
representatives. In July 2005, the authority was briefed by the program
office. According to program officials, this was the first formal
program review to occur since termination of the RIT pilot in December
2003. These officials also stated that quarterly acquisition team
meetings have since resumed--with the first meeting having occurred in
September 2005 and the next scheduled for December 2005--to prepare for
the next milestone review of OOMA.
* The program office notified the milestone decision authority in April
and June 2004 that OOMA failed operational testing and that eNTCSS was
cancelled via e-mail, entries into the Program Manager Assessment on
the RIT Web site, and briefings. According to officials with the
milestone decision authority, they followed up with the program office
and provided guidance; however, these events did not trigger a formal
program review.
* The milestone decision authority did not contact the program office
to inquire as to the reason why monthly reports were not being prepared
as agreed to after the formal RIT pilot had ended. For example, Smart
Charts were not prepared after November 2003. However, according to
milestone decision authority officials, they did not seek an
explanation from the program office as to why. Milestone decision
authority officials told us that they were relying on the Dashboard
report in order to stay informed on the program's progress. However,
they did not require the program office to begin preparing the
Dashboard report until January 2005.
According to DOD and Navy officials, including officials from the
Office of the Assistant Secretary of Defense for Networks and
Information Integration/Chief Information Officer, the Navy milestone
decision authority, and the program office, NTCSS participation in the
RIT pilot resulted in disruption of normal oversight activities, which
have yet to be fully restored. They added that compounding this is the
fact that the Navy's milestone decision authority's staffing has been
reduced in recent years. According to these officials, approximately 2
years ago the number of full time staff in the Office of the Deputy
Assistant Secretary of the Navy for Command, Control, Communication,
Computers and Intelligence, and Space was reduced from 16 to 6 people,
and these 6 are responsible for reviewing approximately 60 acquisition
programs. The officials stated that, given the large number of programs
and limited staffing, they are unable to fully perform oversight
activities so they have increasingly relied on the program executive
office's assistance to perform detailed oversight of this program.
Without adequate oversight by the milestone decision authority, the
NTCSS program has been allowed to proceed despite the program
weaknesses discussed in this report.
Other Navy Organizations Have Not Conducted Program Oversight:
While the milestone decision authority is the main program oversight
entity, two other Navy organizations have oversight responsibilities.
However, these entities also have not performed effective oversight of
the program. Specifically,
* Department of Navy CIO is responsible for reviewing programs at
certain points in the acquisition cycle to ensure, among other things,
that program goals are achievable and executable and that the program
is providing value (i.e., producing a positive return-on-investment).
Navy CIO officials stated that they have overseen NTCSS primarily by
reviewing the Capital Investment Reports[Footnote 45] prepared by the
program office. They stated that they have not performed any proactive
activities to verify and validate the program's status and progress.
Instead, they rely on information in the Capital Investment Reports,
such as economic justification; budget information by appropriation
type; and cost, schedule, progress, and status. However, as was
discussed previously, the program office does not have or has not
reported reliable information on these topics.
* The NTCSS Executive Steering Committee is responsible for
establishing priorities for NTCSS development and implementation and
determining the strategic direction of the program. Among other things,
it is to meet immediately following each major NTCSS program meeting.
However, it has not met since December 2002, even though the program
office convened both a Requirements Integrated Product Team meeting and
a Forum meeting in February 2005. Further, during this period, major
setbacks occurred on the program, including the failure of OOMA to pass
operational testing and the cancellation of eNTCSS, which were issues
that affected the direction of the program and its priorities and thus
were consistent with the committee's charter. Program officials agreed
that the Executive Steering Committee has not formally convened during
this time frame. However, program officials stated that members of the
committee informally met to discuss and provide advice regarding OOMA
concerns, and Navy officials higher than the Executive Steering
Committee made the decision to cancel eNTCSS. Therefore, these
officials stated there was no need to formally convene an Executive
Steering Committee meeting. Program officials stated that the Executive
Steering Committee will be meeting in January 2006.
NTCSS Requirements and Test Management Weaknesses Have Contributed to
Deployment Delays and System Quality Problems:
As we have previously reported,[Footnote 46] the effectiveness of the
processes used to develop a system is a reliable predictor of the
quality of the system products produced. Two key system development
processes are requirements development and management and test
management. For the NTCSS application currently under development, we
found weaknesses with both of these process areas. While improvements
are planned, until they are implemented effectively, the risk of
continued NTCSS cost, schedule, and performance shortfalls persists.
The Navy Has Not Adequately Managed Requirements for the NTCSS
Application Currently Under Development:
Well-defined requirements can be viewed as a cornerstone of effective
system development and implementation. Accordingly, DOD guidance and
industry best practices recognize effective requirements development
and management as an essential system development and acquisition
management process. For the NTCSS application that is currently under
development--OOMA--the Navy has not adequately managed its 732
requirements, as evidenced by a lack of requirements traceability and
prioritization. NTCSS program officials told us that NTCSS requirements
development practices have historically been poor, but that
improvements are under way. Without effective requirements management,
it is likely that the Navy's challenges to date in developing NTCSS
applications that meet user needs on time and on schedule will
continue.
Requirements for OOMA Release 4.10 Were Not Traced:
DOD guidance and industry best practices also recognize the importance
of requirements traceability.[Footnote 47] The purpose of requirements
traceability is to ensure that the finished product is compliant with
the requirements. To do this, the system documentation should be
consistent and thus complete, allowing for requirements traceability.
Requirements traceability involves both the alignment and consistency
backward to system documentation and forward to system design and test
documentation.
OOMA release 4.10 requirements were not traced to an Operational
Requirements Document. According to DOD guidance,[Footnote 48] an
Operational Requirements Document translates nonsystem-specific
statements of a needed operational capability into a set of validated
and prioritized user requirements. However, the Navy did not develop an
Operational Requirements Document for NTCSS. As a result, the Navy did
not take a basic validation step to ensure that the requirements to
which it designed and built the application were complete and correct.
In addition, release 4.10 requirements were not always traceable to
associated system specifications. Specifically, we were unable to trace
215 requirements found in the system segment specification to the
requirements listed in the requirements checklist. Requirements should
also be traced to test cases, but the program office has yet to provide
us with the developmental test cases used to test the OOMA release 4.10
so that we could verify this traceability.
Program officials acknowledged that release 4.10 requirements were not
traceable but that improvements are planned for the next OOMA release.
We found that 97 percent of the OOMA release 5.0 requirements found in
the system segment specification were traceable to the requirements
listed in the requirements checklist. However, these documents have yet
to be approved. Requirements should also be traced to test cases, but
the program office has yet to provide us with the developmental test
cases used to test the OOMA release 5.0 so that we could verify this
traceability. Without this traceability, the Navy has not had a
sufficient basis for knowing that the scope of its development efforts,
including testing, provides adequate assurance that applications will
perform as intended.
Requirements for OOMA Release 4.10 Were Not Prioritized:
According to published best practices guidance,[Footnote 49] any
project with resource limitations should establish the relative
priorities of the requested features or requirements. Prioritization
helps the project office resolve conflicts, make trade-off decisions
among competing requirements, and helps to ensure that the delivered
system will be operationally suitable.
However, OOMA's approximately 732 requirements have never been
prioritized, and a program official told us that they are all
considered to be equally important. This means, for example, that a
requirement that dictates basic application functionality (e.g., if
text can be entered on a particular screen) is as important as a
requirement addressing safety issues that, if not met, could result in
the loss of an aircraft or even a life.
This lack of requirements prioritization contributed to release 4.10
passing developmental testing but failing operational testing. (See
later section of this report for a detailed discussion of OOMA
testing.) A developmental testing threshold that the Navy set for
release 4.10 was that each requirement was to be tested, and 95 percent
of the requirements had to pass in order for the application to proceed
to operational testing. For developmental testing of the OOMA release
4.10, 97 percent of the requirements passed. Of the 3 percent of the
requirements that failed this test, some of these deficiencies
seriously impacted squadron level operations. Further, for operational
testing of OOMA release 4.10, 96 percent of the requirements passed.
However, the remaining 4 percent contained significant defects.
Specifically, the release provided inconsistent and inaccurate flight
and usage hours, as well as incorrect aircraft usage records. According
to the Navy's independent operational test organization, these
deficiencies impacted aircraft and component time-based inspection
cycles and thus were the basis for the system failing operational
testing. The Navy has yet to provide evidence that the requirements
have been prioritized for the OOMA release 5.0.
The Navy's Developmental Testing for OOMA Has Not Been Effective, but
Improvements Planned:
Both DOD policy and relevant guidance recognize that effective testing
is an essential component of system development or acquisition
programs. Generally, testing can be viewed as consisting of two major
phases--a developmental phase in which tests are performed to ensure
that defined system requirements and specifications are met and an
operational phase that includes tests to determine if the system meets
user needs and is suitable in an operational environment. The OOMA
application has failed operational testing twice over the last 4 years
reportedly because of deficiencies in developmental testing. Program
officials attributed developmental testing deficiencies to poor
software development practices, such as the earlier discussed
requirements development problems. These testing deficiencies can also
be attributed to incomplete testing documentation. Without effective
developmental testing, there is an increased risk that application
problems will be detected later in the system life cycle when they are
more expensive and difficult to fix.
Navy Operational Testing Organization Reported That Developmental
Testing Has Failed to Identify Problems:
According to DOD guidance and recognized best practices,[Footnote 50]
the purpose of developmental testing is to provide objective evidence
that the product (e.g., software module, application, system) satisfies
defined requirements and performs as intended. Successful completion of
developmental testing provides the basis for proceeding into
operational testing to determine whether the integrated product (e.g.,
application, system, system of systems) performs as intended in an
operational or real-world setting.
OOMA operational testing results over the last 4 years show that the
program office's developmental testing efforts have not been effective
in identifying critical product problems. In particular, the
application has failed operational testing twice during this time frame
and, according to an official in the office of the Director of Navy
Test and Evaluation and Technology Requirements, the failures occurred
in operational testing because they were not identified during
developmental testing. More specifically,
* In March 2001, the program office certified that OOMA release 3.25
had passed developmental testing and was ready for operational testing.
However, 1 month into a scheduled 3-month operational test, the
decision was made to cease further testing because of significant
problems with system reliability, data transfer between the application
and the database, and user training on the application. As a result,
the program office decertified this release, and the Navy's independent
test organization recommended discontinuing OOMA deployment.
* Using results from the failed operational test, the central design
agency developed release 4.0. In February and March 2002, developmental
testing of this release was conducted. Test results showed that the
application was not ready for operational testing because it did not
satisfy key functional requirements. Subsequently, the central design
agency incorporated software fixes in release 4.10. In August and
September 2002, developmental testing was conducted on this release
and, while a number of deficiencies were verified as fixed, additional
corrections were needed. From January to June 2003, developmental
testing was again conducted on OOMA release 4.10.
* From August 2002 to April 2003, the Naval Audit Service[Footnote 51]
reviewed OOMA and reported several problems that would affect the
application's readiness for operational testing. For example, it
reported that controls to prevent unauthorized access were not in
place, Privacy Act information was not adequately protected, and backup
and recovery procedures were not in place. It also reported that the
program had not adopted and implemented a risk-based system life cycle
management approach. According to the report, these weaknesses could
compromise safety, affect planning, and distort readiness reporting if
OOMA was implemented throughout the Navy.
* In June 2003, the program office certified OOMA release 4.10 as
having passed developmental testing and being ready for operational
testing. The Navy's independent operational test organization
subsequently conducted testing from August to December 2003 and, in May
2004,[Footnote 52] this organization concluded that OOMA was not
operationally effective or suitable and thus it again failed
operational testing. In particular, the operational testing results
showed that the application was incorrectly calculating flight and
component usage hours--defects, which according to an official in the
office of the Director of Navy Test and Evaluation and Technology
Requirements, could have resulted in the loss of aircraft or life. The
Assistant Program Manager also told us that release 4.10 did not
address all of the deficiencies reported by the Naval Audit Service.
For about a year, the central design agency has been developing and
testing OOMA release 5.0 to fix the problems found in the prior
version. The program office expects that this release will be certified
as ready for operational testing sometime between April and June 2006.
In preparation for operational testing, the Navy's independent
operational test organization has been observing OOMA 5.0 developmental
testing. A memo from this organization states that this release is an
improvement over the previous releases.
According to Navy officials, including the NTCSS Assistant Program
Manager and the official responsible for OOMA developmental testing,
previous application development practices were poor, which led to
testing problems. Specifically, they cited poor requirements
definitions, poor documentation, and concurrent development of
application releases as examples. Further, Navy officials stated that
the central design agency has not had a developmental testing lab to
facilitate effective testing of application components and their
integration. To address the poor development practices, program
officials told us that they are in the process of implementing a new
system life cycle management process that they said incorporates
industry best practices, including those related to testing. However,
the program office has yet to provide us with information defining how
the practices in this plan will be implemented. To address the need for
a developmental testing lab, the Naval Air Systems Command organization
representing NTCSS users recently created a lab to strengthen the
program's developmental testing capability. According to officials
associated with the lab, they are finding defects that the central
design agency should have found.
It is important that the NTCSS program improve its developmental
testing. Without effective developmental testing, there is an increased
risk that system application problems will be detected late in the
system life cycle, such as during operational testing. Generally,
problems discovered late in the cycle are more expensive and difficult
to fix than those discovered early.
Developmental Test Documentation Has Not Been Adequate, but
Improvements Planned:
To be effective, testing should be approached in a rigorous and
disciplined fashion. One aspect of such testing is developing and using
various testing documentation. DOD policy, guidance, and related best
practices[Footnote 53] state that such documentation includes a test
and evaluation master plan for the program, as well as documentation
that is system product (e.g., module, application, system) and test
type (e.g., integration, stress, regression, developmental) specific.
This documentation includes approved test plans, test procedures and
cases, and test results. According to DOD and other guidance, test
plans should include, among other things, objectives, responsibilities,
resources (tools, people, and facilities), schedules, and performance
and exit criteria; test procedures should include detailed test
scenarios, test events, steps, inputs, and expected outputs that are
traced back to requirements. Test results include the test scenarios
that passed and failed, assessments of deviations from test plans, and
the extent to which requirements have been met.
The NTCSS test and evaluation master plan identified, among other
things, three phases of developmental testing for OOMA release 4.10.
However, key test documentation for each of these phases was not
produced. Specifically,
* For the first phase, a test report was produced that contained
detailed information on test results, but the program office has yet to
provide us with a test plan or test procedures.
* For the second phase, a test report was produced but it only
contained the number of defects found (organized by severity) and did
not include any other information on test results. Moreover, the
program office has yet to provide us with a test plan or test
procedures.
* For the third phase, both a test plan and test report were produced,
and the plan included the test purpose and objectives, schedule,
responsibilities, and people resources, while the test report described
test issues and contained detailed test results. However, the program
office has yet to provide us with test procedures.
According to Navy officials, including the Assistant Program Manager
and officials responsible for developmental testing, the previously
mentioned poor application development practices contributed to the
absence of testing documentation. To address these poor practices, the
program has developed a system life cycle plan that they said
incorporates industry best practices, including those associated with
testing documentation. However, the program has yet to provide us with
plans defining how these practices will be implemented. Moreover, while
the plan contains a recommended list of testing documents (e.g., test
plan, test procedures, and test results report), our review of OOMA
release 5.0 developmental testing documentation shows that not all the
documentation is being prepared. Specifically, available documentation
included an updated test and evaluation master plan and two test
reports. Documentation not yet provided to us included test procedures,
which would include documentation tracing test cases to requirements.
The lack of a full set of developmental test documentation is a
problem. Without such documentation, the adequacy and reliability of
developmental testing cannot be substantiated, and thus the quality of
the associated system products is in doubt.
Central Design Agency Reports Management Improvements are Under Way:
In an effort to improve its performance on NTCSS and other programs,
central design agency officials told us that they chose to undergo an
SEI Capability Maturity Model Software Capability Appraisal in July and
August 2005. Carnegie Mellon University's SEI, recognized for its
expertise in software and system processes, has developed the
Capability Maturity Model™ for Software (SW-CMM)[Footnote 54] to
provide guidance on how to gain control of their processes for
developing and maintaining software and how to evolve toward a culture
of software engineering and management excellence.
In brief, SW-CMM calls for assessing different process areas--clusters
of related activities such as project planning, requirements
management, and quality assurance--by determining whether key practices
are implemented and whether overarching goals are satisfied. Successful
implementation of these practices and satisfaction of these goals
result in the achievement of successive maturity levels. SW-CMM
maturity levels range from 1 to 5, with level 1 meaning that the
process is either characterized as ad hoc and occasionally even chaotic
with few processes defined and success depending on individual effort;
level 2 meaning that the process is repeatable; level 3 meaning that
the process is defined; level 4 meaning that the process is managed;
and level 5 meaning that the process is optimized.
According to the central design agency they achieved a maturity rating
of level 3 against the SW-CMM based on 13 process areas, including
requirements management, software project planning, software project
tracking and oversight, subcontract management, software quality
assurance, software configuration management, organizational process
focus, organizational process definition, training program, integrated
software management, software product engineering, intergroup
coordination, and peer reviews. Further, we were told that NTCSS was
one of the programs included in the review. However, we have yet to
receive the appraisal report to determine the extent to which the
appraisal addressed the weaknesses discussed in this report.
Nevertheless, our research has shown that properly performing such
appraisals can be a useful starting point for making software and
system related development improvements.
Conclusions:
It is unclear whether the Navy's planned investment in NTCSS is
warranted. Of particular concern is the absence of reliable analysis
showing that further investment will produce future mission benefits
commensurate with estimated costs, as well as the void in information
concerning whether the deployed and operational components of NTCSS are
actually producing expected value. Compounding this uncertainty is the
inherent risk of defining and developing NTCSS outside the context of
either a well-defined DOD or Navy enterprise architecture. Without this
information, the Navy cannot determine whether NTCSS as defined, and as
being developed, is the right solution to meet its strategic business
and technological needs.
Even if these uncertainties were to be addressed, and the Navy had the
data needed to demonstrate that NTCSS plans are the right course of
action, then the manner in which NTCSS is being defined, developed,
tested, measured, and overseen would still be of concern. While any one
of the concerns that we found is troubling, their combination subjects
the program to an unacceptably high risk of failure. These effects are
being realized on NTCSS, as evidenced by the cancellation of one system
component and the repeated failure of another key component to pass
testing.
It is extremely important that Navy and DOD authorities responsible and
accountable for ensuring prudent use of limited resources reassess
whether allowing NTCSS to continue as planned is warranted. It is also
important that the decision on how to proceed be based on reliable data
about program cost, benefits, risk, and status.
Recommendations for Executive Action:
We recommend that the Secretary of Defense direct the Secretary of the
Navy to determine if continued investment in NTCSS, as planned,
represents a prudent use of the department's limited resources. To
accomplish this, the Secretary of the Navy should direct the program
office to take the following three actions:
* collaborate with the Office of the Assistant Secretary of Defense for
Networks and Information Integration/Chief Information Officer, the
Office of Program Analysis and Evaluation, and the Naval Cost Analysis
Division to prepare a reliable economic analysis that encompasses all
viable alternatives, including the Navy's recent enterprise resource
planning program;
* ensure that development of this economic analysis (1) complies with
cost estimating best practices, including recognition of costs to
resolve open trouble reports and change proposals, and relevant OMB
cost benefit guidance and (2) incorporates available data on whether
deployed NTCSS capabilities are actually producing benefits; and:
* collaborate with the Undersecretary of Defense for Acquisition,
Technology, and Logistics and the Under Secretary of Defense
(Comptroller) to ensure that NTCSS is adequately aligned with evolving
DOD and Navy enterprise architectures.
In addition, we recommend that the Secretary of Defense direct the
Secretary of the Navy to present the results of these analyses to the
Deputy Secretary of Defense, or his designee, and seek a departmental
decision on how best to proceed with the program. Until this is done,
we recommend that the Secretary of Defense direct the Secretary of the
Navy to halt further deployment of NTCSS and to limit future investment
in already deployed applications to essential operation and maintenance
activities and only developmental activities deemed essential to
national security needs.
If--based on reliable data--a decision is made to continue the NTCSS
program, we recommend that the Secretary of Defense direct the
Secretary of the Navy to ensure that the following two actions are
taken:
* the NTCSS program implements effective program management activities,
including earned value management, requirements development and
management, and test management; and:
* key stakeholders, such as the central design agency and the
developmental testing organization, have the people, processes, and
tools to effectively execute their respective roles and
responsibilities.
Finally, we recommend that Secretary of Defense reestablish the Office
of the Assistant Secretary of Defense for Networks and Information
Integration/Chief Information Officer as the milestone decision
authority and direct the Secretary of the Navy to take steps to ensure
that Navy oversight entities fulfill their roles and responsibilities
on NTCSS, including ensuring that reliable program reporting occurs and
is acted upon.
Agency Comments and Our Evaluation:
In its written comments on our draft report, signed by the Deputy to
the Assistant Secretary of Defense for Networks and Information
Integration (Command, Control, Communications, Intelligence,
Surveillance, and Reconnaissance and Information Technology
Acquisition) and reprinted in appendix IV along with our detailed
responses, DOD stated that some of our findings are valid. For example,
it acknowledged that NTCSS was defined and implemented without a
complete and formal enterprise architecture. However, it also commented
that our overall findings significantly understated and misrepresented
the program's level of discipline and conformance with applicable
guidance and direction. The department added that NTCSS "has proven to
be the right solution to meet the Navy's strategic business and
technological needs," and that sound program management practices are
in place and improving.
Neither DOD's comment about our overall findings nor its claims about
NTCSS being the right solution and being effectively managed are
adequately supported, as evidenced by the numerous factual instances
that we site in the report where the Navy did not comply with either
DOD acquisition policies and guidelines or industry best practices.
Specifically, the report shows that the program's latest economic
analysis did not provide the Navy a reliable basis upon which to make
investment decisions. For example, the analysis did not include
measurable, quantifiable benefits for each alternative, and the cost
estimates did not meet six of the nine criteria associated with
reliable cost estimates. The analysis also was not independently
reviewed in accordance with DOD guidance and the Navy had yet to
demonstrate that already deployed NTCSS Optimized applications are
producing expected benefits. We appropriately concluded that the Navy
does not know whether the program as defined is the right solution to
meet the Navy's strategic business and technological needs.
With respect to our recommendations, DOD fully concurred with two of
the recommendations and partially concurred with the remaining five
recommendations. The five areas of disagreement, DOD's basis for its
disagreement, and our response to DOD's position follow.
First, DOD stated that it does not see merit in conducting a formal
economic analysis for the NTCSS program that would address all viable
alternatives because, at this late stage, NTCSS is a "very mature
program," and the final application (OOMA) is about to be fielded.
Further, DOD said it saw no merit in seeking Office of Program Analysis
and Evaluation (PA&E) review of the economic analysis. Instead, it said
that it will "coordinate" with PA&E in analyzing the relationship of
NTCSS with other programs that may provide similar functionality and
"brief the results" to selected stakeholders.
We do not agree that NTCSS is a "very mature program." In particular,
the Navy still plans to spend in fiscal years 2006 through 2009 an
additional $348 million, which is approximately one-third of what has
been spent on the program to date. Further, there is no evidence to
support the claim that the OOMA application is about to be fielded.
OOMA has failed operational testing twice and is not yet fully
developed or tested despite the Navy's initial plan to field it in
2001. In addition, the Navy's stated intention to develop an economic
analysis for OOMA only and then, separately, prepare an "analysis to
determine the relationship" of NTCSS and other alternative programs is
not consistent with guidance and best practice, which advocate basing
such analyses on the full scope of the planned investment. In addition,
the proposal to limit key stakeholders' involvement in developing the
economic justification to "coordinating" and "briefing would be
inappropriate." These stakeholders have specific expertise and roles
relative to economically justifying system investments that should be
exploited. Until it conducts a complete and disciplined analysis of the
entire NTCSS program (reviewed and approved by PA&E and the Naval Cost
Analysis Division) and provides this analysis to all key stakeholders,
the Navy's investment decisions will continue to be made without
complete and reliable data.
Second, the department stated that further deployment of NTCSS should
not be limited at this time. Nevertheless, it stated that it will use
the results of the analysis referred to above that depicts NTCSS's
relationship with other programs to provide appropriate direction to
the program. We do not agree that development should not be limited and
would note that the department's own comment acknowledges the need to
decide on an appropriate direction for the program. In our view,
prudent use of taxpayer resources warrant both a reliable economic
analysis that can be used to inform any decision on this direction and
fiscal restraint to investing until an informed decision can be made.
Third, DOD said that the Navy does not need to be directed to ensure
that effective program management activities are implemented because it
is continuously improving program management activities. Further, DOD
stated that, although it is not required to implement an earned value
management system because the individual projects do not meet the
dollar threshold and there are no formal contract deliverables, it is
nevertheless adhering to the 32 earned value management criteria set
forth in applicable standards. The department added that it intends to
have the Navy Inspector General conduct a separate study to further
ensure that the program is using the best program management
activities.
We do not agree with these comments. In particular, neither during our
review nor in its comments did the Navy provide evidence that it has
implemented effective program management activities or has improvements
under way. As we state in our report, neither the decomposition of the
program into small, fiscal year-based projects nor the absence of a
contractual relationship is a valid reason for not effectively
implementing earned value management. Further, the Navy's earned value
management self-assessment showed that it had not adhered to 17 of the
32 earned value management standards. Without reliable, timely, and
auditable earned value management data, the program office cannot
adequately manage technical, cost, and schedule risks and problems.
Fourth, the department stated that key stakeholders of the NTCSS
program have the necessary people, processes, and tools to effectively
execute their respective roles and responsibilities, noting in
particular that the central design agency has demonstrated its
competency and capability and was certified as SW-CMM maturity level 3.
Nevertheless, the department agreed to address this recommendation.
We support the Navy's stated commitment to address this recommendation.
In addition, we would note that DOD's comment that stakeholders have
the resources they need is not consistent with statements from
stakeholders during our review who said that there were manpower and
resource shortfalls that affected the oversight and execution of
program activities. Further, despite the Navy's statement that the
central design agency achieved SW-CMM maturity level 3, no
documentation supporting this statement, such as appraisal reports,
were provided. Furthermore, Navy officials told us that the central
design agency did not have a development testing lab and was therefore
unable to effectively execute testing activities.
Fifth, DOD stated that it is "premature" to reestablish the DOD Chief
Information Officer as the milestone decision authority as NTCSS
development is over 95 percent complete. Instead, it stated that
existing oversight entities would ensure that effective program
management and reporting was occurring.
We do not agree that elevating the milestone decision authority at this
time is premature based on the statement that the program is 95 percent
complete. For programs that have not been developed using industry best
practices and technical and management discipline, which is the case
for NTCSS, such claims of being essentially complete have historically
proven inaccurate because they are not grounded in reliable performance
data. Moreover, the Navy still plans to spend $348 million on NTCSS
over the next three fiscal years. Finally, as stated in our report, the
current milestone decision authority has allowed the program to operate
unchecked although a major application has repeatedly failed
operational testing, and another application was cancelled.
We are sending copies of this report to interested congressional
committees; the Director, Office of Management and Budget; the
Secretary of Defense; the Deputy Secretary of Defense; the Under
Secretary of Defense for Acquisition, Technology and Logistics; the
Under Secretary of Defense (Comptroller); the Assistant Secretary of
Defense (Networks and Information Integration)/Chief Information
Officer; the Deputy Assistant Secretary of the Navy for Command,
Control, Communication, Computers and Intelligence, and Space; the
Program Executive Office for Command, Control, Communication, Computers
and Intelligence, and Space within the Space and Naval Warfare Systems
Command; the Department of the Navy Chief Information Officer; and the
Office of the Chief of Naval Operations for Material Readiness and
Logistics Operations. This report will also be available at no charge
on our Web site at [Hyperlink, http://www.gao.gov].
If you or your staff have any questions on matters discussed in this
report, please contact me at (202) 512-3439 or [Hyperlink,
hiter@gao.gov]. Contact points for our Offices of Congressional
Relations and Public Affairs may be found on the last page of this
report. GAO staff who made major contributions to this report are
listed in appendix V.
Signed by:
Randolph C. Hite:
Director, Information Technology Architecture and Systems Issues:
[End of section]
Appendixes:
Appendix I: Objective, Scope, and Methodology:
Our objective was to determine whether the Naval Tactical Command
Support System (NTCSS) is being managed according to important aspects
of the Department of Defense's (DOD) acquisition policies and guidance,
as well as other relevant acquisition management best practices. To
accomplish our objective, we focused on the program's (1) economic
justification; (2) architectural alignment; (3) program management,
namely progress measurement and reporting, funding disclosure, and
oversight; and (4) key system development activities, namely
requirements development and management, test management, and system
maturity indicators. For requirements and test management, we focused
on the one NTCSS application that is currently being acquired, known as
the Optimized Organizational Maintenance Activity (OOMA).
To determine whether the Navy has economically justified its investment
in NTCSS, we reviewed the latest economic analysis to determine the
basis for the cost and benefit estimates and net present value
calculations. This included evaluating the analysis against DOD and
Office of Management and Budget (OMB) guidance, as well as relevant
best practices.[Footnote 55] It also included interviewing program
officials, including the Assistant Program Manager; the office of the
Deputy Assistant Secretary of the Navy for Command, Control,
Communication, Computers and Intelligence, and Space; the Office of
Program Analysis and Evaluation; and the Naval Cost Analysis Division
as to their respective roles, responsibilities, and actual efforts in
developing and/or reviewing the economic analysis. In addition, we also
interviewed the Assistant Program Manager and the office of the Deputy
Assistant Secretary of the Navy for Command, Control, Communication,
Computers and Intelligence, and Space about the purpose and use of the
analysis for managing the Navy's investment in the NTCSS program
including the extent to which measures and metrics showed that
projected benefits in the economic analysis were actually being
realized.
To determine whether the Navy has aligned NTCSS to either the DOD
business enterprise architecture[Footnote 56] or a Navy architecture,
we relied on our prior reports addressing DOD and Navy architecture
development and implementation efforts, a memo and analysis results on
NTCSS's compliance with the business enterprise architecture, and
documents on the Navy's architecture efforts. We also interviewed Navy
officials from the program office; the office of the Deputy Assistant
Secretary of the Navy for Command, Control, Communication, Computers
and Intelligence, and Space; the office of the Navy Research,
Development, and Acquisition Chief Engineer; and the Office of the
Assistant Secretary of Defense for Networks and Information
Integration/Chief Information Officer about DOD and Navy architecture
efforts and NTCSS's alignment to them.
To determine whether the Navy was effectively measuring, reporting, and
overseeing the program, we did the following:
* We first asked the central design agency to self-assess their
satisfaction of 32 best practice criteria regarding their earned value
management system. Using the results of their self-assessment to target
our analysis, we then assessed those aspects of the earned value
management system the self-assessment reported as meeting the criteria,
by comparing the documentation with relevant DOD guidance and best
practices.[Footnote 57] We selected these two projects as case studies
to determine the degree to which earned value management was being
implemented. The two projects selected were (1) 2004 OOMA software
project and (2) 2004 NTCSS hardware installation and integration (for
both OOMA and Optimized NTCSS). We selected these two because they were
the projects for which Navy provided us the most earned value
management related documentation. To understand the Navy's reasons why
they were not performing certain elements of earned value management,
we interviewed officials including the Assistant Program Manager, and
officials at the central design agency in Norfolk and the in service
engineering agency in Charleston.
* To assess reporting capabilities, we reviewed program documentation
such as Acquisition Program Baselines, program deviation reports, and
Defense Acquisition Executive Summary reports. We also reviewed
information and documentation on the Rapid Improvement Team pilot Web
site including a report that assesses the current health of the program
on a monthly basis and a report that address risks for different
projects within the program.
* To assess compliance with budget policies and guidance, we compared
NTCSS budget documentation with DOD and Navy financial management
policies and guidance.
* To assess oversight of the program, we interviewed the program
manager, milestone decision authority, functional sponsor, Navy Chief
Information Officer, and a representative of the program's executive
steering committee.
* To determine whether the Navy was effectively managing key system
development activities, namely requirements management, testing, and
system maturity indicators, we did the following:
* To assess requirements development and management capabilities, we
reviewed program documentation such as the official list of
requirements and system specifications, and evaluated them against
relevant best practices[Footnote 58] for several characteristics
including traceability and prioritization. We attempted to trace
requirements to both higher level documents and lower level
specifications. We also attended the NTCSS Forum where requirements
were gathered and discussed. We interviewed Navy officials such as the
Assistant Program Manager, Commanding Officer and Executive Director of
the central design agency, and the OOMA Functional Manager to discuss
their roles and responsibilities for developing and managing
requirements.
* To assess test management, we reviewed program documentation such as
the test and evaluation master plan, test plans, test reports, and
guidance. We then compared these documents with DOD guidance and best
practices and focused on the effectiveness of developmental testing and
the adequacy of developmental testing documentation.[Footnote 59] Our
review:
* also included an audit report prepared by the Naval Audit
Service[Footnote 60] and a test report prepared by Navy's independent
operational test organization.[Footnote 61] We interviewed Navy
officials such as the Assistant Program Manager, Commanding Officer and
Executive Director of the central design agency, OOMA Functional
Manager, and an official in the office of the Director of Navy Test and
Evaluation and Technology Requirements to discuss their roles and
responsibilities for test management.
We did not independently validate information on the program's cost and
budget or the number of trouble reports and change proposals.
We conducted our work at DOD headquarters in Arlington, Virginia; at
Space and Naval Warfare Center, San Diego, California; Space and Naval
Warfare Systems Center, Norfolk, Virginia; and Naval Air Systems
Command in Patuxent River, Maryland. We performed our work from
September 2004 through November 2005 in accordance with generally
accepted government auditing standards.
[End of section]
Appendix II: Trouble Reports and Change Proposals Assessment:
One indicator of system quality, and thus the effectiveness of the
development activities used to produce system products, is the volume
and significance of system problems and change proposals. For the Naval
Tactical Command Support System (NTCSS), trouble reports are prepared
to document system defects, and change proposals are prepared to
introduce additional system functionality. Priority levels are assigned
to trouble reports and change proposals, with 1 being the most critical
and 5 being the least critical. Table 11 defines the 5 priority levels.
Table 11: NTCSS Trouble Report and Change Proposal Priorities:
Priority level: Priority 1;
Definition: Prevents the accomplishment of an operational or mission-
essential capability; and jeopardizes safety or security.
Priority level: Priority 2;
Definition: Adversely affects the accomplishment of an operational or
mission-essential capability, and no work-around solution is available.
Priority level: Priority 3;
Definition: Adversely affects the accomplishment of an operational or
mission-essential capability, but a work-around solution is available.
Priority level: Priority 4;
Definition: Results in user/operator inconvenience or annoyance but
does not affect a required operational or mission-essential capability.
Priority level: Priority 5;
Definition: Any other effect.
Source: Navy.
[End of table]
Available data on the number and significance of open trouble reports
and change proposals over the last 2 years do not demonstrate that
NTCSS overall is a high-quality system that is delivering promised or
expected capabilities. Specifically, the data shows that hundreds of
open (yet to be resolved) trouble reports and change proposals have
continued to affect the system.
Trouble Reports:
The total number of NTCSS priority 1, 2, and 3 trouble reports have
stayed about the same over the last 2 years--totaling about 700. Of
this total, NTCSS priority 1 and 2 trouble reports have decreased by
117, with priority 1 trouble reports being virtually eliminated. While
this is movement in a positive direction, about 300 priority 2 trouble
reports still remain open and these by definition are adversely
affecting accomplishment of an operational or mission-essential
capability. (See figs. 1 and 2.)
Figure 1: Total Number of Open NTCSS and OOMA Priority 1, 2, and 3
Trouble Reports:
[See PDF for image]
[End of figure]
Figure 2: Open NTCSS Priority 1 and 2 Trouble Reports:
[See PDF for image]
[End of figure]
Further, open priority 3 trouble reports have increased during this
time to about 250 and, given that priority 3s require work-arounds,
they decrease system capability and performance. Neither the number of
priority 2 trouble reports, which continue to be in the hundreds, nor
the upward trend in priority 3 trouble reports are indicative of a
maturing, high-quality system. (See fig. 3.)
Figure 3: Open NTCSS Priority 3 Trouble Reports:
[See PDF for image]
[End of figure]
With respect to the OOMA application in particular, the trend in the
volume of significant trouble reports shows that this application is
particularly problematic. Specifically, while priority 1 OOMA open
trouble reports have been virtually eliminated, the number of open
priority 2 OOMA trouble reports has risen significantly from 12 to 90
in about the last 2 years. (See fig. 4.)
Figure 4: Open OOMA Priority 1 and 2 Trouble Reports:
[See PDF for image]
[End of figure]
Moreover, the number of open OOMA priority 3 trouble reports has not
significantly declined over the last 2 years, with these remaining at
roughly 160. (See fig. 5.)
Figure 5: Open OOMA Priority 3 Trouble Reports:
[See PDF for image]
[End of figure]
Change Proposals:
The picture for NTCSS change proposals is similar to that for trouble
reports. Specifically, the total number of open NTCSS priority 1, 2,
and 3 change proposals has increased over the last 2 years--going from
about 325 to 425. Of this total, NTCSS priority 2 change proposals have
increased by 72, with 247 priority 2 proposals still being open. (See
figs. 6 and 7.)
Figure 6: Total Number of Open NTCSS and OOMA Priority 1, 2, and 3
Change Proposals:
[See PDF for image]
[End of figure]
Figure 7: Open NTCSS Priority 1 and 2 Change Proposals:
[See PDF for image]
[End of figure]
Further, NTCSS priority 3 change proposals have increased during this
time to about 81, and given that priority 3 change proposals require
current work-arounds, this is not a positive trend. (See fig. 8.)
Figure 8: Open NTCSS Priority 3 Change Proposals:
[See PDF for image]
[End of figure]
With respect to OOMA specifically, the number of open priority 2 change
proposals has risen slightly from 7 to 12. (See fig. 9.) Similarly, the
number of open priority 3 change proposals has also increased somewhat
from 78 to 97. (See fig. 10.) While the number of priority 2 change
proposals is not large, the trend in these, as well as the trend in the
more significant number of priority 3 change proposals, is not
consistent with those of a maturing system.
Figure 9: Open OOMA Priority 1 and 2 Change Proposals:
[See PDF for image]
[End of figure]
Figure 10: Open OOMA Priority 3 Change Proposals:
[See PDF for image]
[End of figure]
[End of section]
Appendix III: Earned Value Management Assessment:
Earned value management (EVM) guidance was developed by the American
National Standards Institute/Electronic Industries Alliance.[Footnote
62] This guidance identifies 32 criteria that reliable EVM systems
should meet. The 32 criteria are organized into the following five
categories:
* Organization: Activities that define the scope of the effort and
assign responsibilities for the work;
* Planning and budgeting: Activities for planning, scheduling,
budgeting, and authorizing the work;
* Accounting: Activities to accumulate the costs of work and material
needed to complete the work;
* Analysis: Activities to compare budgeted, performed, and actual
costs; analyze variances; and develop estimates of final costs; and:
* Revisions and data maintenance: Activities to incorporate internal
and external changes to the scheduled, budgeted, and authorized work.
NTCSS central design agency (CDA) officials provided a self-assessment
of their compliance with each of the criteria, reporting that they met
15 of the 32 criteria (see table 12). Using the results of their self-
assessment to target our analysis, we then assessed those aspects of
the EVM system the self-assessment reported as meeting the criteria, by
comparing the documentation with relevant Department of Defense (DOD)
guidance and best practices.[Footnote 63] Our assessment indicates that
the NTCSS program satisfied two, and partially satisfied one, of the 32
criteria (see table 12).[Footnote 64]
Table 12: Navy Satisfaction of EVM Criteria:
Organization:
Criteria[A]: Define the authorized work elements for the program. A
work breakdown structure, tailored for effective internal management
control, is commonly used in this process;
Definitions: The work breakdown structure is a direct representation of
the work scope in the project, documenting the hierarchy and
description of tasks to be performed and the relationship to the
product deliverables. The work breakdown structure breaks down all
authorized work scope into appropriate elements for planning,
budgeting, scheduling, cost accounting, work authorization, measuring
progress, and management control. It also ensures the statement of work
is entirely captured and allows for integration of technical, schedule,
and cost information;
Self-assessment: Yes;
GAO assessment: Yes;
GAO analysis: The EVM reports for the OOMA software development project
and the NTCSS hardware installation project had a work breakdown
structure.
Criteria[A]: Identify the program organizational breakdown structure,
including the major subcontractors responsible for accomplishing the
authorized work, and define the organizational elements in which work
will be planned and controlled;
Definitions: The organizational structure identifies the organization
responsible for each segment of work, including subcontracted and intra-
organizational effort. In order to meet this guideline, objective
evidence requires a work breakdown structure intersection with an
organizational breakdown structure;
Self-assessment: Yes;
GAO assessment: No;
GAO analysis: CDA officials have yet to provide documentation to
demonstrate satisfaction of this criterion. Such documentation includes
an organizational breakdown structure with detail regarding
subcontractors.
Criteria[A]: Provide for the integration of the company's planning,
scheduling, budgeting, work authorization, and cost accumulation
processes with each other and, as appropriate, the program work
breakdown structure and the program organizational structure;
Definitions: The integration of planning, scheduling, budgeting, work
authorization, and cost accumulation management processes provides the
capability for establishing the performance measurement baseline,
identifying work progress, and collecting of actual costs for
management analysis and corrective actions;
Self-assessment: Yes;
GAO assessment: No;
GAO analysis: The CDA has yet to provide documentation to demonstrate
satisfaction of this criterion. Such documentation includes copies of
master, intermediate, and detailed schedules; operational schedules;
control account plans; performance reports by work breakdown structure
and organizational breakdown structure; responsibility assignment
matrix; statement of work; work authorization documents; and work
breakdown structure and organizational breakdown structure
documentation.
Criteria[A]: Identify the company organization or function responsible
for controlling overhead (indirect costs);
Definitions: Visibility into direct and indirect costs is essential for
successful management of a project. Therefore, project managers should
clearly identify managers who are responsible for controlling indirect
costs, including overhead, burden, general and administrative costs,
and who has authority to approve expenditure of resources. They should
also document the process for management and control of indirect costs;
Self-assessment: No;
GAO assessment: No;
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met.
Criteria[A]: Provide for integration of the program work breakdown
structure and the program organizational structure in a manner that
permits cost and schedule performance measurement by elements of either
or both structures, as needed;
Definitions: The integration of the work breakdown structure and
organizational breakdown structure establishes where the performance
measurement necessary for project management is performed. This
intersection results in designation of a focal point for management
control (the control account manager). It is also the initiation point
for work authorization, performance management, and performance
measurement. The control account manager identifies the plan for work
task accomplishment, including defining the effort required, cost
elements (labor, material, etc.), and the resources required to do the
job;
Self-assessment: No;
GAO assessment: No;
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met.
Planning and budgeting:
Criteria[A]: Schedule the authorized work in a manner that describes
the sequence of work and identifies significant task interdependencies
required to meet the program requirements;
Definitions: The scheduling of authorized work facilitates effective
planning, reporting, and forecasting, which is critical to the success
of all projects. An integrated network scheduling system has distinct
tasks that can be summarized by work breakdown structure and
organizational breakdown structure identifiers to track progress and
measure performance;
Self-assessment: Yes;
GAO assessment: Yes;
GAO analysis: Detailed schedule documents for both projects describe
the sequence and interdependence of work relative to project
requirements.
Criteria[A]: Identify physical products, milestones, technical
performance goals, or other indicators that will be used to measure
progress;
Definitions: Objective indicators enable measurement of work
accomplished, thereby allowing accurate comparison to planned work.
Meaningful performance metrics enable better management insight and
decision making, allowing maximum time for management action to keep
the project on plan;
Self-assessment: Yes;
GAO assessment: No;
GAO analysis: The metrics in the NTCSS hardware installation project
reports contained unexpectedly and unrealistically large improvements
in performance that were not explained. In addition, the program office
told us that the measurement data for the OOMA software project is
distorted due to numerous baseline changes and requirements changes.
Satisfying this criterion requires valid data.
Criteria[A]: Establish and maintain a time-phased budget baseline, at
the control account level, against which program performance can be
measured. Budget for far-term efforts may be held in higher-level
accounts until an appropriate time for allocation at the control
account level. Initial budgets established for performance measurement
will be based on either internal management goals or the external
customer negotiated target cost, including estimates for authorized but
undefinitized work. On government contracts, if an over-target baseline
is used for performance measurement reporting purposes, prior
notification must be provided to the customer;
Definitions: The assignment of budgets to scheduled segments of work
produces a plan against which actual performance can be compared. This
is called the performance measurement baseline. The establishment,
maintenance, and use of the performance measurement baseline are
indispensable to effective program management;
Self-assessment: No;
GAO assessment: No;
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met.
Criteria[A]: Establish budgets for authorized work with identification
of significant cost elements (e.g., labor and material) as needed for
internal management and for control of subcontractors;
Definitions: An essential part of project planning and establishing a
performance measurement baseline is the establishment of budgets for
all work authorized. Identification of the budget cost elements
documents the required resources and integrates the work scope with the
performing organization;
Self-assessment: No;
GAO assessment: No;
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met.
Criteria[A]: To the extent it is practical to identify the authorized
work in discrete work packages, establish budgets for this work in
terms of dollars, hours, or other measurable units. Where the entire
control account is not subdivided into work packages, identify the far-
term effort in larger planning packages for budget and scheduling
purposes;
Definitions: The effort contained within a control account is
distributed into either work packages or planning packages. Work
packages are single tasks, assigned to a performing organization for
completion, and should be natural subdivisions of control account
effort resulting in a definable end product or event. Budgets
established at the work package level provide the detail for effective
execution of the baseline plan. This approach provides meaningful
product or management-oriented events for performance measurement;
Self-assessment: Yes;
GAO assessment: No;
GAO analysis: The CDA has yet to provide documentation to demonstrate
satisfaction of this criterion. Such documentation includes control
account plans divided into work and planning packages, or control
account schedules and time-phased budgets.
Criteria[A]: Provide that the sum of all work package budgets, plus
planning package budgets within a control account, equals the control
account budget;
Definitions: The integrity of the performance measurement baseline is
maintained when the budget of the control account equals the sum of its
work and planning package budgets. This prevents duplicate recording of
budgets;
Self-assessment: No;
GAO assessment: No;
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met.
Criteria[A]: Identify and control the level of effort activity by time-
phased budgets established for this purpose. Only that effort that is
unmeasurable or for which measurement is impractical may be classified
as level of effort;
Definitions: Meaningful events are critical for performance
measurement. Measurement of level of effort activity provides no
visibility into actual performance. Level of effort activity is defined
as having no measurable output or product at the work package level
and, therefore, must be limited to avoid distorting project performance
data;
Self-assessment: No;
GAO assessment: No;
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met.
Criteria[A]: Establish overhead budgets for each significant
organizational component of the company for expenses that will become
indirect costs. Reflect in the program budgets, at the appropriate
level, the amounts in overhead accounts that are planned to be
allocated to the program as indirect costs;
Definitions: Indirect costs are for common activities that cannot be
specifically identified with a particular project or activity and
should typically be budgeted and controlled separately at the
functional or organization manager level. It is important to have an
indirect budgeting and forecasting process because indirect costs
account for a major portion of the cost of any project. As such, the
budgetary control and management of this category cannot be overlooked
or minimized;
Self-assessment: No;
GAO assessment: No;
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met.
Criteria[A]: Identify management reserves and undistributed budget;
Definitions: Project managers need to realize the performance
measurement baseline planning process contains risk and identify a
management reserve contingency for unplanned activity within the
project scope;
Self-assessment: No;
GAO assessment: No;
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met.
Criteria[A]: Provide that the program target cost goal is reconciled
with the sum of all internal program budgets and management reserves;
Definitions: A project baseline that reflects the common agreement
between the two parties provides a common reference point for progress
assessment. It provides recognition of contractual requirements and
precludes unauthorized changes to the performance measurement baseline;
Self-assessment: No;
GAO assessment: No;
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met.
Accounting considerations:
Criteria[A]: Record direct costs in a manner consistent with the
budgets in a formal system controlled by the general books of account;
Definitions: A project cost-charging structure established in the
accounting system ensures that actual direct costs are accumulated and
reported in a manner consistent with the way the work is planned and
budgeted;
Self-assessment: No;
GAO assessment: No;
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met.
Criteria[A]: When a work breakdown structure is used, summarize direct
costs from control accounts into the work breakdown structure without
allocation of a single control account to two or more work breakdown
structure elements;
Definitions: Actual costs need to be available at all levels of the
work breakdown structure to support project management with performance
measurement data. Cost collection accounts mapped to the work breakdown
structure ensure performance measurement data integrity;
Self-assessment: No;
GAO assessment: No;
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met.
Criteria[A]: Summarize direct costs from the control accounts into the
contractor's organizational elements without allocation of a single
control account to two or more organizational elements;
Definitions: To ensure performance measurement data integrity, actual
costs need to be available at all levels of the organizational
breakdown structure;
Self-assessment: No;
GAO assessment: No;
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met.
Criteria[A]: Record all indirect costs that will be allocated to the
project;
Definitions: All indirect costs should be recorded in the accounting
system. Allocating indirect costs to the appropriate direct costs
assures that all projects benefiting from indirect costs receive their
fair share;
Self-assessment: No;
GAO assessment: No;
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met.
Criteria[A]: Identify unit costs, equivalent unit costs, or lot costs
when needed;
Definitions: A manufacturing accounting system capable of isolating
unit and lot costs in a production environment allows the flexibility
to plan, measure performance, and forecast in a more efficient way when
there are multiple projects in the same production line;
Self-assessment: Yes;
GAO assessment: No;
GAO analysis: The CDA has not yet provided documentation to demonstrate
satisfaction of this criterion. Such documentation includes a
manufacturing resource planning project cost collection structure or an
enterprise resource planning system that supports the identification of
unit costs, equivalent unit costs, or lot costs when needed including
differentiation of work in process.
Criteria[A]: For EVM, the material accounting system will provide (1)
accurate cost accumulation and assignment of costs to control accounts
in a manner consistent with the budgets using recognized, acceptable,
costing techniques; (2) cost performance measurement at the point in
time most suitable for the category of material involved, but no
earlier than the time of progress payments or actual receipt of
material; and (3) full accountability of all material purchased for the
program, including the residual inventory;
Definitions: Material items consumed in the production of project
deliverables are accounted for and progress is measured at the point
most closely aligned to the actual consumption. Material accounting
systems should adhere to these three characteristics: (1) the material
accounting system provides full accountability and effective
measurement of all material purchased; (2) material costs should be
accurately charged to control accounts using recognized, acceptable
costing techniques; and (3) when necessary, the use of estimated actual
costs to ensure accurate performance measurement should be used;
Self-assessment: No;
GAO assessment: No;
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met.
Analysis and management reports:
Criteria[A]: At least on a monthly basis, generate the following
information at the control account and other levels as necessary for
management control using actual cost data from, or reconcilable with,
the accounting system: (1) comparison of the amount of planned budget
and the amount of budget earned for work accomplished (this comparison
provides the schedule variance) and (2) comparison of the amount of the
budget earned and the actual (applied where appropriate) direct costs
for the same work (this comparison provides the cost variance);
Definitions: Visibility into project performance helps the project
manager focus resources on those areas in need of attention. Accurate
and reliable EVM data supports management control needs by allowing the
project manager to identify root causes for variances and establish
actions to minimize impact at the control account level;
Self-assessment: Yes;
GAO assessment: No;
GAO analysis: In order to produce reliable and accurate variance
reports, many of the aforementioned criteria that our analysis showed
that the CDA did not perform must be satisfied. Therefore, this
criterion is not being satisfied.
Criteria[A]: Identify, at least monthly, the significant differences
between both planned and actual schedule performance and planned and
actual cost performance and provide the reasons for the variances in
the detail needed by program management;
Definitions: The analysis of deviations from plan for both schedule and
cost at least monthly provides management at all levels the ability to
rapidly and effectively implement corrective actions with an
understanding of the project risk and causes of risk;
Self-assessment: Yes;
GAO assessment: No;
GAO analysis: The metrics in the NTCSS hardware installation project
reports contained unexpectedly and unrealistically large improvements
in performance that were not explained. In addition, the program office
told us that the measurement data for the OOMA software project is
distorted due to numerous baseline changes and requirements changes.
Satisfying this criterion requires valid data.
Criteria[A]: Identify budgeted and applied (or actual) indirect costs
at the level and frequency needed by management for effective control,
along with the reasons for any significant variances;
Definitions: Ongoing indirect cost analysis provides visibility into
potential indirect cost overruns and the opportunity to develop and
implement management action plans to meet project objectives;
Self-assessment: No;
GAO assessment: No;
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met.
Criteria[A]: Summarize the data elements and associated variances
through the program organization and/or work breakdown structure to
support management needs and any customer reporting specified in the
project;
Definitions: Variances provide an understanding of the conditions,
allowing the project manager to properly allocate available resources
to mitigate project risk. They also identify significant problem areas
from all levels of the organization and project scope of work, derived
from the same data sources. Thus, variances provide valuable management
information;
Self-assessment: No;
GAO assessment: No;
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met.
Criteria[A]: Implement managerial actions taken as the result of earned
value information;
Definitions: Earned value data must be utilized by all levels of
management for effective project execution. Because of this, the data
produced by the EVM system must be available to managers on a timely
basis and must be of sufficient quality to ensure that effective
management decisions can be made as a result of its analysis;
Self-assessment: Yes;
GAO assessment: No;
GAO analysis: The metrics in the NTCSS hardware installation project
reports contained unexpectedly and unrealistically large improvements
in performance that were not explained. In addition, the program office
told us that the measurement data for the OOMA software project is
distorted due to numerous baseline changes and requirements changes.
Satisfying this criterion requires valid data.
Criteria[A]: Develop revised estimates of cost at completion based on
performance to date, commitment values for material, and estimates of
future conditions. Compare this information with the performance
measurement baseline to identify variances at completion important to
company management and any applicable customer reporting requirements,
including statements of funding requirements;
Definitions: Estimates at completion based on predictive performance
measures increase the probability that the project can be executed
within the reported estimates at completion. When estimates at
completions are analyzed at least monthly and updated as required, the
robustness of the financial reporting requirements is enhanced, thereby
reducing the potential for surprises. Monthly estimates at completion
reviews are essential for management decisions including the planning
of project future funding requirements;
Self-assessment: No;
GAO assessment: No;
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met.
Revisions and data maintenance:
Criteria[A]: Incorporate authorized changes in a timely manner,
recording the effects of such changes in budgets and schedules. In the
directed effort prior to negotiation of a change, base such revisions
on the amount estimated and budgeted to the program organizations;
Definitions: The incorporation of authorized changes in a timely manner
maintains the integrity of the performance measurement baseline and
thus its effectiveness as a baseline against which to manage and
control performance;
Self-assessment: Yes;
GAO assessment: No;
GAO analysis: The CDA has yet to provide documentation to demonstrate
satisfaction of this criterion. Such documentation includes change
control logs and work authorization documents.
Criteria[A]: Reconcile current budgets to prior budgets in terms of
changes to the authorized work and internal replanning in the detail
needed by management for effective control;
Definitions: Budget changes should be controlled and understood in
terms of scope, resources, and schedule, and that budgets should
reflect current levels of authorized work. Furthermore, budget
revisions should be traceable to authorized contractual targets and
control account budgets;
Self-assessment: Yes;
GAO assessment: No;
GAO analysis: The CDA has yet to provide documentation to demonstrate
satisfaction of this criterion. Such documentation includes change
documents or change control logs.
Criteria[A]: Control retroactive changes to records pertaining to work
performed that would change previously reported amounts for actual
costs, earned value, or budgets. Adjustments should be made only for
correction of errors, routine accounting adjustments, effects of
customer or management directed changes, or to improve the baseline
integrity and accuracy of performance measurement data;
Definitions: Retroactive changes to the baseline may mask variance
trends and prevent use of the performance data to project estimates of
cost and schedule at completion. Retroactive budget adjustments may
delay visibility of overall project variance from plan, thus reducing
the alternatives available to managers for project redirection or
termination;
Self-assessment: Yes;
GAO assessment: No;
GAO analysis: The CDA has yet to provide documentation to demonstrate
satisfaction of this criterion. Such documentation includes change
control logs or approved retroactive change controls.
Criteria[A]: Prevent revisions to the program budget except for
authorized changes;
Definitions: Changes made outside the authorized baseline control
processes compromise the integrity of performance trend data and delay
visibility into overall project variance from plan;
Self-assessment: Yes;
GAO assessment: No;
GAO analysis: The CDA has yet to provide documentation to demonstrate
satisfaction of this criterion. Such documentation includes change
control logs, control accounts, and work package plans.
Criteria[A]: Document changes to the performance measurement baseline;
Definitions: By ensuring that budget and schedule revisions are
documented and traceable, the integrity of the performance measurement
baseline is maintained and can be verified. The performance measurement
baseline should reflect the most current plan for accomplishing the
effort. Authorized changes should be quickly recorded in the system and
incorporated into all relevant planning. Planning and authorization
documents must also be updated accordingly prior to commencement of new
work;
Self-assessment: Yes;
GAO assessment: Partial;
GAO analysis: We were provided documentation showing eight baseline
changes for the NTCSS hardware installation project. However, the
program office told us that the EVM data for the OOMA software project
is distorted due to numerous baseline changes and requirements changes.
Criteria[A]: Number satisfied;
Self-assessment: 15;
GAO assessment: 2.
Criteria[A]: Number partially satisfied;
Self-assessment: 0;
GAO assessment: 1.
Criteria[A]: Number not satisfied;
Self-assessment: 17;
GAO assessment: 29.
Criteria[A]: Total;
Self-assessment: 32;
GAO assessment: 32.
Sources: Navy CDA self-assessment and GAO analysis of Navy provided
data.
[A] Based on the National Defense Industrial Association Program
Management Systems Committee Intent Guide (January 2005).
[End of table]
[End of section]
Appendix IV: Comments from the Department of Defense:
OFFICE OF THE ASSISTANT SECRETARY OF DEFENSE:
NETWORKS AND INFORMATION INTEGRATION:
6000 DEFENSE PENTAGON:
WASHINGTON, DC 20301-6000:
NOV 23 2005:
Mr. Randolph C. Hite:
Director, Information Technology Architecture and Systems Issues:
U.S. General Accounting Office:
441 G Street, N.W.:
Washington, D.C. 20548:
Dear Mr. Hite:
This is the Department of Defense (DoD) response to the GAO draft
report (06-215), "DoD SYSTEMS MODERNIZATION: PLANNED INVESTMENT IN THE
NAVAL TACTICAL COMMAND SUPPORT SYSTEM NEEDS TO BE REASSESSED," dated
November 3, 2005 (GAO Code 310287).
We appreciate the opportunity to comment on the draft report and the
time your staff afforded us during their preparation of the report.
As acquisition milestone decision authority for NTCSS has been
delegated to the Navy since 1999, the Navy provided significant
analysis and comment regarding the report. The Navy recognizes that
some of the report's findings are valid but believes that the overall
findings significantly understate and misrepresent the level of
discipline and conformance with applicable guidance and direction
achieved by the NTCSS Program. They also believe that although NTCSS
was defined and implemented without a complete and formal enterprise
architecture, it has proven to be the right solution to meet Navy's
strategic business and technological needs; and that sound management
practices are in place and improving. The Navy Acquisition Executive
particularly emphasizes that no manpower or resource shortfalls have
hampered oversight or execution of the NTCSS program.
Rather than providing detailed comments on the report's findings, we
have chosen to focus our response on the report's recommendations. Our
reply to each recommendation is enclosed. Our point of contact is Lt
Col Alex Holder at 703-602-2720, ext 123.
Sincerely,
Signed by:
John R. Landon:
Deputy to the ASD(NII) (C3ISR & IT Acquisition):
Enclosure: As stated:
DoD Comments to GAO draft report (06-215), "DOD SYSTEMS MODERNIZATION:
PLANNED INVESTMENT IN THE NAVAL TACTICAL COMMAND SUPPORT SYSTEM NEEDS
TO BE REASSESSED," dated November 3, 2005 (GAO Code 310287):
RECOMMENDATION 1: The GAO recommended that the Secretary of Defense
direct the Secretary of the Navy to determine if continued investment
in Naval Tactical Command Support System (NTCSS) as planned represents
a prudent use of the department's limited resources. To accomplish
this, the Secretary of the Navy should direct the program office to
collaborate with the Office of the Assistant Secretary of Defense for
Networks and Information Integration/Chief Information Officer, the
Office of Program Analysis and Evaluation, and the Naval Cost Analysis
Division to prepare a reliable economic analysis (EA) that encompasses
all viable alternatives, including the Navy's recent enterprise
resource planning (ERP) program. (p. 60/GAO Draft Report):
DoD RESPONSE: Partially concur.
As planned for some time, the NTCSS program office will continue to
collaborate with the Naval Cost Analysis Division to conduct a reliable
EA to support the Optimized Organizational Maintenance Activity (OOMA)
fielding decision. However, at this late phase of this very mature
program, when the final application is about to be fielded, we do not
see merit in conducting a formal EA that would address all viable
alternatives, or in seeking Office of Secretary of Defense (OSD) Office
of Program Analysis and Evaluation (PA&E) review of the EA.
However, we will direct the Navy to (1) conduct, in coordination with
OSD PA&E, an analysis to determine the relationship of NTCSS, the Navy
ERP Program and other programs that may provide similar functionality
and (2) brief the results of that analysis to the Networks and
Information Integration (NII) Overarching Integrated Product Team
(OIPT) within 120 days.
The NTCSS Program Office has conducted three EAs to support the
program's milestone decisions. These EAs were conducted in accordance
with the DoD 5000 series and OMB Circulars A-94 and A-11.
The 1997 and 1999 EAs focused on NTCSS Optimization (less Optimized
Organizational Maintenance Activity (OOMA)).
The 2004 EA focused on OOMA, the final NTCSS application.
* These EAs were independently reviewed by the Office of the Secretary
of Defense (Program Analysis and Evaluation) and the (then) Navy Center
for Cost Analysis, both of whom found that the EAs supported the
milestone decision being requested.
The OOMA 2004 EA was conducted in March 2004 and concurred upon by the
Navy Cost Analysis Division (NCAD) in support of the upcoming Fielding
Decision for the Optimized Organizational Maintenance Activity (OOMA)
application. This OOMA EA is being updated in collaboration with NCAD
to reflect Program plans established pursuant to upcoming OOMA version
831-01.05.00 Follow-on Operational Testing and Fielding Decision.
The NTCSS EAs did not address the Navy's Enterprise Resource Planning
(ERP) Program, and the Navy ERP Program did not exist when the original
NTCSS AoA was conducted.
In addition to EAs supporting milestone decision points, NTCSS has
historically used Business Case Analyses (BCA) to evaluate investment
decisions, including two conducted in 2001 and one in 2004. The 2004
Fielding Alternatives BCA was conducted by the Program Office at the
request of Chief of Naval Operations for Material Readiness and
Logistics Operations (OPNAV N4) in order to assess POM06 funding
decisions and plan a way-ahead for modernizing naval afloat logistics
systems. The 2004 Fielding Alternatives BCA was conducted largely
consistent with the procedures outlined in OMB Circulars A-94 and A-11
and the Federal Chief Information Officers Council, Best Practices
Committee report, "Value Measuring Methodology (VMM), How to Guide."
RECOMMENDATION 2: The GAO recommended that the Secretary of Defense
direct the Secretary of the Navy to determine if continued investment
in NTCSS as planned represents a prudent use of the department's
limited resources. To accomplish this, the Secretary of the Navy should
direct the program office to ensure that development of this economic
analysis (1) complies with cost estimating best practices, including
recognition of costs to resolve open trouble reports and change
proposals, and relevant OMB cost benefit guidance and (2) incorporates
available data on whether deployed NTCSS capabilities are actually
producing benefits. (p. 60/GAO Draft Report):
DoD RESPONSE: Concur.
As stated above, the 2004 OOMA EA is being updated in collaboration
with NCAD to reflect program plans established pursuant to upcoming
Follow-on Operational Testing of OOMA version 831-01.05.00 in support
of the OOMA Fielding Decision. As was the case with the 1997 and 1999
NTCSS Optimized EAs, this OOMA EA is being compiled in compliance with
cost and benefit guidance and best practices of the DoD 5000 series,
OMB Circular A-94 and OMB Circular A-11 as recommended in the GAO
report.
Software Trouble Reports and Change Proposals are prioritized by the
NTCSS Requirements Integrated Product Team in accordance with Program
Office policy and available funding, and, if approved for
implementation, will be slated for subsequent maintenance updates to
the application baseline.
In accordance with DoDI 5000.2 and current DoD CIO guidance on Clinger-
Cohen Act (CCA) Compliance Certification, and to demonstrate the
benefits realized by NTCSS, outcome measures of effectiveness (MOEs)
and a post implementation review (PIR) plan will be developed for the
deployed NTCSS capabilities and for the OOMA Fielding Decision. The
OOMA PIR will be conducted 6 to 12 months after the OOMA Fielding
Decision. Results of these reviews will be reported to the appropriate
Navy and DoD stakeholders.
RECOMMENDATION 3: The GAO recommended that the Secretary of Defense
direct the Secretary of the Navy to determine if continued investment
in NTCSS as planned represents a prudent use of the department's
limited resources. To accomplish this, the Secretary of the Navy should
direct the program office to collaborate with the Under Secretary of
Defense for Acquisition, Technology, and Logistics and the Under
Secretary of Defense (Comptroller) to ensure that NTCSS is adequately
aligned with evolving DoD and Navy enterprise Architecture. (p. 60/GAO
Draft Report):
DoD RESPONSE: Concur.
DoD concurs that the NTCSS program should be adequately aligned with
the evolving DoD and Navy Enterprise Architecture. The NTCSS program
was reviewed in FY05 and FY06 as part of the Secretary of Defense
Business Management Modernization Program (BMMP). As part of these
annual reviews, the program was evaluated against and found to be
aligned with the evolving DoD and Navy Enterprise Architecture.
Ultimately, these reviews led to Under Secretary of Defense
(Comptroller) certification and (in FY06) Defense Business Systems
Management Committee (DBSMC) approval for system modernization
investment. On a continuing basis, the NTCSS Program Office works to
ensure that it is properly aligned to the currently defined DoD and
Navy business architecture.
As part of the BMMP review in FY05, the Program Office worked in
conjunction with OPNAV N4 and the Office of the Deputy Undersecretary
of Defense Logistics and Materiel Readiness within the Office of the
Under Secretary of Defense for Acquisition, Technology, and Logistics
(USD(AT&L)). In FY06, the Program Office worked closely with the Navy
Investment Review Board (IRB) that was established as part of the BMW
process. To ensure architecture alignment, the NTCSS program will
collaborate with the new Business Transformation Agency (BTA). The BTA
reports to the DBSMC through the USD(AT&L) and manages the DoD IRB
process.
RECOMMENDATION 4: The GAO recommended that the Secretary of Defense
direct the Secretary of the Navy to present the result of the analysis
outlined in Recommendation 1 to the Deputy Secretary of Defense, or his
designee, and seek a departmental decision on how to best proceed with
the program. Until this is done, the GAO recommended that the Secretary
of Defense direct the Secretary of the Navy to halt further deployment
of NTCSS and to limit future investment in already deployed
applications to essential operations and maintenance activities and
only developmental activities deemed essential to national security
needs. (p. 61/GAO Draft Report):
DoD RESPONSE: Partially concur.
We do not agree that further deployment of NTCSS should be limited at
this time. However, we will direct the Navy to work with OSD(PA&E) to
conduct the analysis described in our response to Recommendation 1
above, and the Navy will present the results of their analysis to the
NII OIPT within 120 days. Appropriate direction will be given to the
program, based on the results of that analysis.
RECOMMENDATION 5: The GAO recommended that if based on reliable data, a
decision is made to continue the NTCSS program, the Secretary of
Defense direct the Secretary of the Navy to ensure that the NTCSS
program implements effective program management activities, including
earned value management, requirements development and management, and
test management. (p. 61/GAO Draft Report):
DoD RESPONSE: Partially Concur:
Agree that the NTCSS program should implement effective program
management activities but do not agree that the Secretary of Defense
needs to give direction to the Secretary of the Navy regarding this
issue.
The Navy's position is as follows:
The NTCSS Program has implemented and is continuously improving program
management activities, including earned value management, requirements
development and management, and test management in accordance with
applicable directives and industry best practices.
The Program Office has instituted a tailored approach to formal
reporting and extends performance measurement principles to the Central
Design Activity (CDA) even though the projects fall below the dollar
threshold requiring use of an EVMS and despite the absence of formal
contract deliverables. Projects selected for earned value application
are based on risk level, resource limitations, and the level of
management oversight required. The value of the individual projects
contained in NTCSS did not warrant the CDA having a certified EVMS but
adherence to the 32 criteria set forth in ANSI/EIA standards are
actively applied. The CDA has set internal definitions on which
project(s) warrant earned value reporting and these are specified in
the CDA Software Measurement Plan.
* To further ensure that the program is using the best program
management practices, the Navy Acquisition Executive will ask the Navy
Inspector General to conduct a study to see where improvement could be
made in NTCSS program management activities.
RECOMMENDATION 6: The GAO recommended that if based on reliable data, a
decision is made to continue the NTCSS program, the Secretary of
Defense direct the Secretary of the Navy to [ensure that] key
stakeholders, such as the central design agency and the developmental
testing organization, have the people, processes, and tools to
effectively execute their respective roles and responsibilities. (p.
61/GAO Draft Report):
DoD RESPONSE: Partially Concur.
We concur with the GAO that all acquisition programs should have the
necessary people, processes and tools to effectively execute their
respective roles and responsibilities. It is the Navy's position that
key stakeholders of the NTCSS program do, in fact, have the people,
processes and tools to effectively execute their respective roles and
responsibilities. For example, SPAWARSYSCEN Norfolk, the Central Design
Activity, demonstrated their competency and capability and was
certified CMM-SW Maturity Level 3.
A review of this area will be part of the NII OIPT meeting discussed
under Recommendation 1. A determination of whether further action is
needed will be made by the OIPT at that time.
RECOMMENDATION 7: The GAO recommended that the Secretary of Defense
reestablish the Office of the Assistant Secretary of Defense for
Networks and Information Integration/Chief Information Officer as the
milestone decision authority, and direct the Secretary of the Navy to
take steps to ensure that Navy oversight entities fulfill their roles
and responsibilities on NTCSS, including ensuring that reliable program
reporting occurs and is acted upon. (p. 62/GAO Draft Report):
DoD RESPONSE: Partially Concur:
We believe it is premature to reestablish the ASD(NII)/DoD CIO as the
milestone decision authority for NTCSS, given the fact that over 95% of
the development work is complete. However, we will revisit that
decision based on the results of the analysis discussed in our replies
to Recommendations 1, 4 and 6 above.
The NII OIPT Leader and the Navy Acquisition Executive will ensure that
Navy oversight entities fulfill their roles and responsibilities on
NTCSS, including ensuring that reliable program reporting occurs and is
acted upon. Currently, NTCSS is overseen by:
* PEO (C4I and Space);
* SPAWAR Comptroller and NCAD;
* Domain Functional Managers (FAM process);
* OPNAV N4;
* NTCSS Executive Steering Committee;
* DASN (C4I and Space);
* NII OIPT Leader (through the review of quarterly Defense Acquisition
Executive Summaries):
NTCSS will continue to comply with all statutory and regulatory
reporting requirements, including:
* Navy CIO and DoD CIO Clinger-Cohen Act certification for the OOMA
full-rate production decision;
* All reports and documents required for each Milestone Decision.
The ASN (RDA) DASHBOARD:
The PEO (C4I and Space) Acquisition Management Office Database:
The following are GAO's comments on the Department of Defense's letter
dated November 23, 2005.
GAO Comments:
1. See the Agency Comments and Our Evaluation section of this report.
2. We disagree. Our report contains numerous instances where the Navy
did not comply with either DOD acquisition policies and guidelines or
industry best practices, in the areas of (1) economic justification;
(2) architectural alignment; (3) project management, including progress
measurement and reporting, funding disclosure, and oversight
activities; and (4) system development, including requirements
management and testing. Moreover, the Navy has not provided any
evidence to demonstrate that our report is incorrect with respect to
the level of program discipline and conformance with applicable policy
and guidance in the areas that we reviewed.
3. We disagree. Knowing that NTCSS is the right solution to meet the
Navy's strategic business and technological needs would require that a
frame of reference articulating these needs be available as a point of
comparison. Such a frame of reference is an enterprise architecture.
However, the Navy stated the system was defined and implemented without
a complete and formal enterprise architecture. Our experience with
federal agencies has shown that investing in an information technology
solution without defining the solution in the context of an
architecture often results in systems that are duplicative, not well
integrated, and unnecessarily costly to maintain and interface. In
addition, in February 2005, key program stakeholders and
representatives from user organizations questioned whether NTCSS as
defined was the right solution to cost effectively meet users' needs.
At that time, program officials stated their intent to develop a new
economic analysis to gather the information needed to determine whether
to continue investing in NTCSS. In November 2005, program officials
told us that they no longer planned to develop this economic analysis.
Without a well-defined architecture and a reliable economic analysis,
the Navy cannot be sure that NTCSS is the right solution.
4. See comment 2.
5. We acknowledge DOD's comment but would note that it is contrary to
statements made to us during the audit. For example, officials with the
milestone decision authority stated that, due to staffing reductions,
the office was unable to fully perform oversight activities and has had
to delegate completion of these activities. Also, Naval Cost Analysis
Division officials stated that they only review cost estimates that are
prepared for milestone reviews because staffing limitations do not
permit them to review all cost estimates. Further, Navy officials
stated that the central design agency was unable to effectively execute
testing activities because it did not have a development testing lab.
6. We disagree with this approach because its scope is narrower than
our recommendation. Specifically, we recommended that the Navy develop
a reliable economic analysis of the NTCSS program that includes all
viable alternatives, including the Navy's Enterprise Resource Planning
program. DOD acquisition policy and guidance provide detailed
instructions on how economic analyses should be performed to obtain
information that is critical for decisions regarding investments of
scarce resources. Without such information, Navy risks that its
continued investment in the system may not be justified.
7. We disagree. With respect to the statement that NTCSS is a "very
mature program," NTCSS has been under development for about 10 years at
a cost of about $1.1 billion, and the Navy plans to spend an additional
$348 million between fiscal years 2006 and 2009. Further, as appendix
II of our report shows, there are hundreds of open trouble reports and
change proposals that need to be addressed before the system can
deliver promised or expected capabilities. In addition, should the OOMA
application pass operational testing and be fielded, there are over 200
sites where the necessary hardware must be installed and related
training must occur. These two efforts will require a significant
investment of time and resources, and it is therefore critical that the
Navy ensure that NTCSS is the proper system before investing additional
funds. With respect to the statement that "the final application is
about to fielded," there is no evidence to support this. Since its
originally planned fielding date of 2001, OOMA has failed operational
testing twice, and the application is still under development.
Therefore, it is premature to assert that the application will soon
pass developmental and follow-on operational testing.
8. See comment 6. Further, we disagree with the proposal to limit key
stakeholders' involvement in developing the economic justification to
"coordinating" and "briefing." These stakeholders have specific
expertise and roles relative to economically justifying system
investments that should be exploited. Until it conducts a complete and
disciplined analysis of the entire NTCSS program (reviewed and approved
by the Office of Program Analysis and Evaluation and the Naval Cost
Analysis Division) and provides this analysis to all key stakeholders,
the Navy's investment decisions will continue to be made without
complete and reliable data.
9. We disagree. As discussed in our report, the 2004 economic analysis
did not adhere to five of eight criteria elements contained in the
Office of Management and Budget Circulars A-94 and A-11.
10. We disagree. The 2004 economic analysis that the Navy provided us
focused on three fielding alternatives for the NTCSS program, not just
the OOMA application. The Navy did not provide a 2004 economic analysis
for just OOMA as the final NTCSS application.
11. We disagree. As stated in our report, officials from the Office of
Program Analysis and Evaluation and the Naval Cost Analysis Division
told us that they did not review the 2004 NTCSS economic analysis.
12. See comment 10.
13. We agree that the Navy ERP program did not exist when the original
NTCSS analysis of alternatives was conducted. However, the Navy ERP
program was initiated in 1998 and therefore did exist when the Navy
conducted subsequent analysis of alternatives.
14. See comment 9.
15. We do not question whether these annual reviews occurred and what
resulted from them. However, the point in our report is that NTCSS has
not been defined and developed in the context of a DOD or Navy
enterprise architecture because a well-defined version of either has
not existed to guide and constrain the program. As a result, meaningful
analysis showing how NTCSS aligns to evolving DOD and Navy architecture
efforts could not be produced. This means that the Navy does not have a
sufficient basis for knowing if NTCSS, as defined, properly fits within
the context of future DOD and Navy business operational and
technological environments.
16. We disagree. Our recommendation to limit further deployment of
NTCSS is a way of ensuring that the Navy takes a "strategic pause"
while it takes the time to ensure that decisions regarding future
investment are made using reliable information, which our report shows
has not historically been the case. As long as the Navy is not
appropriately limiting work on NTCSS, it is continuing to invest
resources without having justified doing so.
17. See comment 6.
18. See comment 2.
19. We disagree. As we state in our report, neither the decomposition
of the program into small, fiscal year-based projects nor the absence
of a contractual relationship is a valid reason for not effectively
implementing earned value management. Without reliable, timely, and
auditable earned value management data, the program office cannot
adequately manage technical, cost, and schedule risks and problems.
20. We disagree. The Navy's own self-assessment of compliance with the
32 criteria, detailed in appendix III of our report, showed that 17 of
these criteria were not being satisfied. Further, our assessment showed
that the Navy did not satisfy 29 of the 32 criteria, and program
officials did not provide any evidence to refute the results of our
assessment.
21. The Navy did not provide us with a copy of the CDA Software
Measurement Plan.
22. See comment 5. Further, the Navy's position that "key stakeholders
of the NTCSS program do, in fact, have the people, processes and tools
to effectively execute their respective roles and responsibilities," is
not consistent with its comment that this area will be part of a
planned review.
23. We disagree. Although the Navy states that the program is 95
percent complete, it still plans to spend $348 million over the next
three fiscal years, which is approximately 32 percent of what has been
spent on the program to date. In addition, because the Navy lacks
disciplined acquisition management practices, as discussed in our
report, including earned value management, we question how it is able
to reliably determine what percentage of the work has been completed
and the percentage that remains to be done. As stated in our report,
the current milestone decision authority has allowed the program to
proceed while a major application repeatedly failed operational
testing, and another application was cancelled. In addition, the Navy
stated its intent to revisit the need to change milestone decision
authority.
[End of section]
Appendix V: GAO Contact and Staff Acknowledgments:
GAO Contact:
Randolph C. Hite (202) 512-3439 or [Hyperlink, hiter@gao.gov].
Staff Acknowledgments:
In addition to the contact named above, Cynthia Jackson, Assistant
Director; Harold J. Brumm; Calvin L. H. Chang; Jennifer K. Echard;
Joanne Fiorino; Neelaxi Lakhmani; Freda Paintsil; Jamelyn Payan; Karen
Richey; Dr. Karl Seifert; Andrea Smith; and Dr. Rona B. Stillman made
key contributions to this report.
(310287):
FOOTNOTES
[1] DOD, Department of Defense Directive Number 5000.1, The Defense
Acquisition System (May 12, 2003); Department of Defense Instruction
Number 5000.2, Operation of the Defense Acquisition System (May 12,
2003); Interim Defense Acquisition Guidebook (Oct. 30, 2002).
[2] GAO, DOD Business Systems Modernization: Long-standing Weaknesses
in Enterprise Architecture Development Need to Be Addressed, GAO-05-702
(Washington, D.C. July 22, 2005).
[3] We did not independently validate these data.
[4] GAO, High-Risk Series: An Update, GAO-05-207 (Washington, D.C.
January 2005).
[5] GAO, ADP Procurement: Prompt Navy Action Can Reduce Risks to SNAP
III Implementation, GAO/IMTEC-92-69 (Washington, D.C. Sept. 29, 1992).
[6] Force-level ships include large ships, such as aircraft carriers
and submarine tenders. Unit-level ships include command ships, hospital
ships, other auxiliary and support ships, and submarines. Aviation
squadrons are groups of planes that are always based on a specific
aircraft carrier. Naval air stations and Marine aviation logistics
squadrons are groups of planes that are land-based. The Naval air
stations support land-based planes that are not deployed to ships. The
Marine aviation logistics squadrons can be deployed on an aircraft
carrier for a specific mission and, when the mission is completed,
these planes return to their land base.
[7] According to program officials, in addition to development and
support of NTCSS Optimized applications, this amount includes legacy
application support, shore-based legacy application procurements and
installations, and Space and Naval Warfare Systems Command civilian
salaries.
[8] DOD, Blueprint for Establishing Risk-based Governance of IT
Investments in a Net-centric Department of Defense (Apr. 13, 2005).
[9] Net-centricity is a robust, globally interconnected network
environment (including infrastructure, systems, processes, and people)
in which data is shared in real time and seamlessly among users,
applications and platforms. Net-centricity enables transformation by
allowing applications to share data and services more effectively and
flexibly, thereby allowing more agile, effective business practices to
be used at reduced cost.
[10] GAO, Information Technology: DOD's Acquisition Policies and
Guidance Need to Incorporate Additional Best Practices and Controls,
GAO-04-722 (Washington, D.C. July 30, 2004).
[11] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004).
[12] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004).
[13] Carnegie Mellon University's SEI is a government-funded research
organization that is widely considered an authority on software
implementation. The checklist used is CMU/SEI-95-SR-004, A Manager's
Checklist for Validating Software Cost and Schedule Estimates, January
1995. SEI developed these checklists to help evaluate software costs
and schedule. However, SEI states that these checklists are equally
applicable to hardware and systems engineering projects.
[14] Office of Management and Budget, Circular No. A-94: Guidelines and
Discount Rates for Benefit-Cost Analysis of Federal Programs (Oct. 29,
1992); and Circular No. A-11: Planning, Budgeting, Acquisition and
Management of Capital Assets (June 21, 2005).
[15] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004).
[16] Clinger-Cohen Act of 1996, 40 U.S.C. sections 11101-11704, and
Office of Management and Budget (OMB) Circular No. A-130, Management of
Federal Information Resources (Nov. 30, 2000).
[17] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004).
[18] The Office of the Chief of Naval Operations for Material Readiness
and Logistics Operations serves as the program and resource sponsor for
the NTCSS program in order to balance user requirements with available
resources. See table 7: NTCSS Management and Stakeholder Roles and
Responsibilities.
[19] ERP is an automated system using commercial off-the-shelf software
consisting of multiple, integrated functional modules that perform a
variety of business-related tasks such as payroll, general ledger
accounting, and supply chain management. In August 2002, the Assistant
Secretary of the Navy for Research, Development, and Acquisition
established a Navy-wide ERP program to converge four ERP pilot programs
that had been ongoing since 1998.
[20] DOD, Department of Defense Directive Number 5000.1, The Defense
Acquisition System (May 12, 2003) and Department of Defense
Architecture Framework, Version 1.0, Volume 1 (February 2004).
[21] See, for example, Clinger-Cohen Act of 1996, 40 U.S.C. §§ 11312
and 11315(b)(2); E-Government Act of 2002, Pub. L. No. 107-347 (Dec.
17, 2002); GAO, Information Technology: A Framework for Assessing and
Improving Enterprise Architecture Management (Version 1.1), GAO-03-584G
(Washington, D.C. April 2003); Chief Information Officer Council, A
Practical Guide to Federal Enterprise Architecture, Version 1.0
(February 2001); and Institute of Electrical and Electronics Engineers,
Standard for Recommended Practice for Architectural Description of
Software-Intensive Systems 1471-2000 (Sept. 21, 2000).
[22] See, for example, GAO, Homeland Security: Efforts Under Way to
Develop Enterprise Architecture, but Much Work Remains, GAO-04-777
(Washington, D.C. Aug. 6, 2004); DOD Business Systems Modernization:
Limited Progress in Development of Business Enterprise Architecture and
Oversight of Information Technology Investments, GAO-04-731R
(Washington, D.C. May 17, 2004); Information Technology: Architecture
Needed to Guide NASA's Financial Management Modernization, GAO-04-43
(Washington, D.C. Nov. 21, 2003); DOD Business Systems Modernization:
Important Progress Made to Develop Business Enterprise Architecture,
but Much Work Remains, GAO-03-1018 (Washington, D.C. Sept. 19, 2003);
Business Systems Modernization: Summary of GAO's Assessment of the
Department of Defense's Initial Business Enterprise Architecture, GAO-
03-877R (Washington, D.C. July 7, 2003); Information Technology: DLA
Should Strengthen Business Systems Modernization Architecture and
Investment Activities, GAO-01-631 (Washington, D.C. June 29, 2001); and
Information Technology: INS Needs to Better Manage the Development of
Its Enterprise Architecture, GAO/AIMD-00-212 (Washington, D.C. Aug. 1,
2000).
[23] GAO, Information Technology: Architecture Needed to Guide
Modernization of DOD's Financial Operations, GAO-01-525 (Washington,
D.C. May 17, 2001).
[24] GAO, DOD Business Systems Modernization: Improvements to
Enterprise Architecture Development and Implementation Efforts Needed,
GAO-03-458 (Washington, D.C. Feb. 28, 2003); Information Technology:
Observations on Department of Defense's Draft Enterprise Architecture,
GAO-03-571R (Washington, D.C. Mar. 28, 2003); GAO-03-877R; GAO-03-1018;
GAO-04-731R.
[25] GAO-05-702.
[26] GAO, DOD Business Systems Modernization: Billions Being Invested
without Adequate Oversight, GAO-05-381 (Washington, D.C. Apr. 29,
2005).
[27] The Under Secretary of Defense for Acquisition, Technology, and
Logistics and the Under Secretary of Defense (Comptroller) are
responsible for overseeing the development of DOD's business enterprise
architecture.
[28] Ronald W. Reagan National Defense Authorization Act for Fiscal
Year 2005, Pub. L. No. 108-375, § 332, 118 Stat. 1811, 1851-1856, (Oct.
28, 2004) (codified in part at 10 U.S.C. § 2222).
[29] GAO, Information Technology: Enterprise Architecture Use across
the Federal Government Can Be Improved, GAO-02-6 (Washington, D.C. Feb.
19, 2002); and Information Technology: Leadership Remains Key to
Agencies Making Progress on Enterprise Architecture Efforts, GAO-04-40
(Washington, D.C. Nov. 17, 2003).
[30] GAO, DOD Business Systems Modernization: Navy ERP Adherence to
Best Practices Critical to Avoid Past Failures, GAO-05-858 (Washington,
D.C. Sept. 29, 2005).
[31] GAO-05-858.
[32] DOD, Department of Defense Instruction Number 5000.2, Operation of
the Defense Acquisition System (May 12, 2003) and Defense Acquisition
Guidebook, Version 1.0 (Oct. 17, 2004).
[33] American National Standards Institute (ANSI)/Electronic Industries
Alliance (EIA) EVM System Standard (ANSI/EIA-748-98), Chapter 2 (May
19, 1998).
[34] GAO, Missile Defense: Additional Knowledge Needed in Developing
System for Intercepting Long-Range Missiles, GAO-03-600 (Washington,
D.C. Aug. 21, 2003).
[35] Indirect costs are also known as "burden" or overhead costs. All
organizations have indirect costs, which may include, for example, the
cost of an office building, its depreciation, fringe benefits, office
furniture, supplies, computers, vacations, sick pay, and telephone
costs. By omitting indirect costs, NTCSS is understating the true
program costs.
[36] Before April 2005, DOD policy required the use of EVM and the use
of integrated baseline reviews for programs with (1) contracts or
agreements for research and development or test and evaluations over
$73 million or (2) procurement or operations and maintenance contracts
over $315 million (both in fiscal year 2000 constant dollars). Since
April 2005, DOD now requires the use of EVM for all cost or incentive
contracts over $20 million.
[37] DOD, Department of Defense Directive Number 5000.1, The Defense
Acquisition System (Oct. 23, 2000) (current version dated May 12,
2003); DOD Instruction Number 5000.2, Operation of the Defense
Acquisition System (Apr. 5, 2002) (current version dated May 12, 2003);
and DOD 5000.2-R, Mandatory Procedures for Major Defense Acquisition
Programs (MDAPS) and Major Automated Information System (MAIS)
Acquisition Programs (Apr. 5, 2002) (canceled, replaced by DOD Defense
Acquisition Guidebook [Oct. 17, 2004]).
[38] A program breach occurs when the program office has reason to
believe that a cost, schedule or performance goal, as documented in an
Acquisition Program Baseline, will not be reached.
[39] NTCSS participated in the formal RIT pilot program between October
2002 and December 2003, when the pilot ended. However the program
office, with agreement from the milestone decision authority, continued
to use the RIT pilot procedures until December 2004.
[40] DOD Financial Management Regulation 7000.14-R, (FMR) Vol. 2A,
Chap. 1, section 010213 (June 2004).
[41] Navy Financial Management Policy Manual, NAVSO P-1000, section
075371.2.a (December 2002).
[42] In some circumstances, software modernization costs under $250,000
may be considered "expenses," and funded with "Operation and
Maintenance" appropriations. (DOD Financial Management Regulation
7000.14-R, (FMR) Vol. 2A, Chap. 1, section 010212 [June 2004]). The
threshold in the current Navy guidance is $100,000. (Navy Financial
Management Policy Manual, NAVSO P-1000, section 075371. [December
2002]).
[43] DOD, Department of Defense Directive Number 5000.1, The Defense
Acquisition System (May 12, 2003).
[44] There have been three milestone decision authorities for NTCSS
since the program was begun. Initially, the milestone decision
authority was in the Office of the Assistant Secretary of Defense for
Networks and Information Integration/Chief Information Officer. In July
1999, this authority was delegated to the Assistant Secretary of the
Navy for Research, Development and Acquisition, who then delegated
oversight authority to Deputy Assistant Secretary of the Navy for
Command, Control, Communication, Computers and Intelligence, and Space
in March 2000.
[45] Capital Investment Reports, also known as Exhibit 300s, are
prepared annually by DOD for each major IT initiative and submitted to
OMB.
[46] GAO, Customs Service Modernization: Serious Management and
Technical Weaknesses Must Be Corrected, GAO/AIMD-99-41 (Washington,
D.C. Feb. 26, 1999).
[47] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004).
Software Engineering Institute, Software Acquisition Capability
Maturity Model® version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA:
March 2002).
[48] Defense Acquisition University, Test and Evaluation Management
Guide, Fourth Edition (November 2001).
[49] Software Engineering Institute, Issues in Requirements
Elicitation, CMU/SEI-92-TR-12 (Pittsburgh, PA: September 2002).
[50] Software Engineering Institute, Software Acquisition Capability
Maturity Model® version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA:
March 2002); and Defense Acquisition University, Test and Evaluation
Management Guide, Fourth Edition (November 2001).
[51] Naval Audit Service, Audit Report Reliability and Validity of the
Optimized Naval Aviation Logistics Command Management Information
System, July 22, 2003. NAVAUDSVC P-7520.1, N2003-0060.
[52] Naval Aviation Logistics Command Management Information System
(NALCOMIS) Optimization for Organizational Maintenance Activities
(OOMA) Follow-on Operational Test and Evaluation OT-IIIA Report to the
Chief of Naval Operations, May 7, 2004, Commander, Operational Test and
Evaluation.
[53] Software Engineering Institute, Software Acquisition Capability
Maturity Model® version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA:
March 2002); Defense Acquisition University, Test and Evaluation
Management Guide, Fourth Edition (November 2001); and DOD Instruction
Number 5000.2, Operation of the Defense Acquisition System (Apr. 5,
2002) (current version dated May 12, 2003).
[54] CMM®, Capability Maturity Model, and Capability Maturity Modeling
are registered in the U.S. Patent and Trademark Office.
[55] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004).
Office of Management and Budget, Circular No. A-94: Guidelines and
Discount Rates for Benefit-Cost Analysis of Federal Programs, October
29, 1992; and Circular No. A-11: Planning, Budgeting, Acquisition and
Management of Capital Assets, June 21, 2005. Software Engineering
Institute, A Manager's Checklist for Validating Software Cost and
Schedule Estimates, CMU/SEI-95-SR-004 (Pittsburgh, PA. January 1995).
[56] GAO-05-702; GAO-02-6; and GAO-04-40.
[57] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004);
and American National Standards Institute (ANSI)/Electronic Industries
Alliance (EIA) EVM System Standard (ANSI/EIA-748-98), Chapter 2 (May
19, 1998).
[58] Software Engineering Institute, Software Acquisition Capability
Maturity Model® Version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA:
March 2002).
[59] Software Engineering Institute, Software Acquisition Capability
Maturity Model® Version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA:
March 2002). Defense Acquisition University, Test and Evaluation
Management Guide, Fourth Edition (November 2001).
[60] Naval Audit Service, Audit Report Reliability and Validity of the
Optimized Naval Aviation Logistics Command Management Information
System, July 22, 2003. NAVAUDSVC P-7520.1, N2003-0060.
[61] Naval Aviation Logistics Command Management Information System
(NALCOMIS) Optimization for Organizational Maintenance Activities
(OOMA) Follow-on Operational Test and Evaluation OT-IIIA Report to the
Chief of Naval Operations, May 7, 2004, Commander, Operational Test and
Evaluation.
[62] American National Standards Institute (ANSI)/Electronic Industries
Alliance (EIA) EVM System Standard (ANSI/EIA-748-98), Chapter 2 (May
19, 1998).
[63] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004);
and American National Standards Institute (ANSI)/Electronic Industries
Alliance (EIA) EVM System Standard (ANSI/EIA-748-98), Chapter 2 (May
19, 1998).
[64] "Yes" means that the program provided documentation demonstrating
satisfaction of the criterion. "Partially" means that the program
provided documentation demonstrating satisfaction of part of the
criterion. "No" means that the program has yet to provide documentation
demonstrating satisfaction of the criterion.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
