Information Technology
DOD's Acquisition Policies and Guidance Need to Incorporate Additional Best Practices and Controls Gao ID: GAO-04-722 July 30, 2004The way in which the Department of Defense (DOD) has historically acquired its business systems has been cited as a root cause for its limited success in delivering promised system capabilities and benefits on time and within budget. In response, DOD recently revised its systems acquisition policies and guidance to incorporate best practices, including those pertaining to business systems. GAO was asked to determine whether DOD's revised systems acquisition policies and guidance (1) are consistent with industry best practices, including those pertaining to commercial component-based systems, and (2) provide the necessary controls to ensure that DOD component organizations adhere to the practices.
DOD's revised policies and guidance largely incorporate 10 best practices for acquiring any type of information technology (IT) business system. For example, the revisions include the requirement that acquisitions be economically justified on the basis of costs, benefits, and risks. However, the revisions generally do not incorporate 8 best practices relating to the acquisition of commercial component-based systems. For example, they do not address basing any decision to modify commercial components on a thorough analysis of the impact of doing so or on preparing system users for the business process and job roles and responsibilities changes that are embedded in the functionality of commercial IT products. In total, GAO found that DOD's acquisition policies and guidance fully incorporate 8 of the 18 best practices that they were evaluated against, partially incorporate 5 practices, and do not incorporate the remaining 5 practices. DOD intends to expand its acquisition guidance to incorporate additional best practices by September 30, 2004, but department officials cite other priorities as a reason why they have not been able to complete this effort and could not provide a plan specifying how this will be accomplished. Until DOD's revised policies and guidance incorporate key systems acquisition best practices, the risk that system investments will not consistently deliver promised capabilities and benefits on time and within budget is increased. DOD's revised policies also do not contain sufficient controls to ensure that DOD components appropriately follow the best practices that are incorporated in its policies and guidance. According to acquisition best practices experts, as well as GAO's internal control guidance, controls are effective if they are backed by measurements that are verified. Although the revised policies and guidance require acquisition managers to examine and, as appropriate, adopt best practices, they do not cite what that examination entails. DOD believes existing controls are sufficient, even though these controls do not provide for measuring and validating the practices' use. Without specific requirements to measure and validate the use of best practices, the risk that they will not be followed increases, which, in turn, increases the risk that system investments will not meet expectations.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-04-722, Information Technology: DOD's Acquisition Policies and Guidance Need to Incorporate Additional Best Practices and Controls
This is the accessible text file for GAO report number GAO-04-722
entitled 'Information Technology: DOD's Acquisition Policies and
Guidance Need to Incorporate Additional Best Practices and Controls'
which was released on July 30, 2004.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
July 2004:
INFORMATION TECHNOLOGY:
DOD's Acquisition Policies and Guidance Need to Incorporate Additional
Best Practices and Controls:
GAO-04-722:
GAO Highlights:
Highlights of GAO-04-722, a report to congressional requesters:
Why GAO Did This Study:
The way in which the Department of Defense (DOD) has historically
acquired its business systems has been cited as a root cause for its
limited success in delivering promised system capabilities and benefits
on time and within budget. In response, DOD recently revised its
systems acquisition policies and guidance to incorporate best
practices, including those pertaining to business systems.
GAO was asked to determine whether DOD‘s revised systems acquisition
policies and guidance (1) are consistent with industry best practices,
including those pertaining to commercial component-based systems, and
(2) provide the necessary controls to ensure that DOD component
organizations adhere to the practices.
What GAO Found:
DOD‘s revised policies and guidance largely incorporate 10 best
practices for acquiring any type of information technology (IT)
business system. For example, the revisions include the requirement
that acquisitions be economically justified on the basis of costs,
benefits, and risks. However, the revisions generally do not
incorporate 8 best practices relating to the acquisition of commercial
component-based systems. For example, they do not address basing any
decision to modify commercial components on a thorough analysis of the
impact of doing so or on preparing system users for the business
process and job roles and responsibilities changes that are embedded in
the functionality of commercial IT products. In total, GAO found that
DOD‘s acquisition policies and guidance fully incorporate 8 of the 18
best practices that they were evaluated against, partially incorporate
5 practices, and do not incorporate the remaining 5 practices
(see figure). DOD intends to expand its acquisition guidance to
incorporate additional best practices by September 30, 2004, but
department officials cite other priorities as a reason why they have
not been able to complete this effort and could not provide a plan
specifying how this will be accomplished. Until DOD‘s revised policies
and guidance incorporate key systems acquisition best practices, the
risk that system investments will not consistently deliver promised
capabilities and benefits on time and within budget is increased.
DOD‘s revised policies also do not contain sufficient controls to
ensure that DOD components appropriately follow the best practices that
are incorporated in its policies and guidance. According to acquisition
best practices experts, as well as GAO‘s internal control guidance,
controls are effective if they are backed by measurements that are
verified. Although the revised policies and guidance require
acquisition managers to examine and, as appropriate, adopt best
practices, they do not cite what that examination entails. DOD believes
existing controls are sufficient, even though these controls do not
provide for measuring and validating the practices‘ use. Without
specific requirements to measure and validate the use of best
practices, the risk that they will not be followed increases, which, in
turn, increases the risk that system investments will not meet
expectations.
DOD Incorporation of Best Practices:
[See PDF for image]
[End of figure]
What GAO Recommends:
To improve DOD‘s ability to acquire IT business systems, GAO recommends
that the Secretary of Defense incorporate additional best practices in
DOD‘s acquisition policies and guidance, and that the department
strengthen controls for ensuring that best practices are appropriately
followed.
In commenting on a draft of this report, DOD agreed or partially agreed
with our recommendations for incorporating additional best practices,
but did not agree that it needed (1) a plan to govern its incorporation
of these practices or (2) stronger controls for ensuring that best
practices are followed.
www.gao.gov/cgi-bin/getrpt?GAO-04-722.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Randolph C. Hite at
(202) 512-3439 or hiter@gao.gov.
Contents:
Letter:
Results in Brief:
Background:
DOD's Acquisition Policy and Guidance Are Consistent with Some, but Not
All, Key Acquisition Best Practices:
DOD's Acquisition Policies Do Not Contain Sufficient Controls to Ensure
That the Requirement Is Met for Appropriately Applying Best Practices:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendixes:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Best Practices:
Best Practices Relevant to Any IT Business Systems Acquisition:
Complementary Best Practices Relevant to Commercial Component-Based IT
Business Systems Acquisitions:
Appendix III: Comments from the Department of Defense:
GAO Comments:
Appendix IV: GAO Contact and Staff Acknowledgments:
GAO contact:
Staff acknowledgments:
Tables:
Table 1: Organizational Responsibilities for the DOD 5000 Series
Documents:
Table 2: Summary of Business Systems Acquisition Best Practices:
Table 3: Activity-by-activity Comparison of the 5000 Series to Best
Practices Relevant to Any IT Business Systems Acquisition:
Table 4: Activity-by-Activity Comparison of the 5000 Series to Best
Practices Relevant to Commercial Component-based Business Systems
Acquisitions:
Figures:
Figure 1: Revisions to the DOD 5000 Series Documents:
Figure 2: Simplified Diagram of DOD's Acquisition Management Framework :
Figure 3: DOD Acquisition Policies and Guidance Incorporation of Best
Practices:
Abbreviations:
AT&L: Acquisition, Technology, and Logistics:
DOD: Department of Defense:
IT: information technology:
NII: Networks and Information Integration:
OT&E: Operational Test and Evaluation:
Letter:
July 30, 2004:
The Honorable John Ensign:
Chairman:
The Honorable Daniel K. Akaka:
Ranking Minority Member:
Subcommittee on Readiness and Management Support:
Committee on Armed Services:
United States Senate:
This report responds to your request that we assess the Department of
Defense's (DOD) recently revised policies and guidance for acquiring
business systems for incorporation of acquisition best practices. As
you know, the way in which DOD has historically acquired information
technology (IT) systems has been cited as a root cause of these systems
failing to deliver promised capabilities and benefits on time and
within budget. The use of these best practices--which includes those
practices pertaining to any business system, whether custom developed
or based on commercially available product components, as well as those
unique to commercial component-based systems--is intended to improve on
this performance.
As agreed with your offices, our objectives were to determine whether
DOD's recently revised policies and guidance for acquiring business
systems (1) are consistent with industry best practices, including
those pertaining to commercial component-based systems, and (2) provide
the necessary controls to ensure that DOD component organizations
adhere to best practices. We conducted our work between December 2003
and May 2004 in accordance with generally accepted government auditing
standards. Details of our objectives, scope, and methodology are
included in appendix I.
Results in Brief:
DOD's revisions to its systems acquisition policies and guidance
incorporate many best practices for acquiring business systems. For
example, the revisions recognize such practices as (1) economically
justifying system investments on the basis of costs, benefits, and
risks and (2) continually measuring an acquisition's performance, cost,
and schedule against approved baselines. However, the revised policies
and guidance do not incorporate a number of other best practices,
particularly those associated with acquiring commercial component-
based business systems. For example, they do not address basing a
decision to modify a commercial component on a thorough analysis of the
impact of doing so; evaluating contractors on their ability to
implement commercial components and using the results in source
selection decisions; and preparing system users for the business
process and job roles and responsibilities changes that are embedded in
the functionality of commercial software products. According to
officials responsible for revising DOD acquisition policies and
guidance, additional best practices and lessons learned will be
incorporated into the acquisition guidance by September 30, 2004.
However, documented plans for this task do not exist, and the
associated resources needed to complete this task have not been
assigned due to higher priority needs. Until these missing best
practices are included in DOD's acquisition policies and guidance, the
risk is increased that systems acquisitions will not deliver planned
capabilities and benefits on time and within budget.
DOD's revised acquisition policies also do not contain sufficient
controls to ensure that military services and defense agencies
appropriately follow the practices. Controls are considered effective
if they are measured and verified. Although DOD's revised policies
require both the project manager and the investment decision authority
to examine the adoption of best practices, neither the policies nor the
associated guidance provide for measuring and verifying the use of the
practices. As a result, DOD is increasing the risk that best practice
adoption and use will not occur, which, in turn, increases the risk
that systems acquisitions will not deliver what is planned--on time and
within budget.
This report makes 14 recommendations to the Secretary of Defense that
are aimed at strengthening DOD's acquisition policy and guidance by
including additional business systems acquisition best practices and
controls for ensuring that best practices are followed.
In its written comments (reprinted in app. III) on a draft of this
report signed by the Principal Director, Deputy Assistant Secretary of
Defense (Command, Control, Communications, Space and Information
Technology Programs), DOD agreed with the importance and relevance of
the best practices that we cite in the report. DOD also agreed or
partially agreed with most of our recommendations, stating that it
would either incorporate those practices that we reported as missing
from the department's acquisition policies and guidance, or consider
augmenting its coverage of those practices that deserve greater
emphasis in its policies and guidance. Further, while DOD acknowledged
that incorporation of additional best practices in its acquisition
policies and guidance should be undertaken or considered, it did not
agree that it needed an explicit plan to govern its ongoing and future
policy and guidance revision activities, stating that our
recommendation to this effect was inappropriate. We do not agree with
DOD that a plan governing incorporation of the practices is not needed.
Given the importance of DOD's acquisition policies and guidance as well
as best practices, we believe that having an explicit plan that defines
how and when incorporation of best practices will be added is
essential. Among other things, a plan would highlight the resource
constraints that this revision effort has been subject to, would allow
measurement against defined milestones, and would allow disclosure of
progress and impediments. In its comments, DOD also did not agree that
stronger controls are needed for ensuring adherence to the best
practices contained in its acquisition policies and guidance. We do not
agree with the department's position on this matter because its
existing controls do not provide for either the measurement or
verification of whether the practices are employed--both recognized
elements of effective process controls.
Background:
Best practices are tried and proven methods, processes, techniques, and
activities that organizations define and use to minimize risks and
maximize chances for success. As we have previously reported, using
best practices can result in better outcomes--including cost savings;
improved service and product quality; and, ultimately, a better return
on investment. For example, two software engineering analyses of nearly
200 systems acquisitions projects indicated that teams using systems
acquisition best practices produced cost savings of at least 11 percent
over similar projects conducted by teams that did not employ the kind
of rigor and discipline embedded in these practices.[Footnote 1] In
addition, our research[Footnote 2] shows that best practices are a
significant factor in successful acquisition outcomes, including
increasing the likelihood that programs and projects will be executed
within cost and schedule estimates.
DOD, GAO, and the Congress have all advocated the use of best
practices. For example, in September 2000, DOD established a steering
group to promote the use of systems acquisition best practices and
lessons learned. Further, our 2001 report[Footnote 3] cited the
benefits of DOD adoption of best practices and provided recommendations
for establishing a mechanism for sharing best practices throughout DOD.
In the fiscal year 2003 Defense Authorization Act,[Footnote 4] the
Congress used our recommendations in directing DOD to expand its use of
best practices. Specifically, it required the Under Secretary of
Defense (Acquisition, Technology, and Logistics (AT&L)) and the
Assistant Secretary of Defense (Command, Control, Communications, and
Intelligence--now called Networks and Information Integration (NII))--
to identify and serve as a clearinghouse for information regarding
software acquisition best practices in the public and private sectors.
In response, DOD assigned AT&L responsibility for serving as that
clearinghouse. Further, the Defense Information Systems Agency created
a Web site to provide information about acquisition best practices.
DOD Relies Extensively on IT Systems to Perform a Variety of Business
Functions:
DOD is one of the largest and most complex organizations in the world.
In fiscal year 2003, DOD reported that its operations involved over $1
trillion in assets, nearly $1.6 trillion in liabilities, disbursements
of more than $416 billion, and approximately 3.3 million military and
civilian personnel. Execution of these operations spans a wide range of
defense organizations, including the military services, defense
agencies and field activities, and various combatant and joint
operation commands. To execute these military operations, the
department performs an assortment of business functions, including
logistics management, procurement, healthcare management, and
financial management.
To support its business functions, DOD reports that it currently relies
on about 2,300 IT systems, including accounting, acquisition,
logistics, and personnel systems. Moreover, its future investment in
business systems is expected to be sizable. For fiscal year 2004, DOD
requested approximately $28 billion in IT funding to support a wide
range of military and business operations. Approximately $9 billion of
this amount is to support primarily command and control systems, and
the remaining $19 billion is to support the operation, maintenance, and
modernization of business systems.
Overview of DOD's Acquisition Management Framework:
Since the 1980s, DOD's oversight of its systems acquisitions had been
defined by a series of three documents--commonly called the 5000
series--that provided the policies and guidance for departmental
efforts to acquire service capabilities and systems:
* DOD Directive 5000.1, The Defense Acquisition System--describes the
management principles for DOD's acquisition programs.
* DOD Instruction 5000.2, Operation of the Defense Acquisition System-
-outlines the framework for managing acquisition programs.
* DOD 5000.2-R, Mandatory Procedures for Major Defense Acquisition
Programs (MDAPS) and Major Automated Information System (MAIS)
Acquisition Programs--provides the mandatory procedures for acquiring
major defense programs.
These documents have been revised several times. Most recently, in
October 2002, the Deputy Secretary of Defense determined that the
existing versions of these three documents required further revisions
to improve acquisition efficiency, flexibility, creativity, and
innovation. As a result, the Deputy Secretary canceled the existing
versions of each document and instructed the Under Secretary for AT&L;
the Assistant Secretary for NII; and the Director, Operational Test and
Evaluation (OT&E) to jointly revise the documents. (Table 1 describes
selected responsibilities of these three entities.) The revised
directive and instruction were issued in May 2003. Both were shortened
and modified to focus on required outcomes and legal requirements and
to eliminate the "how-to" details in the previous versions. In doing
so, DOD intended the revisions to provide program managers with more
flexibility in executing their respective programs.
Table 1: Organizational Responsibilities for the DOD 5000 Series
Documents:
Organization: Office of the Under Secretary of Defense (AT&L)/Defense
Acquisition Executive;
Responsibility: Advises the Secretary of Defense on all matters
pertaining to DOD's acquisition framework as well as research and
development; advanced technology; developmental test and evaluation;
production; logistics; installation management; military construction;
procurement; environmental security; and nuclear, chemical, and
biological matters.
Organization: Office of the Assistant Secretary of Defense (NII)/DOD
Chief Information Officer;
Responsibility: Advises the Secretary of Defense on achieving and
maintaining information superiority through the collection, processing,
and dissemination of an uninterrupted flow of information in support of
DOD missions while exploiting or denying an adversary's ability to do
the same; ; Serves as the principal assistant to the secretary for
electronic business, information management, information operations
and assurance, and IT.
Organization: Office of the Director (OT&E);
Responsibility: Advises the Secretary of Defense on OT&E; Issues DOD
OT&E policy and procedures, and reviews and analyzes OT&E conducted
for each major acquisition program and provides reports on adequacy
and results to the Secretary, the Under Secretary for AT&L, and the
Congress.
Source: GAO, based on DOD documentation.
[End of table]
DOD 5000.2-R was renamed the Interim Defense Acquisition Guidebook and
was made optional guidance on best practices and lessons learned. (See
fig. 1.) According to DOD officials, improvements to the guidebook are
currently under way. Until they are completed, DOD 5000.2-R serves as
the guidebook.
Figure 1: Revisions to the DOD 5000 Series Documents:
[See PDF for image]
[End of figure]
According to the revised policy, an acquisition is a directed, funded
effort that provides a new, improved, or continuing materiel, weapon or
information system, or service capability. The revised 5000 series
applies to acquisitions conducted by all of the department's
organizational components.[Footnote 5] These components include the
military services and the defense agencies, such as the Defense
Information Systems Agency.
The 5000 series describes a management framework that is intended to
translate mission needs and requirements into systems acquisition
programs. To accomplish this, the framework specifies five phases:
* Concept refinement: Intended to refine the initial system concept and
produce a strategy for acquiring a system capability. A decision is
made at the end of this phase (milestone A decision) whether to move to
technology development.
* Technology development: Intended to determine the appropriate set of
technologies to be integrated into the system by iteratively assessing
the viability of various technologies while simultaneously refining
user requirements. Once the technology has been demonstrated in a
relevant environment, a decision is made at the end of this phase
(milestone B decision) whether to move to system development and
demonstration.
* System development and demonstration: Intended to develop a system or
a system increment and demonstrate through developer testing that the
system/system increment can function in its target environment. A
decision is made at the end of this phase (milestone C decision)
whether to move to production and deployment.
* Production and deployment: Intended to achieve an operational
capability that satisfies the mission needs, as verified through
independent operational test and evaluation, and ensures that the
system is implemented at all applicable locations.
* Operations and support: Intended to provide a support program to meet
operational support requirements and sustain the system in the most
cost-effective manner over its total life cycle.
According to the framework, an acquisition program may begin at
milestone A, B, or C, and its progress depends on obtaining sufficient
knowledge to make an informed decision about whether to continue to the
next acquisition phase. Although the framework permits programs to be
managed as a single project, DOD Instruction 5000.2 states that the
department prefers an evolutionary acquisition strategy that delivers a
mature product in increments. Under such a strategy, the instruction
states that each increment is to begin with a milestone B decision, and
the production and deployment phase of each increment is to begin with
a milestone C decision. Figure 2 provides a simplified diagram of the
department's acquisition management framework.
Figure 2: Simplified Diagram of DOD's Acquisition Management Framework:
[See PDF for image]
[End of figure]
Typically, the Under Secretary for AT&L or a designee serves as the
investment decision authority[Footnote 6] for DOD acquisitions, but the
Assistant Secretary for NII/Chief Information Officer or a designee
serves as the decision authority for IT systems acquisitions.
Past Evaluations of DOD Business Systems Have Revealed Acquisition
Management Weaknesses:
Due in part to its long-standing and pervasive IT acquisition
management weaknesses, DOD has had limited success in acquiring IT
resources to replace its outdated business systems. Both inspector
general and departmental studies have cited these weaknesses on a
variety of acquisition projects. We have also reported on business
systems acquisition weaknesses.[Footnote 7] For example, in 2002 we
reported that the Defense Logistics Agency did not have effective
corporate processes for consistently acquiring software (the most
costly and complex component of systems), and that the agency did not
have a software process improvement program in place to effectively
strengthen its software acquisition processes. In 2002, we also
reported acquisition management problems with the Military Health
System's acquisition of DOD's primary medical information system,
including weaknesses in incremental economic justification, risk
management, and contract management. In 2003, we reported that DOD had
not economically justified four finance and accounting systems that
have an estimated cost of more than $1 billion. In each of our reports,
we have made recommendations for strengthening acquisition management
through the adoption of best practices. DOD has largely agreed with our
recommendations, but its progress to date in implementing them across
the department has been uneven.
DOD's Acquisition Policy and Guidance Are Consistent with Some, but Not
All, Key Acquisition Best Practices:
We and others, such as Carnegie Mellon University's Software
Engineering Institute, have identified and promoted the use of a number
of best practices associated with acquiring IT systems. For the
purposes of this report, we have identified 18 relevant best practices
and grouped them into two categories: (1) 10 best practices for
acquiring any type of business system and (2) 8 complementary best
practices that relate specifically to acquiring commercial component-
based business systems. Examples of best practices relevant to any
business systems acquisition include ensuring that (1) reasonable
planning for all parts of the acquisition occur, (2) a clear
understanding of system requirements exists, and (3) risks are
proactively identified and systematically mitigated. Examples of best
practices relevant to commercial component-based systems acquisitions
include ensuring that (1) commercial product modification is
effectively controlled, (2) relationships among commercial products are
understood before acquisition decisions are made, and (3) the
organizational impact of using new system functionality is proactively
managed. Each of these practices is composed of from one to eight
activities and is described in table 2. DOD officials responsible for
revising the 5000 series told us that each of these 18 practices are
relevant to DOD business systems acquisitions. Appendix II provides
additional details on each of these practices.
Table 2: Summary of Business Systems Acquisition Best Practices:
Best practices relevant to any business systems acquisition:
Best practices: Acquisition planning: To ensure that reasonable
planning for all parts of the acquisition is conducted;
Activity:
* Plans are prepared during acquisition planning and maintained
throughout the acquisition;
* Planning addresses the entire acquisition process, as well as life
cycle support of the products being acquired;
* The acquisition organization has a written policy for planning the
acquisition;
* Responsibility for acquisition planning activities is designated.
Best practices: Architectural alignment: To ensure that the acquisition
is consistent with the organization's enterprise architecture;
Activity:
* The system being acquired is assessed for alignment with the
enterprise architecture at key life cycle decision points, and any
deviations from the architecture are explicitly understood and
justified by an explicit waiver to the architecture;
* Product line requirements---rather than just the requirements for
the system being acquired---are an explicit consideration in each
acquisition.
Best practices: Contract tracking and oversight: To ensure that
contract activities are performed in accordance with contractual
requirements;
Activity:
* The acquiring organization has sufficient insight into the
contractor's activities to manage and control the contractor and
ensure that contract requirements are met;
* The acquiring organization and contractor maintain ongoing
communication;
commitments are agreed to and implemented by both parties;
* All contract changes are managed throughout the life of the
contract;
* The acquisition organization has a written policy for contract
tracking and oversight;
* Responsibility for contract tracking and oversight activities is
designated;
* The acquiring organization involves contracting specialists in the
execution of the contract;
* A quantitative set of software and system metrics are used to define
and measure product quality and contractor performance;
* In addition to incentives for meeting cost and schedule estimates,
measurable, metrics-based product quality incentives are explicitly
cited in the contract.
Best practices: Economic justification: To ensure that system
investments have an adequate economic justification;
Activity:
* System investment decisions are made on the basis of reliable
analyses of estimated costs, expected benefits, and anticipated risks;
* Large systems acquisitions are (to the maximum extent practical)
divided into a series of smaller, incremental acquisition efforts, and
investment decisions on these smaller efforts are made on the basis of
reliable analyses of estimated costs, expected benefits, and
anticipated risks.
Best practices: Evaluation: To ensure that evidence showing that the
contract products satisfy the defined requirements are provided prior
to accepting contractor products;
Activity:
* Evaluation requirements are developed in conjunction with the
contractual requirements and are maintained over the life of the
acquisition;
* Evaluations are planned and conducted throughout the total
acquisition period to provide an integrated approach that satisfies
evaluation requirements and takes advantage of all evaluation results;
* Evaluations provide an objective basis to support the product
acceptance decision;
* The acquiring organization has a written policy for managing the
evaluation of the acquired products;
* Responsibility for evaluation activities is designated.
Best practices: Project management: To ensure that the project office
and its supporting organizations function efficiently and effectively;
Activity:
* Project management activities are planned, organized, controlled,
and communicated;
* The performance, cost, and schedule of the acquisition are
continually measured, compared with planned objectives, and
controlled;
* Problems discovered during the acquisition are managed and
controlled;
* The acquisition organization has a written policy for project
management;
* Responsibility for project management is designated.
Best practices: Requirements development and management: To ensure
that contractual requirements are clearly defined and understood by
the acquisition stakeholders;
Activity:
* Contractual requirements are developed, managed, and maintained;
* The end user and other affected groups have input to the contractual
requirements over the life of the acquisition;
* Contractual requirements are traceable and verifiable;
* The contractual requirements baseline is established prior to
release of the solicitation package;
* The acquisition organization has a written policy for establishing
and managing the contractual requirements;
* Responsibility for requirements development and management is
designated;
* Requirements that are mandatory versus optional are clearly
delineated and used in deciding what requirements can be eliminated or
postponed to meet other project goals, such as cost and schedule
constraints.
Best practices: Risk management: To ensure that risks are proactively
identified and systematically mitigated;
Activity:
* Projectwide participation in the identification and mitigation of
risks is encouraged;
* The defined acquisition process provides for the identification,
analysis, and mitigation of risks;
* Milestone reviews include the status of identified risks;
* The acquisition organization has a written policy for managing
acquisition risk;
* Responsibility for acquisition risk management activities is
designated.
Best practices: Solicitation: To ensure that a quality solicitation is
produced and a best-qualified contractor is selected;
Activity:
* The solicitation package includes the contractual requirements and
the proposal evaluation criteria;
* The technical and management elements of proposals are evaluated to
ensure that the requirements of the contract will be satisfied;
* The selection official selects a supplier who is qualified to
satisfy the contract's requirements;
* The acquiring organization has a written policy for conducting the
solicitation;
* Responsibility for the solicitation is designated;
* A selection official has been designated to be responsible for the
selection process and decision;
* The acquiring team includes contracting specialists to support
contract administration.
Best practices: Transition to support: To ensure proper transfer of
the system from the acquiring organization to the support organization;
Activity:
* The acquiring organization ensures that the support organization has
the capacity and capability to provide the required support;
* There is no loss in continuity of support to the products during
transition from the supplier to the support organization;
* Configuration management of the products is maintained throughout
the transition;
* The acquiring organization has a written policy for transitioning
the products to the support organization;
* The acquiring organization ensures that the support organization is
involved in planning for transition to support;
* Responsibility for transition to support activities is designated.
Complementary best practices relevant to commercial component-based
business systems acquisitions:
Best practices: Component modification: To ensure that commercial
product modification is effectively controlled;
Activity:
* Modification of commercial components is discouraged and allowed
only if justified by a thorough analysis of life-cycle costs and
benefits.
Best practices: Configuration management: To ensure the integrity and
consistency of system commercial components;
Activity:
* Project plans explicitly provide for evaluation, acquisition, and
implementation of new, often frequent, product releases;
* Modification or upgrades to deployed versions of system components
are centrally controlled and unilateral user release changes are
precluded.
Best practices: Dependency analysis: To ensure that relationships
between commercial products are understood before acquisition
decisions are made;
Activity:
* Decisions about acquisition of commercial components are based on
deliberate and thorough research, analysis, and evaluation of the
components' interdependencies.
Best practices: Legacy systems integration planning: To ensure
reasonable planning for integration of commercial products with
existing systems;
Activity:
* Project plans explicitly provide for the necessary time and
resources for integrating commercial components with legacy systems.
Best practices: Organization change management: To ensure that the
organizational impact of using new system functionality is proactively
managed;
Activity:
* Project plans explicitly provide for preparing users on the impact
that the business processes embedded in the commercial components will
have on the users' respective roles and responsibilities;
* The introduction and adoption of changes to how users will be
expected to execute their jobs are actively managed.
Best practices: Solicitation: To ensure that a quality solicitation is
produced and a best-qualified contractor is selected;
Activity:
* Systems integration contractors are explicitly evaluated on their
ability to implement commercial components.
Best practices: Tradeoff analysis: To ensure that system requirements
alone do not drive the system's solution;
Activity:
* Investment decisions throughout a system's life cycle are based on
tradeoffs among the availability of commercial products (current and
future), the architectural environment in which the system is to
operate (current and future), defined system requirements, and
acquisition cost/schedule constraints.
Best practices: Vendor and product research and evaluation: To ensure
that vendor and product characteristics are understood before
acquisition decisions are made;
Activity:
* Commercial component and vendor options are researched, evaluated/
tested, and understood, both early and continuously;
* A set of evaluation criteria for selecting among commercial
component options is established that includes both defined system
requirements and vendor/commercial product characteristics (e.g.,
customer satisfaction with company and product line).
Sources: See sources listed in appendix I of this report.
[End of table]
DOD's acquisition policies and guidance largely incorporate the 10 best
practices that are relevant to any business systems
acquisition.[Footnote 8] More specifically, they fully incorporate 7 of
the 10 practices and partially incorporate the other 3 practices.
However, they generally do not incorporate the 8 best practices that
relate to acquiring commercial component-based business systems. In
particular, they fully incorporate 1 best practice, partially
incorporate 2, and do not incorporate the remaining 5. (See fig. 3 for
a summary of our analysis.) At our request, DOD officials responsible
for the 5000 series also assessed it against those 18 practices, and we
incorporated information from their assessment into ours. These
officials also told us that the acquisition guidebook is currently
being expanded to incorporate additional best practices, but they did
not provide us with a plan for accomplishing this. Until this is
accomplished, DOD is increasing the risk that important and beneficial
best practices will not be followed and that DOD business systems
investments will not deliver promised capabilities and benefits on time
and within budget.
Figure 3: DOD Acquisition Policies and Guidance Incorporation of Best
Practices:
[See PDF for image]
[End of figure]
The 5000 Series Largely Incorporates Best Practices Relevant to Any
Business Systems Acquisition:
Of the 10 best practices that we categorized as relevant to the
acquisition of any business system, whether custom-developed or
developed using commercial packages and products, essentially all have
been incorporated into DOD's acquisition policies and guidance. (See
table 3 for our detailed comparative analysis of the 5000 series
against the 10 best practices.) For example, those practices aimed at
ensuring that the acquisition is well planned, that the system is
adequately tested and evaluated against contractual requirements, and
that the requirements are clearly defined and understood by all
stakeholders are all addressed in the 5000 series. Similarly, for the 3
practices that are not fully addressed, this is the case because one
activity associated with the practice is not addressed. According to
DOD officials responsible for revising the 5000 series, the policies
contain those best practices mandated by either law or DOD regulation,
and other, optional best practices are contained in the interim
guidance represented by the former DOD 5000.2-R.
Nevertheless, the activities that are missing from the 3 practices are
important, and their absence increases the risk that the activities,
and thus the practice, will not be adequately performed. In turn, this
increases the risk that acquisition projects will fall short of
expectations. The best practice aimed at ensuring that risks are
proactively identified and systematically mitigated is a case in point.
This practice has five activities associated with it, one of which the
5000 series does not address--project reviews include the status of
identified risks. As with all the activities under this practice, this
activity plays an important role in ensuring that the appropriate level
of attention and visibility is regularly given to risk identification
and mitigation to ensure that it is effectively executed. Conversely,
if the activities are not provided for in policy and guidance, it is
unlikely that they will be performed, and it is likely that acquisition
risks will become cost, schedule, and performance problems.
Table 3: Activity-by-activity Comparison of the 5000 Series to Best
Practices Relevant to Any IT Business Systems Acquisition:
Best practice: Acquisition planning;
5000 series incorporates this best practice? Fully.
Best practice: Acquisition planning;
Activity: Plans are prepared during acquisition planning and maintained
throughout the acquisition;
5000 series addresses this activity? Yes.
Best practice: Acquisition planning;
Activity: Planning addresses the entire acquisition process, as well
as life cycle support of the products being acquired;
5000 series addresses this activity? Yes.
Best practice: Acquisition planning;
Activity: The acquisition organization has a written policy for
planning the acquisition;
5000 series addresses this activity? Yes.
Best practice: Acquisition planning;
Activity: Responsibility for acquisition planning activities is
designated;
5000 series addresses this activity? Yes.
Best practice: Architectural alignment;
5000 series incorporates this best practice? Partially.
Best practice: Architectural alignment;
Activity: The system being acquired is assessed for alignment with the
enterprise architecture at key life cycle decision points, and any
deviations from the architecture are understood and justified by an
explicit waiver to the architecture;
5000 series addresses this activity? Yes.
Best practice: Architectural alignment;
Activity: Product line requirements--rather than just the requirements
for the system being acquired--are an explicit consideration in each
acquisition;
5000 series addresses this activity? No.
Best practice: Contract tracking and oversight;
5000 series incorporates this best practice? Fully.
Best practice: Contract tracking and oversight;
Activity: The acquiring organization has sufficient insight into the
contractor's activities to manage and control the contractor and ensure
that contract requirements are met;
5000 series addresses this activity? Yes.
Best practice: Contract tracking and oversight;
Activity: The acquiring organization and contractor maintain ongoing
communication;
commitments are agreed to and implemented by both parties;
5000 series addresses this activity? Yes.
Best practice: Contract tracking and oversight;
Activity: All contract changes are managed throughout the life of the
contract;
5000 series addresses this activity? Yes.
Best practice: Contract tracking and oversight;
Activity: The acquisition organization has a written policy for
contract tracking and oversight;
5000 series addresses this activity? Yes.
Best practice: Contract tracking and oversight;
Activity: Responsibility for contract tracking and oversight activities
is designated;
5000 series addresses this activity? Yes.
Best practice: Contract tracking and oversight;
Activity: The acquiring organization involves contracting specialists
in the execution of the contract;
5000 series addresses this activity? Yes.
Best practice: Contract tracking and oversight;
Activity: A quantitative set of software and system metrics are used
to define and measure product quality and contractor performance;
5000 series addresses this activity? Yes.
Best practice: Contract tracking and oversight;
Activity: In addition to incentives for meeting cost and schedule
estimates, measurable, metrics-based product quality incentives are
explicitly cited in the contract;
5000 series addresses this activity? Yes.
Best practice: Economic justification;
5000 series incorporates this best practice? Fully.
Best practice: Economic justification;
Activity: System investment decisions are made on the basis of
reliable analyses of estimated costs, expected benefits, and
anticipated risks;
5000 series addresses this activity? Yes.
Best practice: Economic justification;
Activity: Large system projects are (to the maximum extent practical)
divided into a series of smaller, incremental acquisition efforts, and
investment decisions on these smaller efforts are made on the basis of
reliable analyses of estimated costs, expected benefits, and
anticipated risks;
5000 series addresses this activity? Yes.
Best practice: Evaluation;
5000 series incorporates this best practice? Fully.
Best practice: Evaluation;
Activity: Evaluation requirements are developed in conjunction with the
contractual requirements and are maintained over the life of the
acquisition;
5000 series addresses this activity? Yes.
Best practice: Evaluation;
Activity: Evaluations are planned and conducted throughout the total
acquisition period to provide an integrated approach that satisfies
evaluation requirements and takes advantage of all evaluation results;
5000 series addresses this activity? Yes.
Best practice: Evaluation;
Activity: Evaluations provide an objective basis to support the product
acceptance decision;
5000 series addresses this activity? Yes.
Best practice: Evaluation;
Activity: The acquiring organization has a written policy for managing
the evaluation of the acquired products;
5000 series addresses this activity? Yes.
Best practice: Evaluation;
Activity: Responsibility for evaluation activities is designated;
5000 series addresses this activity? Yes.
Best practice: Project management;
5000 series incorporates this best practice? Partially.
Best practice: Project management;
Activity: Project management activities are planned, organized,
controlled, and communicated;
5000 series addresses this activity? Partly--communication not cited.
Best practice: Project management;
Activity: The performance, cost, and schedule of the acquisition are
continually measured, compared with planned objectives, and controlled;
5000 series addresses this activity? Yes.
Best practice: Project management;
Activity: Problems discovered during the acquisition are managed and
controlled;
5000 series addresses this activity? Yes.
Best practice: Project management;
Activity: The acquisition organization has a written policy for
project management;
5000 series addresses this activity? Yes.
Best practice: Project management;
Activity: Responsibility for project management is designated;
5000 series addresses this activity? Yes.
Best practice: Requirements development and management;
5000 series incorporates this best practice? Fully.
Best practice: Requirements development and management;
Activity: Contractual requirements are developed, managed, and
maintained;
5000 series addresses this activity? Yes.
Best practice: Requirements development and management;
Activity: The end user and other affected groups have input into the
contractual requirements over the life of the acquisition;
5000 series addresses this activity? Yes.
Best practice: Requirements development and management;
Activity: Contractual requirements are traceable and verifiable;
5000 series addresses this activity? Yes.
Best practice: Requirements development and management;
Activity: The contractual requirements baseline is established prior to
release of the solicitation package;
5000 series addresses this activity? Yes.
Best practice: Requirements development and management;
Activity: The acquisition organization has a written policy for
establishing and managing the contractual requirements;
5000 series addresses this activity? Yes.
Best practice: Requirements development and management;
Activity: Responsibility for requirements development and management is
designated;
5000 series addresses this activity? Yes.
Best practice: Requirements development and management;
Activity: Requirements that are mandatory versus optional are clearly
delineated and used in deciding what requirements can be eliminated or
postponed to meet other project goals, such as cost and schedule
constraints;
5000 series addresses this activity? Yes.
Best practice: Risk management;
5000 series incorporates this best practice? Partially.
Best practice: Risk management;
Activity: Projectwide participation in the identification and
mitigation of risks is encouraged;
5000 series addresses this activity? Yes.
Best practice: Risk management;
Activity: The defined acquisition process provides for the
identification, analysis, and mitigation of risks;
5000 series addresses this activity? Yes.
Best practice: Risk management;
Activity: Milestone reviews include the status of identified risks;
5000 series addresses this activity? No.
Best practice: Risk management;
Activity: The acquisition organization has a written policy for
managing acquisition risk;
5000 series addresses this activity? Yes.
Best practice: Risk management;
Activity: Responsibility for acquisition risk management activities is
designated;
5000 series addresses this activity? Yes.
Best practice: Solicitation;
5000 series incorporates this best practice? Fully.
Best practice: Solicitation;
Activity: The solicitation package includes the contractual
requirements and the proposal evaluation criteria;
5000 series addresses this activity? Yes.
Best practice: Solicitation;
Activity: The technical and management elements of proposals are
evaluated to ensure that the requirements of the contract will be
satisfied;
5000 series addresses this activity? Yes.
Best practice: Solicitation;
Activity: The selection official selects a supplier who is qualified
to satisfy the contract's requirements;
5000 series addresses this activity? Yes.
Best practice: Solicitation;
Activity: The acquiring organization has a written policy for
conducting the solicitation;
5000 series addresses this activity? Yes.
Best practice: Solicitation;
Activity: Responsibility for the solicitation is designated;
5000 series addresses this activity? Yes.
Best practice: Solicitation;
Activity: A selection official has been designated to be responsible
for the selection process and decision;
5000 series addresses this activity? Yes.
Best practice: Solicitation;
Activity: The acquiring team includes contracting specialists to
support contract administration;
5000 series addresses this activity? Yes.
Best practice: Transition to support;
5000 series incorporates this best practice? Fully.
Best practice: Transition to support;
Activity: The acquiring organization ensures that the support
organization has the capacity and capability to provide the required
support;
5000 series addresses this activity? Yes.
Best practice: Transition to support;
Activity: There is no loss in continuity of support to the products
during transition from the supplier to the support organization;
5000 series addresses this activity? Yes.
Best practice: Transition to support;
Activity: Configuration management of the products is maintained
throughout the transition;
5000 series addresses this activity? Yes.
Best practice: Transition to support;
Activity: The acquiring organization has a written policy for
transitioning the products to the support organization;
5000 series addresses this activity? Yes.
Best practice: Transition to support;
Activity: The acquiring organization ensures that the support
organization is involved in planning for transition to support;
5000 series addresses this activity? Yes.
Best practice: Transition to support;
Activity: Responsibility for transition to support activities is
designated;
5000 series addresses this activity? Yes.
Source: GAO, based on analysis of DOD data.
[End of table]
The 5000 Series Generally Does Not Incorporate Best Practices Relevant
to Commercial Component-Based Business Systems Acquisitions:
Of the 8 best practices relevant to acquiring commercial component-
based business systems, few have been incorporated into DOD systems
acquisition policies and guidance. (See table 4 for our detailed
comparative analysis of the 5000 series against the 8 best practices.)
For example, while the practice aimed at ensuring that adequate
planning takes place for integrating commercial products with legacy
systems is incorporated into the 5000 series, practices associated with
closely controlling any modification to the software of these packages
and products, thoroughly analyzing and understanding the dependencies
among commercial products before acquiring them, and proactively
managing the institutional change that results from implementing the
functionality in commercial packages and products are not incorporated.
According to DOD officials responsible for revising the 5000 series,
these practices were not included in the recently revised version of
DOD's acquisition policies because they included only those in existing
law or regulation.
Nevertheless, the absence of these practices from the 5000 series
increases the risk that the practices will not be performed, which, in
turn, increases the risk that acquisition projects will fall short of
expectations. The practice intended to ensure development of a quality
solicitation and selection of a best-qualified contractor illustrates
this. Specifically, this practice calls for contract bidders to be
evaluated on their ability to implement commercial components. This
evaluation is important because integrating and implementing these
component products is sufficiently different from developing customized
system solutions; it requires different core competencies and
experiences to be successful. By explicitly taking this into
consideration in evaluating and selecting a contractor, the risk of
contract award to a less-than-best-qualified contractor is reduced.
Table 4: Activity-by-Activity Comparison of the 5000 Series to Best
Practices Relevant to Commercial Component-based Business Systems
Acquisitions:
Best practice: Component modification;
Does the 5000 series incorporate this best practice? No.
Best practice: Component modification;
Activity: Modification of commercial components is discouraged and
allowed only if justified by a thorough analysis of life-cycle costs
and benefits;
Does the 5000 series address this activity? No.
Best practice: Configuration management;
Does the 5000 series address this activity? Partially.
Best practice: Configuration management;
Activity: Project plans provide for evaluation, acquisition, and
implementation of new, often frequent, product releases;
Does the 5000 series address this activity? Yes.
Best practice: Configuration management;
Activity: Modification or upgrades to deployed versions of system
components are centrally controlled and unilateral user release
changes are precluded;
Does the 5000 series address this activity? No.
Best practice: Legacy systems integration planning;
Does the 5000 series address this activity? Fully.
Best practice: Legacy systems integration planning;
Activity: Project plans explicitly provide for the necessary time and
resources for integrating commercial components with legacy systems;
Does the 5000 series address this activity? Yes.
Best practice: Dependency analysis;
Does the 5000 series address this activity? No.
Best practice: Dependency analysis;
Activity: Decisions about acquisition of commercial components are
based on deliberate and thorough research, analysis, and evaluation of
the components' interdependencies;
Does the 5000 series address this activity? No.
Best practice: Organization change management;
Does the 5000 series address this activity? No.
Best practice: Organization change management;
Activity: Project plans provide for preparing users for the impact
that the business processes embedded in the commercial components will
have on the users' respective roles and responsibilities;
Does the 5000 series address this activity? No.
Best practice: Organization change management;
Activity: The introduction and adoption of changes to how users will
be expected to use the system to execute their jobs are actively
managed;
Does the 5000 series address this activity? No.
Best practice: Solicitation;
Does the 5000 series address this activity? No.
Best practice: Solicitation;
Activity: Systems integration contractors are explicitly evaluated on
their ability to implement commercial components;
Does the 5000 series address this activity? No.
Best practice: Tradeoff analysis;
Does the 5000 series address this activity? No.
Best practice: Tradeoff analysis;
Activity: Investment decisions throughout a system's life cycle are
based on tradeoffs among the availability of commercial products
(current and future), the architectural environment in which the system
is to operate (current and future), defined system requirements, and
acquisition cost/schedule constraints;
Does the 5000 series address this activity? No.
Best practice: Vendor and product research and evaluation;
Does the 5000 series address this activity? Partially.
Best practice: Vendor and product research and evaluation;
Activity: Commercial component and vendor options are researched,
evaluated/tested, and understood, both early and continuously;
Does the 5000 series address this activity? Yes.
Best practice: Vendor and product research and evaluation;
Activity: A set of evaluation criteria for selecting among commercial
component options is established that includes both defined system
requirements and vendor/commercial product characteristics (e.g.,
customer satisfaction with company and product line);
Does the 5000 series address this activity? Partly--vendor/commercial
product characteristics not cited.
Source: GAO, based on analysis of DOD data.
[End of table]
DOD Officials Report That They Are in the Process of Revising the
Interim Defense Acquisition Guidebook:
The DOD officials responsible for revising the 5000 series told us they
recognize the need for the 5000 series to incorporate additional best
practices. To this end, they reported that efforts are under way to
expand the Interim Defense Acquisition Guidebook to include additional
best practices and lessons learned across the department. However, the
officials could not provide us with a documented plan and associated
documentation showing how this task will be accomplished, what
resources are needed and assigned to accomplish it, when it will be
accomplished, and where the department stands in accomplishing it.
Instead, the officials told us that progress on it has been slowed by
other priorities, such as the need to first revise DOD Directive 5000.1
and DOD Instruction 5000.2. They also said that there is only a small
number of staff available to work on what they described as being an
extensive revision of the guidebook. According to these officials, 80
to 90 percent of the revision has been completed and reviewed and their
goal is to publish the initial version of the revised guidebook by
September 30, 2004. In our view, until the missing best practices that
we cite in this report are included in DOD's acquisition policies and
guidance, the chance that business systems acquisitions will follow the
policies and guidance and consistently produce a successful outcome is
diminished.
DOD's Acquisition Policies Do Not Contain Sufficient Controls to Ensure
That the Requirement Is Met for Appropriately Applying Best Practices:
Federal laws and regulations define the need for effective controls
over agency programs, and controls are a key factor in achieving
program results, minimizing operational problems, and managing evolving
demands and priorities.[Footnote 9] Controls over defined processes,
procedures, and support activities are considered effective if they
entail measuring and verifying whether a given practice is followed.
Without sufficient controls, it is unlikely that practices will be
consistently employed, which, in turn, increases the probability that
the positive program and project outcomes these practices are designed
to produce will not occur.
DOD's acquisition policy requires program managers and investment
decision authorities to examine and, as appropriate, adopt best
practices. However, neither the policies nor the accompanying guidance
explain what "examine" means, including whether practice use is to be
measured and verified. Instead, the policies state that any issues
regarding the intent of the 5000 series, which would include whether
practice adoption is to be measured and validated, shall be resolved by
the investment decision authority, meaning that it is entirely up to
this individual what information relative to the use of best practices
is relevant and necessary to ensure that best practices are
appropriately followed. According to the Chairman of the Defense
Acquisition Policy Working Group,[Footnote 10] this control is
sufficient, and explicit requirements for measuring and validating the
use of best practices are not necessary.
In our view, not requiring that decision authorities examine the
measurement and validation of best practices' use increases the chance
that important best practices will not be appropriately followed, as:
required by DOD policy. As we have previously reported,[Footnote 11] a
lack of explicit controls that require review of relevant information
at key decision points raises the risk of making uninformed project
decisions, as well as the risk that investments will not meet cost,
schedule, capability, and benefit commitments.
Conclusions:
DOD recognizes the importance of business systems acquisition best
practices by including best practices in their revised policy and
guidance. However, other practices that, if followed, could increase
the odds of acquisitions delivering promised system capabilities and
benefits on time and within budget have yet to be similarly included.
In particular, those practices associated with the successful
acquisition of commercial component-based business systems have not
been sufficiently incorporated into either the policies or the
guidance. Moreover, effective controls for ensuring that best practices
are appropriately followed are not adequately provided for in the
policies. Although DOD officials intend to expand the coverage of best
practices in future versions of DOD's acquisition guidance, it is
unclear what the scope, nature, and status of these intentions are
because explicit plans for revising the guidance and associated
progress reports were not available. Until DOD incorporates the best
practices we found missing in the 5000 series, and until it strengthens
the means by which the appropriate use of these practices will be
ensured, its business systems acquisitions will be exposed to
unnecessary risk. Therefore, it is important for DOD to treat further
revisions of its acquisition policy and guidance relative to business
systems as a priority and move quickly to incorporate missing best
practices and associated controls for ensuring that the practices are
followed.
Recommendations for Executive Action:
To improve DOD's ability to acquire business systems, we recommend that
the Secretary of Defense direct the Under Secretary for AT&L, in
collaboration with the Assistant Secretary for NII and the Director,
OT&E, to take the following actions:
1. Develop and implement an explicit plan for incorporating into the
5000 series the best practices and associated activities currently
missing from the series. We recommend that the plan specify tasks to be
performed, resources needed and assigned, and milestones for completing
tasks.
2. We further recommend that progress against this plan be tracked and
reported as appropriate, and that the plan, at a minimum, incorporate
each of the following best practice activities:
* Product line requirements--rather than just the requirements for the
system being acquired--are an explicit consideration in each
acquisition.
* Acquisition project management activities are communicated to all
stakeholders.
* Acquisition reviews include the status of identified risks.
* Modification of commercial components is discouraged and allowed only
if justified by a thorough analysis of life-cycle costs and benefits.
* Modification or upgrades to deployed versions of system components
are centrally controlled, and unilateral user release changes are
precluded.
* Acquisition decisions about commercial components are based on
deliberate and thorough research, analysis, and evaluation of the
components' interdependencies.
* Acquisition plans provide for preparing users for the impact that the
business processes embedded in the commercial components will have on
their respective roles and responsibilities.
* Changes affecting how users will be expected to use the system to
execute their jobs are actively managed.
* Systems integration contractors are explicitly evaluated on their
ability to implement commercial components.
* Investment decisions throughout a system's life cycle are based on a
continuous set of tradeoffs among capabilities available in commercial
components (current and future), the architectural environment in which
the system is to operate, defined system requirements, and existing
cost/schedule constraints.
* Evaluation criteria are established for selecting among commercial
component options that include both defined system requirements and
vendor/commercial product characteristics.
3. To ensure that the best practices provided for in DOD acquisition
policies and guidance are appropriately followed, we also recommend
that the above recommended plan incorporate steps to include in DOD's
acquisition policies a provision for measurement and verification of
best practices.
Agency Comments and Our Evaluation:
In written comments on a draft of this report, signed by the Principal
Director for Command, Control, Communications, Space, and Information
Technology Programs in the office of the DOD Assistant Secretary for
Networks and Information Integration, DOD agreed with the importance
and relevance of the best practices that we cite in the report.
Additionally, DOD agreed with 2 of our 13 recommendations for
incorporating additional best practices, stating that the department
would incorporate the 2 practices in its policies and guidance.
DOD also partially agreed with 9 of our recommendations for
incorporating additional practices, stating that it would consider
augmenting its coverage of 5 of the practices and that it believed that
4 practices already existed in its policies and guidance. With regard
to the 5 practices, DOD stated that it needed to review each practice
further and determine the need for its emphasis or endorsement in the
5000 series. We understand DOD's desire to carefully consider changes
to its acquisition policies and guidance, and believe that such careful
deliberation is consistent with the spirit of our recommendations.
With regard to the remaining 4 practices that DOD partially agreed
with, we do not agree with the department's comment that these best
practices adequately exist in the 5000 series. For example, DOD
commented that because its existing policies and guidance provide for
the use of integrated product teams, which, according to DOD, are a
means for promoting collaboration and facilitating communication among
stakeholders, its policies and guidance therefore already provide for
communicating information about management of a given acquisition
project to all relevant project stakeholders. While we do not question
the use of integrated product teams as a way to communicate
information, the point of our recommendation is that there needs to be
an explicit recognition in policy or guidance of the type of
information to be communicated and with whom it is to be communicated.
Restated, our recommendation for incorporating the best practice of
communicating acquisition management activities to all stakeholders is
intended to permit communication vehicles, such as integrated product
teams, to be more effective by explicitly providing for this best
practice in relevant policies and guidance. As another example, we do
not agree with DOD's comment that 2 of the best practices that we
recommended for incorporation in its policies and guidance--preparing
users for the impact that business process changes embedded in
commercial components will have on their roles and responsibilities,
and actively managing changes in how users will use new systems--are
already sufficiently contained in the 5000 series. In particular, while
we agree that the series references an acquisition management toolkit
that addresses these 2 best practices, this reference is provided only
once in the 5000 series, and this reference is only in relation to one
phase of the acquisition cycle (the technology development phase).
Given the importance and relevance of these practices to successful
implementation of commercial component-based systems, our position, and
thus the basis for our recommendations, is that the practices'
implementation would be more likely to occur if the practices were
visible and better recognized in all relevant stages of DOD's
acquisition cycle.
Also in its comments, DOD did not agree with our recommendations to
develop and implement an explicit plan to govern its ongoing and future
policy and guidance revision activities, specifically stating that the
recommendation was inappropriate and offering updated information on
the status of and associated milestone for completing its activities.
While we have updated our report to include the revised status and
milestone information, we do not agree with DOD that a plan governing
these efforts is not needed. Given the importance of DOD's acquisition
policies and guidance, and the need for their continuous review and
update to reflect new acquisition best practices, we believe that
having an explicit plan that defines how and when these policies and
guidance will be incorporated is essential. Among other things, a plan
would highlight the resource constraints that this revision effort has
been subject to, would allow measurement against defined milestones,
and would allow disclosure of progress and impediments.
DOD also did not agree with our recommendation to add stronger controls
for ensuring adherence to the best practices that are contained in its
acquisition policies and guidance, stating that its existing oversight
process includes the necessary compliance activities. We disagree. As
we state in the report, DOD's existing policy leaves these compliance
activities to the discretion of the program manager and the investment
decision authority, and it does not provide for measurement and
verification of the use of best practices, both of which are recognized
components of effective control processes.
A copy of DOD's comments is reprinted in appendix III, along with our
response.
We are sending copies of this report to the Chairmen and Ranking
Minority Members of the Senate and House Committees on Armed Services;
Subcommittees on Defense, Senate and House Committees on
Appropriations; and the Subcommittee on Military Readiness, House
Committee on Armed Services. We are also sending copies to the
Director, Office of Management and Budget; the Secretary of Defense;
the Under Secretary of Defense (AT&L); the Assistant Secretary of
Defense (NII)/Chief Information Officer; and the Director, OT&E. We
will make copies available to others on request. This report will also
be available at no charge on our Web site at http: //www.gao.gov.
If you or your staff has any questions concerning this report, please
contact me at (202) 512-3439. I can also be reached by e-mail at
[Hyperlink, hiter@gao.gov]. Other contacts and key contributors to this
report are listed in appendix IV.
Signed by:
Randolph C. Hite:
Director, Information Technology Architecture and Systems Issues:
[End of section]
Appendixes:
Appendix I: Objectives, Scope, and Methodology:
Our objectives were to determine whether the Department of Defense's
(DOD) revised systems acquisition policies for acquiring information
technology (IT) business systems (1) are consistent with industry best
practices, including those pertaining to commercial component-based
systems, and (2) provide the necessary controls to ensure that the
department's component organizations adhere to the practices.
To accomplish the first objective, we identified the DOD policies and
guidance relevant to business systems. These policies and guidance are
contained in three documents--DOD Directive 5000.1, DOD Instruction
5000.2, and the Interim Defense Acquisition Guidebook--and are
generally referred to as the 5000 series. We then reviewed each of
these documents and discussed with DOD officials responsible for
developing and revising the documents what steps were taken to
incorporate best practices into each document. The DOD officials that
we interviewed were from the offices of the Under Secretary of Defense
(Acquisition, Technology, and Logistics (AT&L)); the Assistant
Secretary of Defense (Networks and Information Integration (NII)); and
the Director, Operational Test and Evaluation (OT&E). Next, we
researched prior GAO reports; the work of federally funded research and
development organizations, such as the Software Engineering Institute
and The Aerospace Corporation;[Footnote 12] and other authoritative
sources to identify business systems acquisition best practices. Our
research produced 18 best practices, including associated activities,
that we placed into two categories--one category for the practices that
are relevant to any business systems acquisition and one category for
the practices that are relevant to commercial component-based business
systems acquisitions. In particular, we drew extensively from the
Software Engineering Institute's Software Acquisition Capability
Maturity Model.[Footnote 13] In doing so, we selected practices from
the model's repeatable level of process maturity, which is level two on
the model's five-level scale. We used the repeatable level of process
maturity because it is intended to provide the necessary process
discipline to allow an organization to repeat earlier successes on
similar projects. In addition, we included one Software Acquisition
Capability Maturity Model level-three process area--risk management--
because many experts consider it to be one of the most important
process areas. We did not attempt to develop an exhaustive list of best
practices and, in fact, fully recognize that additional best practices
exist, such as ensuring that the appropriate level of human capital
knowledge, skills, and abilities are employed, as well as that
additional activities for the practices that we have identified exist,
such as those configuration management activities associated with
identifying, controlling, reporting on, and auditing configuration
items and components. For the purposes of this report, we identified
those practices that are embodied, recognized, and accepted acquisition
models or frameworks, as well as those practices that are now being
recognized as being unique to commercial component-based systems and
for which there appears to be general agreement, including agreement
with DOD officials responsible for revising the 5000 series, that the
practices are relevant and important. Last, we analyzed each of the DOD
5000 series documents to determine whether the documents addressed,
either directly or indirectly by reference to another authoritative
document, the 18 best practices that we identified. Based on this
analysis, we judged whether the 5000 series documents fully, partially,
or did not incorporate each best practice. In making these judgments,
we used the following criteria:
* To fully incorporate the practice, the 5000 series addressed all of
the practice's activities.
* To partially incorporate the practice, the 5000 series addressed
some, but not all, of the practice's activities.
* To not incorporate the practice, the 5000 series did not address any
of the practice's activities.
Additionally, we provided the DOD officials responsible for revising
the 5000 series with the 18 practices that we identified to obtain
their views on whether the practices were relevant to DOD business
systems acquisitions. The officials agreed that they were. We also
requested that these officials perform their own assessment of the 5000
series against these practices, and we used these officials' assessment
in making our judgments as to whether the practices were fully,
partially, or not incorporated into DOD's acquisition policies and
guidance. For a number of the activities, DOD identified the Federal
Acquisition Regulation and the Defense Federal Acquisition Regulation
Supplement as evidence that the activity was being performed within a
particular practice. We accepted that as proof that the activity was
being covered within DOD's business systems acquisition policy.
To address our second objective, we researched federal internal control
standards and controls inherent in the business systems acquisition
best practices that we identified. In particular, we reviewed the
Software Engineering Institute's Software Acquisition Capability
Maturity Model framework and GAO's internal control standards.[Footnote
14] We then analyzed DOD's revised acquisition policies and guidance to
identify whether these controls were cited and to provide assurance
that relevant best practices were being followed. We also interviewed
DOD officials responsible for revising the 5000 series to determine
reasons why controls were addressed or not addressed in the policies
and guidance.
We conducted our work at DOD offices in Arlington, Virginia, between
December 2003 and May 2004 in accordance with generally accepted
government auditing standards:
[End of section]
Appendix II: Best Practices:
Additional information on each of the 18 best practices that we
identified is provided in this appendix.
Best Practices Relevant to Any IT Business Systems Acquisition:
1. Acquisition Planning:
Purpose: To ensure that reasonable planning for all parts of the
acquisition is conducted.
Description: Acquisition planning is the process for conducting and
documenting acquisition planning activities beginning early and
covering all parts of the project. It extends to all acquisition areas,
such as budgeting, scheduling, resource estimating, risk
identification, and requirements definition, as well as the overall
acquisition strategy. Acquisition planning begins with the earliest
identification of a requirement that is to be satisfied through an
acquisition.
Activities: (1) Plans are prepared during acquisition planning and
maintained throughout the acquisition. (2) Planning addresses the
entire acquisition process, including life cycle support of the
products being acquired. (3) The acquisition organization has a written
policy for planning the acquisition. (4) Responsibility for acquisition
planning activities is designated.
2. Architectural Alignment:
Purpose: To ensure that the acquisition is consistent with the
organization's enterprise architecture.
Description: Architectural alignment is the process for analyzing and
verifying that the proposed architecture of the system being acquired
is consistent with the enterprise architecture for the organization
acquiring the system. Such alignment is needed to ensure that acquired
systems can interoperate and are not unnecessarily duplicative of one
another. Exceptions to this alignment requirement are permitted, but
only when justified and only when granted an explicit waiver from the
architecture. A particular architectural consideration is whether
requirements that extend beyond the specific system being acquired
should be considered when selecting system components. Such product
line (i.e., systems that are developed from a common set of assets and
share a common and managed set of features) considerations can provide
substantial production economies over acquiring systems from scratch.
Activities: (1) The system being acquired is assessed for alignment
with the enterprise architecture at key life cycle decision points, and
any deviations from the architecture are explicitly understood and
justified by an explicit waiver to the architecture. (2) Product line
requirements--rather than just the requirements for the system being
acquired--are an explicit consideration in each acquisition.
3. Contract Tracking and Oversight:
Purpose: To ensure that contract activities are performed in accordance
with contractual requirements.
Description: Contract tracking and oversight is the process by which
contractual agreements are established and contractor efforts to
satisfy those agreements are supervised. It involves information
sharing between the acquirer and contractor to ensure that contractual
requirements are understood, that there are regular measurements to
disclose overall project status and whether problems exist, and that
there are appropriate incentives for ensuring that cost and schedule
commitments are met and that quality products are delivered. Contract
tracking and oversight begins with the award of the contract and ends
at the conclusion of the contract's period of performance.
Activities: (1) The acquiring organization has sufficient insight into
the contractor's activities to manage and control the contractor and
ensure that contract requirements are met. (2) The acquiring
organization and contractor maintain ongoing communication;
commitments are agreed to and implemented by both parties. (3) All
contract changes are managed throughout the life of the contract. (4)
The acquiring organization has a written policy for contract tracking
and oversight. (5) Responsibility for contract tracking and oversight
activities is designated. (6) The acquiring organization involves
contracting specialists in the execution of the contract. (7) A
quantitative set of software and system metrics is used to define and
measure product quality and contractor performance.[Footnote 15] (8)
In addition to incentives for meeting cost and schedule estimates,
measurable, metrics-based product quality incentives are explicitly
cited in the contract.[Footnote 16]
4. Economic Justification:
Purpose: To ensure that system investments have an adequate economic
justification.
Description: Economic justification is the process for ensuring that
acquisition decisions are based on reliable analyses of the proposed
investment's likely costs versus benefits over its useful life, as well
as an analysis of the risks associated with actually realizing the
acquisition's forecasted benefits for its estimated costs. Moreover, it
entails minimizing the risk and uncertainty of large acquisitions that
require spending large sums of money over many years by breaking the
acquisition into smaller, incremental acquisitions. Economic
justification is not a one-time event, but rather is performed
throughout an acquisition's life cycle in order to permit informed
investment decision making.
Activities: (1) System investment decisions are made on the basis of
reliable analyses of estimated system life cycle costs, expected
benefits, and anticipated risks. (2) Large systems acquisitions are (to
the maximum extent practical) divided into a series of smaller,
incremental acquisition efforts, and investment decisions on these
smaller efforts are made on the basis of reliable analyses of estimated
costs, expected benefits, and anticipated risks.
5. Evaluation:
Purpose: To ensure that evidence showing that the contract products
satisfy the defined requirements are provided prior to accepting
contractor products.
Description: Evaluation is the process by which contract deliverables
are analyzed to determine whether they meet contract requirements. It
includes developing criteria such as product acceptance criteria to be
included into both the solicitation package and the contract. It should
be conducted continuously throughout the contract period as products
are delivered. It begins with development of the products' requirements
and ends when the acquisition is completed.
Activities: (1) Evaluation requirements are developed in conjunction
with the contractual requirements and are maintained over the life of
the acquisition. (2) Evaluations are planned and conducted throughout
the total acquisition period to provide an integrated approach that
satisfies evaluation requirements and takes advantage of all evaluation
results. (3) Evaluations provide an objective basis to support the
product acceptance decision. (4) The acquisition organization has a
written policy for managing the evaluation of the acquired products.
(5) Responsibility for evaluation activities is designated.
6. Project Management:
Purpose: To ensure that the project office and its supporting
organizations function efficiently and effectively.
Description: Project management is the process for planning,
organizing, staffing, directing, and managing all project-office-
related activities, such as defining project tasks, estimating and
securing resources, scheduling activities and tasks, training, and
accepting products. Project management begins when the project office
is formed and ends when the acquisition is completed.
Activities: (1) Project management activities are planned, organized,
controlled, and communicated. (2) The performance, cost, and schedule
of the acquisition are continually measured, compared with planned
objectives, and controlled. (3) Problems discovered during the
acquisition are managed and controlled. (4) The acquisition
organization has a written policy for project management. (5)
Responsibility for project management is designated.
7. Requirements Development and Management:
Purpose: To ensure that contractual requirements are clearly defined
and understood by the acquisition stakeholders.
Description: Requirements development is the process for developing and
documenting contractual requirements, including evaluating
opportunities for reusing existing assets. It involves participation
from end users to ensure that product requirements are well understood,
and that optional versus mandatory requirements are clearly delineated.
Requirements management is the process for establishing and maintaining
agreement on the contractual requirements among the various
stakeholders and for ensuring that the requirements are traceable,
verifiable, and controlled. This involves baselining the requirements
and controlling subsequent requirements changes. Requirements
development and management begins when the solicitation's requirements
are documented and ends when system responsibility is transferred to
the support organization.
Activities: (1) Contractual requirements are developed, managed, and
maintained. (2) The end user and other affected groups have input into
the contractual requirements over the life of the acquisition. (3)
Contractual requirements are traceable and verifiable. (4) The
contractual requirements baseline is established prior to release of
the solicitation package. (5) The acquisition organization has a
written policy for establishing and managing the contractual
requirements. (6) Responsibility for requirements development and
management is designated. (7) Requirements that are mandatory versus
optional are clearly delineated and used in deciding what requirements
can be eliminated or postponed to meet other project goals, such as
cost and schedule constraints.[Footnote 17]
8. Risk Management:
Purpose: To ensure that risks are identified and systematically
mitigated.
Description: Risk management is the process for identifying potential
acquisition problems and taking appropriate steps to avoid their
becoming actual problems. It includes risk identification and
categorization based on estimated impact, development of risk
mitigation strategies, and execution of and reporting on the
strategies. Risk management occurs early and continuously in the
acquisition life cycle.
Activities: (1) Projectwide participation in the identification and
mitigation of risks is encouraged. (2) The defined acquisition process
provides for the identification, analysis, and mitigation of risks. (3)
Milestone reviews include the status of identified risks. (4) The
acquisition organization has a written policy for managing acquisition
risk. (5) Responsibility for acquisition risk management activities is
designated.
9. Solicitation:
Purpose: To ensure that a quality solicitation is produced and a best-
qualified contractor selected.
Description: Solicitation is the process for developing, documenting,
and issuing the solicitation package; developing and implementing a
plan to evaluate responses; conducting contract negotiations; and
awarding the contract. Solicitation ends with contract award.
Activities: (1) The solicitation package includes the contractual
requirements and the proposal evaluation criteria. (2) The technical
and management elements of proposals are evaluated to ensure that the
requirements of the contract will be satisfied. (3) The selection
official selects a supplier who is qualified to satisfy the contract's
requirements. (4) The acquiring organization has a written policy for
conducting the solicitation. (5) Responsibility for the solicitation is
designated. (6) A selection official has been designated to be
responsible for the selection process and decision. (7) The acquiring
team includes contracting specialists to support contract
administration.
10. Transition to Support:
Purpose: To ensure proper transfer of the system from the acquisition
organization to the eventual support organization.
Description: Transition to support is the process for developing and
implementing the plans for transitioning products to the support
organization. This includes engaging relevant stakeholders in the
acquisition and sharing information about the system's supporting
infrastructure. Transition to support begins with requirements
development and ends when the responsibility for the products is turned
over to the support organization.
Activities: (1) The acquiring organization ensures that the support
organization has the capacity and capability to provide the required
support. (2) There is no loss in continuity of support to the products
during transition from the supplier to the support organization. (3)
Configuration management of the products is maintained throughout the
transition. (4) The acquiring organization has a written policy for
transitioning products to the support organization. (5) The acquiring
organization ensures that the support organization is involved in
planning for transition to support. (6) Responsibility for transition
to support activities is designated.
Complementary Best Practices Relevant to Commercial Component-Based IT
Business Systems Acquisitions:
1. Component Modification:
Purpose: To ensure that commercial product modification is effectively
controlled.
Description: Component modification is the process for limiting the
chances of a commercial product being modified to the point that it
becomes a one-of-a-kind solution because doing so can result in
extensive life cycle costs. Such modifications, if not incorporated
into the commercially available version of the product by the supplier,
mean that every product release has to be modified in accordance with
the custom changes, thus precluding realization of some of the benefit
of using a commercial product.
Activity: (1) Modification of commercial components is discouraged and
allowed only if justified by a thorough analysis of life cycle costs
and benefits.[Footnote 18]
2. Configuration Management:
Purpose: To ensure the integrity and consistency of system commercial
components.
Description: Configuration management relative to commercial
component-based systems is the process for ensuring that changes to the
commercial components of a system are strictly controlled. It
recognizes that when using commercial components, it is the vendor, not
the acquisition or support organization, that controls the release of
new component versions and that new versions are released frequently.
Thus, acquisition management needs to provide for both receiving new
product releases and controlling the implementation of these releases.
Activities: (1) Project plans explicitly provide for evaluation,
acquisition, and implementation of new, often frequent, product
releases.[Footnote 19] (2) Modification or upgrades to deployed
versions of system components are centrally controlled, and unilateral
user release changes are precluded.
3. Dependency Analysis:
Purpose: To ensure that relationships between commercial products are
understood before acquisition decisions are made.
Description: Dependency analysis relative to commercial component-
based systems is the process for determining and understanding the
characteristics of these products so that inherent dependencies among
them can be considered before they are acquired. It involves
recognizing that the logical and physical relationships among products
impact one another. This is necessary because commercial products are
built around each vendor's functional and architectural assumptions and
paradigms, such as approaches to error handling and data access, and
these assumptions and paradigms are likely to be different among
products from different sources. Such differences complicate product
integration. Further, some commercial products have built-in
dependencies with other products that, if not known, can further
complicate integration.
Activity: (1) Decisions about the acquisition of commercial components
are based on deliberate and thorough research, analysis, and evaluation
of the components' interdependencies.[Footnote 20]
4. Legacy Systems Integration Planning:
Purpose: To ensure reasonable planning for integration of commercial
products with existing systems.
Description: Legacy systems integration planning is the process for
ensuring that the time and resources needed to integrate existing
systems with the system being acquired are identified and provided for.
It involves identifying which legacy systems will interact with the
system being acquired and what kinds and levels of testing are
required. Integration planning recognizes that, although some
commercial products may provide mechanisms and information that is
helpful in integration with legacy systems, the unavailability of the
source code for commercial products and the different organizations
that are responsible for the two will likely require additional time
and effort.
Activity: (1) Project plans explicitly provide for the necessary time
and resources for integrating commercial components with legacy
systems.
5. Organization Change Management:
Purpose: To ensure that the organizational impact of using new system
functionality is proactively managed.
Description: Organization change management relative to commercial
component-based systems is the process for preparing system users for
the business process changes that will accompany implementation of the
system. It involves engaging users and communicating the nature of
anticipated changes to system users through training on how jobs will
change. This is necessary because commercial products are created with
the developers' expectations of how they will be used, and the
products' functionality may require the organization implementing the
system to change existing business processes.
Activities: (1) Project plans explicitly provide for preparing users on
the impact that the business processes embedded in the commercial
components will have on the user's respective roles and
responsibilities. (2) The introduction and adoption of changes to how
users will be expected to execute their jobs are actively
managed.[Footnote 21]
6. Solicitation:
Purpose: To ensure that a quality solicitation is produced and a best-
qualified contractor is selected.
Description: Solicitation relative to commercial component-based
systems is the process for ensuring that a capable contractor is
selected. It involves ensuring that the selected contractor has
experience with integrating commercial component products. This is
important because expertise in developing custom system solutions is
different from expertise in implementing commercial components; it
requires different core competencies and experiences to be successful.
Activity: (1) Systems integration contractors are explicitly evaluated
on their ability to implement commercial components.[Footnote 22]
7. Tradeoff Analysis:
Purpose: To ensure that system requirements alone do not drive the
system solution.
Description: Tradeoff analysis relative to commercial product-based
systems is the process for analyzing and understanding the tradeoffs
among competing acquisition variables so as to produce informed
acquisition decision making. It involves planning and executing
acquisitions in a manner that recognizes four competing interests:
defined system requirements, the architectural environment (current and
future) in which the system needs to operate, acquisition cost and
schedule constraints, and the availability of products in the
commercial marketplace (current and future). This analysis should be
performed early and continuously throughout an acquisition's life
cycle.
Activity: (1) Investment decisions throughout a system's life cycle are
based on tradeoffs among the availability of commercial products
(current and future), the architectural environment in which the system
is to operate (current and future), defined system requirements, and
acquisition cost/schedule constraints.[Footnote 23]
8. Vendor and Product Research and Evaluation:
Purpose: To ensure that vendor and product characteristics are
understood before acquisition decisions are made.
Description: Vendor and product research and evaluation relative to
commercial component-based systems is the process for obtaining
reliable information about both the product being considered and the
vendor offering the product. It involves taking additional steps beyond
vendor representations, such as obtaining information about the
vendor's history, obtaining information on the vendor's business
strategy relative to evolution and support of the product, and
evaluating copies of the product in a test environment.
Activities: (1) Commercial component and vendor options are researched,
evaluated/tested, and understood, both early and continuously. (2) A
set of evaluation criteria for selecting among commercial component
options is established that includes both defined system requirements
and vendor/commercial product characteristics (e.g., customer
satisfaction with company and product line).
[End of section]
Appendix III: Comments from the Department of Defense:
OFFICE OF THE ASSISTANT SECRETARY OF DEFENSE:
6000 DEFENSE PENTAGON:
WASHINGTON, DC 20301-6000:
NETWORKS AND INFORMATION INTEGRATION:
July 1 2004:
Mr. Joel C. Willemssen:
Managing Director:
Information Technology Issues:
U.S. General Accounting Office:
441 G Street, N.W.
Washington, D.C. 20548:
Dear Mr. Willemssen:
This is the Department of Defense (DoD) response to the GAO draft
report (04-722), "INFORMATION TECHNOLOGY: DOD'S Acquisition Policies
and Guidance Need To Incorporate Additional Best Practices and
Controls," dated June 10, 2004 (GAO Code 310274).
We appreciate the opportunity to comment on the draft report and the
time your staff afforded us during their preparation of the report. Our
reply to each of the 14 recommendations is attached.
We recognize that the recommended best practices are based on guidance
published by such organizations as the Software Engineering Institute,
and we do not disagree that they are best practices. Our partial
concurrences are based on the fact that in some cases, we disagree with
the assertion that the proposed best practice does not exist in our
current directives and guidance system. However, in all cases, we will
consider including, or further emphasizing, the recommended best
practice in the Department's directives and guidance system.
Our point of contact is Dave Mullins at 703-602-2585.
Sincerely,
Signed by:
John R. Landon:
Principal Director Deputy Assistant Secretary of Defense (C3, Space &
IT Programs):
DoD Comments to GAO draft report (04-722), "INFORMATION TECHNOLOGY:
DOD'S Acquisition Policies and Guidance Need To Incorporate Additional
Best Practices and Controls," dated June 10, 2004 (GAO Code 310274).
RECOMMENDATION 1: The GAO recommended that the Secretary of Defense
direct the Under Secretary for AT&L, in collaboration with the
Assistant Secretary for NII and the Director, OT&E, to develop and
implement an explicit plan for incorporating into the 5000 series the
best practices and associated activities currently missing from the
series. (p. 22/GAO Draft Report):
DOD RESPONSE: Nonconcur. The Department believes this recommendation
and Recommendation 2 below are inappropriate. We agree that the best
practices with which we concur should be incorporated into the 5000
series or in related DoD policy or guidance documents, but do not agree
that a detailed plan with resources and milestones is needed.
Based on discussions with GAO staff, we understand that the intent of
this recommendation is for the Department to have a written program
plan for completing the DoD Acquisition Guidebook that is now under
development. We expect the Guidebook to be completed this summer. About
80 to 90 percent of it has been through an initial review throughout
the Department. The remaining sections will undergo such a review
within the next few weeks. After that, the entire document will be sent
out for a final review, and the primary authors will devote a few days
at an off site to make any final changes before final approval and
publication. The planned date for completion of these activities is not
later than September 30, 2004.
RECOMMENDATION 2: The GAO recommended that the Secretary of Defense
direct the Under Secretary for AT&L, in collaboration with the
Assistant Secretary for NII and the Director, OT&E, to develop and
implement an explicit plan that specify tasks to be performed,
resources needed and assigned, and milestones for completing tasks, and
that progress against the plan be tracked and reported as appropriate.
(p. 22/GAO Draft Report):
DOD RESPONSE: Nonconcur. See above for explanation.
RECOMMENDATION 3: The GAO recommended that the Secretary of Defense
direct the Under Secretary for AT&L, in collaboration with the
Assistant Secretary for NII and the Director, OT&E, to implement a
specific plan that, at a minimum provides for incorporating product
line requirements-rather than just the requirements for the system
being acquired-are an explicit consideration in each acquisition. (p.
22/GAO Draft Report):
DOD RESPONSE: Partially concur. We find this recommendation somewhat
confusing, but we believe it relates to considering the reuse of
products that may already have been developed or acquired within a
particular functional area or domain. The Enterprise Integration (EI)
Toolkit, which was developed and is maintained by the Office Deputy
Under Secretary of Defense for Logistics and Materiel Readiness,
strongly endorses the reuse of reports, interfaces, conversions, and
extensions (RICE) that have been built or acquired by other programs.
In fact, an initial operating capability for a repository of custom
software components that help adapt commercial components for defense
use and reuse is available. It can be accessed via the Reports,
Interfaces, Conversions, Extensions (RICE) Repository in the El Toolkit
at www.eitoolkit.com. We will research the Software Enterprise
Institute document that advocates this best practice to decide whether
it is a practice we want to endorse in either the DoD 5000 series or
the Guidebook.
RECOMMENDATION 4: The GAO recommended that the Secretary of Defense
direct the Under Secretary for AT&L, in collaboration with the
Assistant Secretary for NII and the Director, OT&E, to implement a
specific plan that, at a minimum communicates acquisition management
activities to all stakeholders. (p. 22/GAO Draft Report):
DOD RESPONSE: Partially concur. We find this to be another confusing
recommendation. The primary purposes of the DoD 5000 series and the
Acquisition Guidebook are to communicate acquisition management
activities to all stakeholders. The confusion may stem from the fact
that the report appears to use the terms "acquisition management" and
"project management" interchangeably. The DoD 5000 series directs
program managers and acquisition officials to conduct acquisition and
acquisition oversight through the integrated product team (IPT)
process. The essence of the IPT process is that acquisition and
oversight are conducted in a collaborative manner, thus facilitating
maximum communication among all stakeholders. In light of the above,
while we agree that communication of acquisition management activities
to all stakeholders is a best practice, we believe it already exists in
our directives system and our practices.
RECOMMENDATION 5: The GAO recommended that the Secretary of Defense
direct the Under Secretary for AT&L, in collaboration with the
Assistant Secretary for NII and the Director, OT&E to implement a
specific plan that, at a minimum includes acquisition reviews that
include the status of identified risks. (p. 22/GAO Draft Report):
DOD RESPONSE: Partially concur. We agree that reviewing the status of
identified risk is a best practice but disagree that it needs to be
added to the DoD 5000 series or the Guidebook. DoD Instruction 5000.2
has many references to various types of risks and the management or
mitigation of those risks. One of the primary purposes of the
acquisition phases in the DoD Acquisition Framework is to reduce risk.
For example, DoDI 5000.2 states that one of the entrance criteria for
the Systems Development and Demonstration phase is "the management and
mitigation of technology risk." We will review the 5000 series and the
Guidebook to detennine if there is a need to further emphasize this
best practice.
RECOMMENDATION 6: The GAO recommended that the Secretary of Defense
direct the Under Secretary for AT&L, in collaboration with the
Assistant Secretary for Nil and the Director, OT&E, to implement a
specific plan where at a minimum, modification of commercial components
is discouraged and allowed only if justified by a thorough analysis of
life-cycle costs and benefits. (p. 22/GAO Draft Report):
DOD RESPONSE: Concur. The current draft of the DoD Acquisition
Guidebook now under development discourages the modification of
commercial components. Additional changes to acquisition policy that
emphasize this policy will be considered.
RECOMMENDATION 7: The GAO recommended that the Secretary of Defense
direct the Under Secretary for AT&L, in collaboration with the
Assistant Secretary for NII and the Director, OT&E to implement a
specific plan where at a minimum, provide modification or upgrades to
deployed versions of system components are centrally controlled, and
unilateral user release changes are precluded. (p. 22/GAO Draft
Report):
DOD RESPONSE: Concur. This best practice will be added to the
Acquisition Guidebook.
RECOMMENDATION 8: The GAO recommended that the Secretary of Defense
direct the Under Secretary for AT&L, in collaboration with the
Assistant Secretary for NII and the Director, OT&E, to implement a
specific plan where, acquisition decisions about commercial components
are based on deliberate and thorough research, analysis, and
evaluation of the components' interdependencies. (p. 22/GAO Draft
Report):
DOD RESPONSE: Partially concur. The research, analysis and evaluation
of components' interdependencies is considered in the conduct of the
Joint Capabilities Integration and Development System (JCIDS) process
and analyzed in detail in a key acquisition system deliverable; i.e.,
the Information Support Plan. We will emphasize these analyses in the
Acquisition Guidebook.
RECOMMENDATION 9: The GAO recommended that the Secretary of Defense
direct the Under Secretary for AT&L, in collaboration with the
Assistant Secretary for NII and the Director, OT&E to implement a
specific plan that, at a minimum provide for incorporating: Acquisition
plans provide for preparing users for the impact that the business
processes embedded in the commercial components will have on their
respective roles and responsibilities. (p. 22/GAO Draft Report):
DOD RESPONSE: Partially concur. We agree that change management
activities such as those described in this recommendation are a best
practice but disagree that they need to be added to the DoD 5000
series. The El Toolkit mentioned previously contains a Change
Management roadmap that addresses organization change, readiness and
preparing the users. It also includes several samples of communications
to users, training aides and actual "getting familiar with your new
job" type of examples from actual programs. The El Toolkit is
referenced in DoDI 5000.2.
RECOMMENDATION 10: The GAO recommended that the Secretary of Defense
direct the Under Secretary for AT&L, in collaboration with the
Assistant Secretary for NII and the Director, OT&E to implement a
specific plan where, at a minimum changes affecting how users will be
expected to use the system to execute their jobs are actively managed.
(p. 22/GAO Draft Report):
DOD RESPONSE: Partially concur. Same explanation as for the previous
recommendation.
RECOMMENDATION 11: The GAO recommended that the Secretary of Defense
direct the Under Secretary for AT&L, in collaboration with the
Assistant Secretary for NII and the Director, OT&E, to implement a
specific plan where, systems integration contractors are explicitly
evaluated on their ability to implement commercial components. (p. 22/
GAO Draft Report):
DOD RESPONSE: Partially concur. The FAR and DFARS address the
evaluation of past performance, and the Department has published a
guide, titled A Guide to the Collection and Use of Past Performance
Information. In addition the FAR and DFARS prescribe some evaluation
factors while giving the contracting officer wide discretion in
determining other appropriate factors. The Department strives to state
its requirements in terms of the capabilities required; and commercial
items are only one means for providing those required capabilities.
However, acquiring commercial items is not an end in itself. Making the
ability to implement commercial components a mandatory evaluation
factor might have the unintended result of selecting a commercial
component over a better alternative. In light of the above, we believe
it is best to leave the selection of evaluation factors to the source
selection authority, who is the person best acquainted with the
particular circumstances.
Notwithstanding the above view, the Department has taken steps to
ensure that we have the opportunity to select from a group of qualified
contractors. The DoD Enterprise Software Initiative recently
established agreements with five leading systems integration firms to
provide commercial software integration services on a firm-fixed price
basis. These Blanket Purchase Agreements, which are available on the
GSA Schedule, encourage performance-based contracting and identify
particular performance incentives for each firm. The five integrators
were chosen based on an exhaustive 18-month market research and
evaluation period, using industry best practices guides - and
explicitly proven abilities to implement commercial software
solutions. Major DoD IT Program Managers are highly encouraged to use
these agreements.
RECOMMENDATION 12: The GAO recommended that the Secretary of Defense
direct the Under Secretary for AT&L, in collaboration with the
Assistant Secretary for NII and the Director, OT&E, to implement a
specific plan that, at a minimum provides for incorporating: Investment
decisions throughout a system's life cycle are based on a continuous
set of tradeoffs among capabilities available in commercial components
(current and future), the architectural environment in which the system
is to operate, defined system requirements, and existing cost/schedule
constraints. (p. 22/GAO Draft Report):
DOD RESPONSE: Partially concur. The JCIDS process described in Chairman
of the Joint Chiefs of Staff Instruction (CJCSI) 3170.0113 and the DoD
5000 series already require such analyses. CJCSI 3170.011) requires
that three analyses be conducted before a needed capability is
permitted to enter the technology development phase. These are a
functional area analysis, functional needs analysis and a functional
solutions analysis (FSA). These analyses and similar analyses required
by DoDI 5000.2 at subsequent investment decision points, require
tradeoffs among various available capabilities (including but not
limited to those available in commercial components) and consideration
of the architectural environment in which the system is to operate,
defined system requirements, and existing cost/schedule constraints.
For example, the FSA required at Milestone A is an analysis of
available alternatives. The Capabilities Development Document (CDD)
required at Milestone B addresses the architectural environment in
which the system is to operate and describes the system requirements.
The other Milestone B information requirements required by DoDI 5000.2,
such as the Economic Analysis and the Acquisition Program Baseline
address cost and schedule constraints. Tradeoff analyses are integral
to systems engineering, which is required by the DoD 5000 series. At
the direction of the Defense Acquisition Executive, more detailed
guidance on systems engineering will be added to the 5000 series and
the Guidebook. This recommendation will be considered when we develop
that guidance.
RECOMMENDATION 13: The GAO recommended that the Secretary of Defense
direct the Under Secretary for AT&L, in collaboration with the
Assistant Secretary for NII and the Director, OT&E, to implement a
specific plan where, evaluation criteria are established for selecting
among commercial component options that include both defined system
requirements and vendor/commercial products characteristics. (p. 22/
GAO Draft Report):
DOD RESPONSE: Partially concur. As stated in our response to
Recommendation 11, we are reluctant to limit the contracting officer's
discretion in choosing evaluation criteria. We are concerned that the
same unintended consequence cited in the response to Recommendation 11
could result from requiring the evaluation of commercial component
options. Moreover, as stated in the reply to Recommendation 13, each
acquisition is required to develop a CDD that describes system
requirements. However, we will further review the intent of this
recommendation, the 5000 series and the Guidebook to determine if there
is a need to further emphasize this best practice.
RECOMMENDATION 14: The GAO recommended that the Secretary of Defense
direct the Under Secretary for AT&L, in collaboration with the
Assistant Secretary for NII and the Director, OT&E, to implement a
specific plan that, at a minimum includes a provision for measurement
and verification of DOD's acquisition policies best practices. (p. 22/
GAO Draft Report):
DOD RESPONSE: Nonconcur. Auditing statutory and regulatory compliance
is the central focus of the oversight process described in DoDI 5000.2.
Review is accomplished in the context of the Working-level IPT process,
and non-compliance is corrected at each Milestone decision point. In
addition, the Department periodically reviews the entire acquisition
process to ensure that it is achieving desired outcomes and that it
continues to reflect sound business practice:
The following are GAO's comments on the Department of Defense's letter
dated July 1, 2004.
GAO Comments:
1. We disagree. While we have updated our report to reflect the
additional information provided in DOD's comments on the status of its
efforts and the associated milestone, the importance of revising and
maintaining DOD's acquisition policies and guidance, and their
incorporation of acquisition best practices, makes it essential to have
an explicit plan. Among other things, a plan would highlight the
resource constraints that this revision effort has been subject to,
would allow measurement against defined milestones, and would allow
disclosure of progress and impediments.
2. See comment 1.
3. We do not question DOD's statement that it has an initial repository
of custom software components that can help adapt commercial components
to defense use and reuse. However, this repository does not satisfy the
product line requirements in this best practice and our recommendation.
According to the Software Engineering Institute, the product line
requirements best practice involves more than just reuse. Under the
approach described in DOD's comments, reuse generally involves items,
such as software modules or components, that developers are encouraged
to use. However, the product line requirements best practice is not
simply encouraging reuse of items in a repository. Rather, it is
planned, enabled, and enforced reuse of such assets as requirements,
models, and architectures that have been designed and optimized for use
in multiple systems. In short, it is proactive rather than reactive
reuse.
4. We have modified our recommendation to use the terminology
"acquisition project management activities" instead of "acquisition
management activities" to eliminate any confusion. Further, we do not
question DOD's use of integrated product teams as a way to communicate
information. Further, the point of our recommendation is that there
needs to be an explicit recognition in policy and guidance of the type
of information to be communicated and to whom. Incorporating this best
practice is based on the need to ensure that communication vehicles,
such as integrated product teams, are effective.
5. We agree that the 5000 series contains information on acquisition
risk management, as we state in our report, and that one of the
purposes of DOD's acquisition framework is to reduce risks. However,
there is no provision in DOD's acquisition policy or guidelines to
ensure that the status of identified risks are discussed at key
decision points. For example, DOD policy states that a criterion for
passing milestone A is "management and mitigation of technology risk."
However, it does not provide for what is to be done to manage and
mitigate risks, and it does not provide for reviewing risk status at
milestones B or C.
6. While we agree that the toolkit provides relevant change management
information, the toolkit is referenced only once in the 5000 series;
and the reference is only in relation to one phase of the acquisition
cycle (the technology development phase). Given the importance and
relevance of this practice to the successful implementation of
commercial component-based systems, our position, and thus the basis
for our recommendation, is that the best practice's implementation
would more likely occur if it was visible and recognized in all
relevant stages of DOD's acquisition cycle.
7. See comment 6.
8. While the regulations and guidance that DOD cited and referenced in
its acquisition policies discuss the use of information on contractors'
past performance, they do not discuss evaluating systems integration
contractors on their ability to implement commercial components, which
is the point of the best practice. Further, DOD's objection to
incorporating this best practice is not consistent with its own
comments. Specifically, DOD commented that it has already taken steps,
through its enterprise systems initiative, to establish blanket
agreements with five contractors who were evaluated on, among other
things, "explicitly proven abilities to implement commercial systems
solutions." Additionally, while we appreciate DOD's concern that
incorporation of this best practice can have unintended consequences,
it is important to also recognize that our recommendation is not
intended to restrict a contracting officer's options. Rather, it is
intended to be one of the many factors considered in the source
selection process when it is relevant.
9. We support the Defense Acquisition Executive's decision to add more
specifics on systems engineering to the 5000 series, including
provisions that address this best practice and recommendation.
10. See comment 8.
11. We disagree. While we do not question that statutory and regulatory
compliance are referenced in DOD's integrated process team and
milestone decision point processes, we do not believe that these
reviews are adequately defined with respect to implementation of best
practices because DOD's policy does not require that the practices' use
be measured and verified. Rather, it leaves these reviews to the
discretion of the program manager and investment decision authority. As
we state in our report, not requiring that the use of best practices be
measured and verified increases the chance that the practices will not
be followed. Therefore, our position remains that DOD's policies do not
provide effective controls for ensuring that best practices are
appropriately followed.
[End of section]
Appendix IV: GAO Contact and Staff Acknowledgments:
GAO contact:
Carl L. Higginbotham, (404) 679-1824:
Staff acknowledgments:
In addition to the individual named above, key contributorsto this
report included Nabajyoti Barkakati, Nancy Glover,Madhav Panwar,
Morgan Walts, and Thomas Wright.
(310274):
FOOTNOTES
[1] Donald E. Harter, Mayuram S. Krishnan, and Sandra A. Slaughter,
"Effects of Process Maturity on Quality, Cycle Time, and Effort in
Software Product Development," Management Science, vol. 46, no. 4,
2000; and Bradford K. Clark, "Quantifying the Effects of Process
Improvement on Effort," IEEE Software (November/December 2000).
[2] U.S. General Accounting Office, Defense Acquisitions: Stronger
Management Practices Are Needed to Improve DOD's Software-Intensive
Weapon Acquisitions, GAO-04-393 (Washington, D.C.: Mar. 1, 2004).
[3] U.S. General Accounting Office, DOD Information Technology:
Software and Systems Process Improvement Programs Vary in Use of Best
Practices, GAO-01-116 (Washington, D.C.: Mar. 30, 2001).
[4] Bob Stump National Defense Authorization Act for 2003 (Pub.L. No.
107-314).
[5] Collectively, these oversight policy and guidance documents cover
most--but not all--major acquisitions. The Secretary of Defense has
delegated authority to the Missile Defense Agency and to the National
Security Space Team to develop separate guidance for missile defense
and space systems, respectively.
[6] DOD policy establishes a decision authority, called a milestone
decision authority, as the designated individual who has overall
responsibility for an IT investment. This person has the authority to
approve an investment's move from one phase to the next phase of the
acquisition process and is responsible for reporting cost, schedule,
and performance results to higher authorities, including the Congress.
[7] U.S. General Accounting Office, Information Technology:
Inconsistent Software Acquisition Processes at the Defense Logistics
Agency Increase Project Risks, GAO-02-9 (Washington, D.C.: Jan. 10,
2002); Information Technology: Greater Use of Best Practices Can Reduce
Risks in Acquiring Defense Health Care System, GAO-02-345 (Washington,
D.C.: Sept. 26, 2002); and DOD Business Systems Modernization:
Continued Investment in Key Accounting Systems Needs to be Justified,
GAO-03-465 (Washington, D.C.: Mar. 28, 2003).
[8] We defined "fully incorporate" to mean the 5000 series addressed
all of the practice's activities; "partially incorporate" to mean it
addressed some, but not all, of the activities; and "not incorporate"
to mean it did not address any of the activities.
[9] The Federal Managers' Financial Integrity Act of 1982 (Pub.L. No.
97-255); Government Performance and Results Act of 1993 (Pub.L. No.
103-62); Office of Management and Budget Circular A-123, June 21, 1995.
[10] The Defense Acquisition Policy Working Group is the standing DOD
acquisition policy working group that revised the 5000 series.
[11] U.S. General Accounting Office, Defense Acquisitions: DOD's
Revised Policy Emphasizes Best Practices, but More Controls Are Needed,
GAO-04-53 (Washington, D.C.: Nov. 10, 2003).
[12] For example, the Software Engineering Institute is a federally
funded research and development center operated by Carnegie Mellon
University and sponsored by DOD. The Software Engineering Institute's
objective is to provide leadership in software engineering and in the
transition of new software engineering technology into practice. The
Aerospace Corporation is a private, nonprofit organization that
operates a federally funded research and development center for DOD
that focuses on the government's need to develop space-related hardware
and software.
[13] Software Engineering Institute, Software Acquisition Capability
Maturity Model® version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA:
March 2002).
[14] U.S. General Accounting Office, Standards for Internal Control in
the Federal Government, AIMD-00-21.3.1 (Washington, D.C.: November
1999).
[15] Richard J. Adams, Suellen Eslinger, Karen L. Owens, and Mary A.
Rich, Reducing Risk in the Acquisition of Software-Intensive Systems:
Best Practices from the Space System Domain (Los Angeles, CA: 2003).
[16] Adams and Eslinger, "COTS-Based Systems: Lessons Learned from
Experiences with COTS Software Use on Space Systems." (Paper presented
to the Southern California SPIN: Oct. 6, 2003.)
[17] Software Engineering Institute, Real-Time Systems Engineering:
Lessons Learned from Independent Technical Assessments, CMU/SEI-2001-
TN-004 (Pittsburgh, PA: June 2001); and Adams, Eslinger, Owens, and
Rich, Reducing Risk in the Acquisition of Software Intensive-Systems:
Best Practices from the Space System Domain.
[18] Adams and Eslinger, "COTS-Based Systems: Lessons Learned from
Experiences with COTS Software Use on Space Systems." (Paper presented
to the Southern California SPIN: Oct. 6, 2003.)
[19] Donald J. Reifer, Victor R. Basili, Barry W. Boehm, Betsy Clark,
"COTS-Based Systems--Twelve Lessons Learned about Maintenance."
(Presentation, 3RD International Conference on COTS-Based Software
Systems, Redondo Beach, CA, Feb. 4, 2004.)
[20] Tricia Oberndorf, Lisa Brownsword, and Carol A. Sledge, Ph.D., An
Activity Framework for COTS-Based Systems, Technical Report CMU/SEI-
2000-TR-010 (Pittsburgh, Pa.: Software Engineering Institute, Carnegie
Mellon University, October 2000).
[21] Suzanne Garcia, John Roberts, and Len Estrin, "Managed Technology
Adoption Risk." (Presentation, 3RD International Conference on COTS-
Based Software Systems, Redondo Beach, CA, Feb. 2, 2004).
[22] Adams and Eslinger, "COTS-Based Systems: Lessons Learned from
Experiences with COTS Software Use on Space Systems." (Paper presented
to the Southern California SPIN: Oct. 6, 2003.)
[23] Software Engineering Institute, Evolutionary Process for
Integrating COTS-Based Systems (EPIC): An Overview, CMU/SEI-2002-TR-
009 (Pittsburgh, PA: July 2002).
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
