Information Technology
DOD Needs to Leverage Lessons Learned from Its Outsourcing Projects Gao ID: GAO-03-371 April 25, 2003Given the magnitude of its reported spending on information technology (IT) services--more than $6.2 billion in fiscal year 2001--it is critical that the Department of Defense (DOD) adopt effective practices for acquiring IT services. GAO researched leading commercial practices for the outsourcing of IT services, and, in November 2001, published a framework consisting of seven phases that span the full range of activities that are performed during the outsourcing of those services (this is an acquisition in which a client organization transfers responsibility for performing services to an external provider). GAO was asked to determine (1) the extent to which selected DOD projects for outsourcing IT services use leading commercial practices as specified in GAO's framework and (2) whether DOD is sharing lessons learned from its IT outsourcing projects across the department.
The projects in GAO's review substantially used leading commercial practices as specified in GAO's framework for outsourcing IT services. Specifically, the agencies fully implemented 88 percent of the practices (not including practices not applicable to a particular project). This framework consists of practices organized into seven phases: (I) determine sourcing strategy, (II) define operational model, (III) develop the contract, (IV) select the provider(s), (V) transition to provider(s), (VI) manage the performance of the provider(s), and (VII) ensure services are provided. Although DOD has acted on gathering and disseminating lessons learned and commercial leading practices related to general acquisition issues, its actions have generally not been focused on outsourcing or on sharing the lessons learned from IT services outsourcing across the department. By not systematically capturing and disseminating such information across the department, DOD is losing the opportunity to leverage the knowledge gained on IT services projects like those in GAO's review. Lessons learned that are pragmatic and easily accessible would give DOD managers a more informed understanding of important issues to be addressed when making outsourcing decisions, as well as the factors to be considered to help ensure the success of these endeavors.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-03-371, Information Technology: DOD Needs to Leverage Lessons Learned from Its Outsourcing Projects
This is the accessible text file for GAO report number GAO-03-371
entitled 'Information Technology: DOD Needs to Leverage Lessons Learned
from Its Outsourcing Projects' which was released on April 25, 2003.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
Report to the Subcommittee on Readiness and Management Support,
Committee on Armed Services,
U.S. Senate:
April 2003:
Information Technology:
DOD Needs to Leverage Lessons Learned from Its Outsourcing Projects:
GAO-03-371:
GAO Highlights:
Highlights of GAO-03-371, a report to the Subcommittee on Readiness and Management Support, Committee on Armed Services, U.S. Senate
Why GAO Did This Study:
Given the magnitude of its reported spending on information technology
(IT) services”more than $6.2 billion in fiscal year 2001”it is critical
that the Department of Defense (DOD) adopt effective practices for
acquiring IT services.
GAO researched leading commercial practices for the outsourcing of IT
services, and, in November 2001, published a framework consisting of
seven phases that span the full range of activities that are performed
during the outsourcing of those services (this is an acquisition in
which a client organization transfers responsibility for performing
services to an external provider).
GAO was asked to determine (1) the extent to which selected DOD
projects for outsourcing IT services use leading commercial practices
as specified in GAO‘s framework and (2) whether DOD is sharing lessons
learned from its IT outsourcing projects across the department.
What GAO Found:
The projects in GAO‘s review substantially used leading commercial
practices as specified in GAO‘s framework for outsourcing IT services.
Specifically, the agencies fully implemented 88 percent of the practices
(not including practices not applicable to a particular project). This
framework consists of practices organized into seven phases: (I)
determine sourcing strategy, (II) define operational model, (III)
develop the contract, (IV) select the provider(s), (V) transition to
provider(s), (VI) manage the performance of the provider(s), and (VII)
ensure services are provided. The figure below shows the percentage of
practices that were implemented in each phase.
Although DOD has acted on gathering and disseminating lessons learned
and commercial leading practices related to general acquisition issues,
its actions have generally not been focused on outsourcing or on
sharing the lessons learned from IT services outsourcing across the
department. By not systematically capturing and disseminating such
information across the department, DOD is losing the opportunity to
leverage the knowledge gained on IT services projects like those in
GAO‘s review. Lessons learned that are pragmatic and easily accessible
would give DOD managers a more informed understanding of important
issues to be addressed when making outsourcing decisions, as well as
the factors to be considered to help ensure the success of these
endeavors.
What GAO Recommends:
GAO is making recommendations to the Secretary of Defense aimed at
leveraging lessons learned across the department from its components‘
IT outsourcing experiences.
DOD agreed that capturing lessons learned related to IT outsourcing
initiatives is important and stated that it intends to explore a
variety of mechanisms to do so. DOD‘s plans are consistent with our
recommendations.
www.gao.gov/cgi-bin/getrpt?GAO-03-371.
To view the full report, including the scope
and methodology, click on the link above.
For more information, contact Randolph C. Hite at (202) 512-3439 or
hiter@gao.gov.
[End of section]:
Letter:
Results in Brief:
Background:
Projects Substantially Used Leading Commercial Practices:
Leveraging Lessons Learned DOD-wide
Could Assist Other DOD Projects:
Conclusions:
Recommendations:
Agency Comments and Our Evaluation:
Appendixes:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Projects‘ Implementation of Phase I: Determine Sourcing
Strategy:
Appendix III: Project‘s Implementation of Phase II: Define Operational
Model:
Appendix IV: Project‘s Implementation of Phase III: Develop the
Contract:
Appendix V: Projects‘ Implementation of Phase IV: Select the
Provider(s):
Appendix VI: Projects‘ Implementation of Phase V: Transition to
Provider(s):
Appendix VII: Projects‘ Implementation of Phase VI: Manage Provider(s)
Performance:
Appendix VIII: Projects‘ Implementation of Phase VII: Ensure Services
are Provided:
Appendix IX: Comments from the Department of Defense:
GAO Comments:
Appendix X: GAO Contact and Staff Acknowledgments:
GAO Contact:
Staff Acknowledgments:
Tables:
Table 1: Definition of Phases for IT Outsourcing:
Table 2: Profile of Outsourcing Projects Reviewed:
Table 3: Percentage of Practices Implemented, by Project:
Table 4: Summary of Projects' Use of Phase I Practices:
Table 5: Summary of Projects' Use of Phase II Practices:
Table 6: Summary of Projects' Use of Phase III Practices:
Table 7: Summary of Projects' Use of Phase IV Practices:
Table 8: Summary of Projects' Use of Phase V Practices:
Table 9: Summary of Projects' Use of Phase VI Practices:
Table 10: Summary of Projects' Use of Phase VII Practices:
Figures:
Figure 1: Percentage of Practices Implemented in Each Phase:
Figure 2: Roles of the Client and Provider in an Outsourcing
Relationship:
Figure 3: GAO's Framework for Outsourcing IT Services:
Figure 4: Use of Leading Commercial Practices for Outsourcing of IT
Services, by Project:
Figure 5: Percentage of Practices Implemented in Each Phase:
Figure 6: Project Implementation of Phase I: Determine Sourcing
Strategy:
Figure 7: Project Implementation of Phase II: Define Operational Model:
Figure 8: Project Implementation of Phase III: Develop the Contract:
Figure 9: Project Implementation of Phase IV: Select the Provider(s):
Figure 10: Project Implementation of Phase V: Transition to Provider(s):
Figure 11: Project Implementation of Phase VI: Managing Provider
Performance:
Figure 12: Project Implementation of Phase VII: Ensure Services Are
Provided:
Abbreviations:
C3I: command, control, communications, and intelligence:
C4: command, control, communications, and computer :
CIO: chief information officer:
DOD: Department of Defense:
IT: information technology:
IT/IS: Information Technology/Information Services:
MHS/ITO: Military Health System/Information Technology Organization:
NETCOM: Network Enterprise Technology Command:
NIMA: National Imagery and Mapping Agency:
NMCI: Navy and Marine Corps Intranet: :
OMB: Office of Management and Budget:
RFP: request for proposals:
SLA: service-level agreement:
TAC-SWA: Total Army Communications--Southwest Asia:
This is a work of the U.S. Government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. It may contain
copyrighted graphics, images or other materials. Permission from the
copyright holder may be necessary should you wish to reproduce
copyrighted materials separately from GAO‘s product.
Letter April 25, 2003:
The Honorable John Ensign
Chairman
The Honorable Daniel K. Akaka
Ranking Minority Member
Subcommittee on Readiness and Management Support
Committee on Armed Services
United States Senate:
The Department of Defense (DOD) is the government's largest purchaser
of information technology (IT) services, such as desktop support,
network operations, and software development services. In fiscal year
2001, DOD reportedly obligated more than $6.2 billion on IT
services,[Footnote 1] and this amount is expected to grow
substantially. Given the magnitude of DOD's spending on such services,
it is critical that the department adopt effective practices for
acquiring IT services.
Since 1996, we have conducted a series of studies for the Senate
Committee on Armed Services concerning how DOD can improve its
acquisition processes by adopting proven practices of leading
commercial organizations. In this vein, in November 2001, we issued a
guide that organized leading commercial practices for the
outsourcing[Footnote 2] of IT services into a framework of seven phases
that span the full range of activities that are performed during IT
services outsourcing.[Footnote 3]
This report responds to your request that we determine (1) the extent
to which selected DOD IT services outsourcing projects use leading
commercial practices as specified in our framework and
(2) whether DOD is sharing lessons learned from its IT outsourcing
projects across the department. To address the first objective, we
selected five projects from a group of projects to outsource IT
services that were
identified by the military services and other DOD components.[Footnote
4] We then asked the component responsible for each project to perform
a self-assessment against selected practices in our framework for
outsourcing IT services.[Footnote 5] Next, we obtained and reviewed
agencies' supporting documentation and interviewed the appropriate
agency and provider project officials to independently determine
whether a practice was met. To address the second objective, we
reviewed applicable DOD approaches for capturing and disseminating
lessons learned from IT services outsourcing projects and interviewed
the applicable acquisition and IT officials. Details of our objectives,
scope, and methodology are discussed in appendix I.
Results in Brief:
The projects in our review substantially used leading commercial
practices as specified in our framework on outsourcing IT services.
Specifically, the agencies fully implemented 88 percent of the
practices.[Footnote 6] This framework consists of practices organized
into seven phases that span the full range of activities that are
performed during IT outsourcing: (I) determine sourcing strategy, (II)
define operational model, (III) develop the contract, (IV) select the
provider(s), (V) transition to provider(s), (VI) manage provider(s)
performance, and (VII) ensure services are provided. Figure 1
illustrates the percentage of practices that were followed in each
phase. Collectively, the projects fully implemented from 70 to 97
percent of the practices in each phase.
Figure 1: Percentage of Practices Implemented in Each Phase:
[See PDF for image]
Note: Not applicable--The practice was not relevant to the project's
particular circumstances.
No--The agency did not implement the practice. Limited--The agency
fully implemented some but not all aspects of the practice and did not
take alternative actions that fully satisfied the practice. Yes--The
agency fully implemented the practice or took an alternative action
that fully satisfied its intent.
[End of figure]
Although implementing the leading commercial practices in our framework
does not guarantee the success of an outsourcing project, the consensus
view of the leading commercial activities that we studied is that these
practices are the most critical to success when acquiring IT
services.[Footnote 7] In addition, not implementing or only partially
implementing particular practices can produce negative consequences or
add risk to a project. For example, the Department of the Navy
project's baseline of its existing environment (a phase I practice) was
limited because it did not include an assessment of its legacy
applications. Instead, project officials decided to rely on a
preexisting inventory developed to address the Year 2000 challenge. The
Navy subsequently found that it had substantially underestimated the
number of legacy applications, which, according to program officials,
later contributed to the transition period slipping from
2-˝ years to 3-˝ years.
As DOD gathers more experience in implementing projects for outsourcing
IT services, it can benefit from leveraging the lessons derived from
these initiatives. For example, the projects in our review have
identified lessons learned in such areas as transitioning to the
provider and partnering with the provider. Although DOD has taken
action to gather and disseminate lessons learned and best practices on
general acquisition issues, these efforts generally do not focus on
outsourcing or include sharing the lessons learned from IT outsourcing
projects across the department. By not capturing and disseminating such
information in a systematic manner across the department, DOD is losing
the opportunity to leverage the knowledge gained on IT services
projects like those in our review.
We are making recommendations to the Secretary of Defense aimed at
leveraging lessons learned across the department from its components'
IT services outsourcing experiences.
In written comments on a draft of this report signed by the DOD
Principal Director, Deputy Assistant Secretary of Defense (Deputy Chief
Information Officer), DOD agreed that capturing lessons learned in the
development and implementation of its IT outsourcing initiatives is
important to continually improving its outsourcing methods and results
achieved. The department also stated that it intends to explore a
variety of mechanisms for best exploiting lessons learned from its IT
outsourcing initiatives. We agree that it is prudent to consider
alternative means to leveraging these lessons learned, and we believe
that this is consistent with our recommendations.
Background:
To protect the security of the United States, DOD relies on a complex
array of computer-dependent and mutually supportive organizational
components, including the military services, Commanders in Chief, and
Defense agencies. As such, it invests tens of billions of dollars each
year in a broad array of computer systems, which include weapon
systems, command and control systems, satellite systems, inventory
management systems, transportation management systems, health systems,
financial systems, personnel systems, and payment systems. In addition,
DOD spends billions of dollars annually on IT services, which include
database management, help-desk operations, software maintenance, and
network services. In fiscal year 2001, DOD reportedly obligated more
than $6.2 billion on IT services alone.[Footnote 8]
Decisions regarding the purchasing of services are critical to ensuring
the effectiveness of DOD's operations as well as those of the
government as a whole. Our November 2001 report recognizes the
importance of such sourcing decisions and provides a framework that
spans the full range of activities that are performed during IT
services outsourcing.[Footnote 9] At the same time, governmentwide
policies, initiatives, and challenges exist that significantly
influence the government's sourcing decisions.
GAO's Framework for Outsourcing IT Services:
Outsourcing of IT services has become increasingly popular in both the
public and private sector. For example, according to the Giga
Information Group, Inc., a leading research firm, such outsourcing is
expected to grow an average of 5 to 6 percent in 2003.[Footnote 10] The
federal sector's outsourcing is predicted to rise at an even greater
rate. For example, INPUT, an IT market research firm, forecasts that
defense IT outsourcing will increase about 143 percent between fiscal
years 2002 and 2007.[Footnote 11]
IT outsourcing involves the activities associated with acquiring
services from one or more external providers. During outsourcing, a
client organization transfers responsibilities for performing one or
more IT services to one or more external providers. This responsibility
is executed through control and management of the processes, people,
and technology associated with these services.
Figure 2 depicts the roles of the client and provider organizations in
an outsourcing relationship.
Figure 2: Roles of the Client and Provider in an Outsourcing
Relationship:
[See PDF for image]
[End of figure]
Our November 2001 guide on leading commercial practices for outsourcing
IT services provides a generic framework of practices from leading
commercial organizations that can improve purchasing decisions and
manage the resulting government/provider relationship.[Footnote 12]
The framework is represented in figure 3 as a hierarchy of phases,
practices, and critical success factors.
Figure 3: GAO's Framework for Outsourcing IT Services:
[See PDF for image]
Note: The arrow from phase VII to phase I represents the need to
reflect on lessons learned from previous phases. The arrows between
phase III and IV represent the iterative nature of developing the
contract and selecting the provider. Although there is a logical order
to the sequence of the common phases, the order of the practices within
each phase does not imply any priority or sequence.
[End of figure]
Table 1 provides a definition of each phase of the framework. Each of
the phases has specific practices associated with it. Implementing
these practices does not guarantee the success of an outsourcing
project. However, our November 2001 study reflected a consensus view
that these practices were the most critical to success when IT services
are being acquired.[Footnote 13] Restated, application of these
practices increases the probability of a successful outsourcing
project.
Table 1: Definition of Phases for IT Outsourcing:
Phase number: I; Title: Determine sourcing strategy; Definition: Client
organizations determine whether internal capability or external
expertise can more effectively meet IT needs.
Phase number: II; Title: Define operational model; Definition: Client
organizations formalize executive leadership, team composition, client
responsibilities, and operating relationships between client and
provider organizations.
Phase number: III; Title: Develop the contract; Definition: Client
organizations establish the legal terms for the IT outsourcing
relationship.
Phase number: IV; Title: Select the provider(s); Definition: Client
organizations find one or more providers who can help them reach their
IT outsourcing goals.
Phase number: V; Title: Transition to provider(s); Definition: Client
organizations transfer responsibility of IT functions to one or more
providers.
Phase number: VI; Title: Manage provider(s) performance; Definition:
Client organizations make sure each provider is meeting performance
requirements.
Phase number: VII; Title: Ensure services are provided; Definition:
Client organizations make sure that services are provided and end-user
needs are met.
Source: GAO.
[End of table]
The organizations that we studied also identified certain capabilities
(identified as critical success factors) that were essential for
implementing the practices identified in our framework. First,
executive leadership strengthens the interaction between executive
management and the employees of the client organization. Second,
partner alignment strengthens the interaction between the client and
provider organization at the executive level, which ensures that the
goals and objectives of these organizations support each other. Third,
relationship management strengthens the interaction between the client
and provider organization at the operational level.
Influences on Government Sourcing Decisions:
The federal government is one of the world's largest users of services.
Because of the large dollar value and the number of private-and public-
sector jobs involved, deciding whether the public or private sector
would be the most appropriate provider of the services the government
needs (IT or otherwise) is an important, and often highly charged,
question. Among the factors that agencies must consider as they
determine how best to meet their missions is whether the public or
private sector would be the most appropriate provider of the services
the government needs. Phase I of our framework, determine the sourcing
strategy, addresses the client's assessment of whether expertise from
within or outside of the organization can more effectively meet the
client's needs.
The National Defense Authorization Act for Fiscal Year 2001 required
the Comptroller General of the United States to convene a panel of
experts to study the current process used by the government to make
sourcing decisions. The resulting Commercial Activities Panel conducted
a year-long study and heard repeatedly about the importance of
competition and its central role in fostering economy, efficiency, and
continuous performance improvement. In particular, the panel reviewed
the government's implementation of the Office of Management and
Budget's (OMB) Circular A-76, which sets forth federal policy for
determining whether federal employees or private contractors will
perform commercial activities for the government.[Footnote 14] Circular
A-76 (1) outlines conditions under which agencies are permitted to
perform a commercial activity with government employees or by contract
and (2) provides guidance for whether, and if so, how, agencies should
conduct a cost comparison when they are considering transferring the
performance of commercial activities from the public to the private
sector (or vice versa). The panel reported that there were positive
elements to Circular A-76 but that both federal employees and private
firms complained that it does not meet the standard of a clear,
transparent, and consistently applied process. For example, both
federal employees and private firms criticized the Circular A-76
process as unequal and therefore unfair.
The Commercial Activities Panel strongly supported continued emphasis
on competition and concluded that whenever the government is
considering converting work from one sector to another, public/private
competitions should be the norm. In addition, the panel made four
recommendations, including that all sourcing decisions be consistent
with the principles adopted unanimously by the panel, such as the
principle that federal policy provide for accountability in connection
with all sourcing decisions.[Footnote 15]
As part of the administration's efforts to implement the
recommendations of the Commercial Activities Panel, OMB has published
proposed changes to Circular A-76. Key highlights of the proposed
changes include presuming that all functions are commercial in nature
unless they are justified as inherently governmental;[Footnote 16]
limiting the length of time for competitions; and emphasizing awarding
contracts on the basis of best value, not just lowest cost. Best value
allows the contracting official to consider technical superiority,
quality, innovation, and past performance as well as price.
However, we reported that there are several areas in which the proposed
revisions to the circular are not consistent with the principles or
recommendations of the Commercial Activities Panel.[Footnote 17]
Specifically, the proposed revision does not include a link between
sourcing policy and agency missions, has unnecessarily complicated
source selection procedures, contains certain unrealistic time frames,
and includes insufficient guidance on calculating savings.
Beyond the Commercial Activities Panel, other bodies have identified
challenges that the federal government faces in reaching and executing
effective sourcing decisions. For example, members of the Coalition for
Government Procurement, the Professional Services Council, and the
Information Technology Association of America told us that
organizational culture is one of the biggest differences between the
commercial sector and the federal government and one of the greatest
barriers to the government's use of commercial practices. Also, as we
have previously reported, moving to outsourcing solutions can involve a
cultural change for government organizations because it may require a
change to an agency's operating model, such as using a contractor to
provide IT services previously performed by government staff or using a
performance-based contract.[Footnote 18] This view was echoed by a 2001
study of DOD competitive
sourcing that found cultural, process, execution, and training
barriers.[Footnote 19] The study stated that these barriers need to be
understood and mitigated before the benefits of outsourcing can be
fully realized. Barriers such as these can be overcome by strong
executive leadership, which is a critical success factor in our
framework.
Another challenge is creating a productive agency/provider
relationship--another critical success factor in our framework.
According to a report sponsored by the PricewaterhouseCoopers Endowment
for the Business of Government, such public/private partnerships are
based on trust, commitment to problem or conflict resolution, and the
recognition that flexibility is necessary and that the relationship
will evolve and change over time.[Footnote 20] If deadlines are not
met, or public agency goals change with differing political climates,
the partners need to discuss the basis of the partnership and construct
a different relationship. Our prior report on desktop outsourcing found
that developing a productive agency/contractor relationship is not
always easy.[Footnote 21] Both sides must recognize and understand each
other's underlying motives and strive to achieve established
expectations.
Finally, human capital issues are another challenge facing federal
agencies that affect their ability to implement outsourcing. Our
framework recognizes the importance of having the right skills in place
to support the outsourcing relationship. However, as we have previously
reported, procurement reforms and technological changes have placed
unprecedented demands on the acquisition workforce.[Footnote 22]
Contracting personnel are now expected to have a much greater knowledge
of market conditions, industry trends, and the technical details of the
commodities and services they procure. The Commercial Activities Panel
report stated that developing and maintaining a skilled acquisition
workforce is the critical first step in managing this more complex
procurement environment. The panel also reported that DOD bore the
brunt of a 22 percent downsizing of the federal acquisition workforce
in the last decade, going from 96,000 staff in 1991 to about 68,000 in
fiscal year 2001.
Addressing human capital issues is not just a matter of the size of the
workforce; it is also a knowledge and skills issue. According to the
Commercial Activities Panel, it is critically important that federal
agencies adequately address human capital needs in meeting the current
and emerging needs of government and its citizens in the most
effective, efficient, and economical manner possible. This will require
increased emphasis on training and development, particularly in the
area of technology.
Description of Five Projects Reviewed:
The five projects in our study varied in how they approached
outsourcing IT services, such as in using various solicitation methods,
including holding a public/private competition under the policies
outlined in OMB Circular A-76 or carrying out a negotiated competitive
procurement. In addition, the types of services being outsourced
differed: services ranged from the narrowly focused (e.g., help-desk
services) to the very broad (e.g., enterprisewide end-to-end
information services); contract terms ranged from 5 to 15 years
(assuming all option years are exercised); and estimated contract
values ranged from $23 million to $8.8 billion. Table 2 provides
information on the variety of IT services and outsourcing approaches
taken by the projects.
:
Table 2: Profile of Outsourcing Projects Reviewed:
Agency/Project: Air Force/Kirtland Air Force Base's Command, Control,
Communications, and Computer (C4) Services; Solicitation method:
Competitive, under OMB Circular A-76 and small business set-aside
rules; Date of contract award: April 2000; Contract term: 1 year, with
4 option years; Contract type: Firm, fixed-price; Estimated total
contract value[A]: $23 million; Project description: Management,
operations, and maintenance of command, control, communications, and
computer systems, multimedia services, and information management for
the Kirtland Air Force Base.
Agency/Project: Army/Network Enterprise Technology Command's (NETCOM)
Total Army Communications--Southwest Asia (TAC-SWA); Solicitation
method: Negotiated competitive solicitation; Date of contract award:
March 2001; Contract term: 1 year, with 4 option years; Contract type:
Firm, fixed-price; Estimated total contract value[A]: $204 million;
Project description: Operation and maintenance services, including
repair, installation, and supply, for communications equipment in
Southwest Asia.
Agency/Project: Military Health System/Information Technology
Organization (MHS/ITO) Help Desk; Solicitation method: Negotiated
competitive solicitation; Date of contract award: June 2001; Contract
term: 1 year, with 7 option years; Contract type: Firm, fixed-price
with incentive awards; Estimated total contract value[A]: $71 million;
Project description: Call and help-desk services for all MHS software
applications.
Agency/Project: Department of the Navy/Navy and Marine Corps Intranet
(NMCI); Solicitation method: Negotiated competitive solicitation; Date
of contract award: October 2000; Contract term: 7 years, with an option
for 3 additional years; Contract type: Firm, fixed-price with incentive
awards; Estimated total contract value[A]: $8.8 billion; Project
description: Department of the Navy-wide end-to-end information
services through a common computing and communication environment.
Agency/Project: National Imagery and Mapping Agency's (NIMA)
Information Technology/ Information Services (IT/IS); Solicitation
method: Sole-source, using a statutory preferential provider (Alaska
native corporation)[B]; Date of contract award: December 2001; Contract
term: 1 year, with 14 option years; Contract type: Cost plus award
fees; Estimated total contract value[A]: $2.1 billion; Project
description: NIMA-wide IT/IS support services for printing, digital
replication, networks, distributed and centralized systems and
services, video and voice communications, information research, and
help-desk operations.[C].
Source: DOD.
[A] Estimated value if all option years are exercised.
[B] NIMA performed an OMB Circular A-76 analysis and, on the basis of
this analysis, implemented a direct conversion to a preferential
provider rather than holding a public/private competition or obtaining
an agency cost-comparison waiver.
[C] As of February 24, 2003, NIMA had transitioned four of these
functions to the provider: printing, digital replication, video and
voice communications, and help-desk operations.
[End of table]
Projects Substantially Used Leading Commercial Practices:
As illustrated in figure 4, the five IT services projects substantially
used leading commercial practices. Specifically, each project used at
least 76 percent of the practices.[Footnote 23] Reasons for projects
implementing different percentages of the practices include differences
in their individual circumstances and objectives. For example, the
Army's Total Army Communications--Southwest Asia (TAC-SWA) project,
which used the fewest number of practices and had the largest number of
practices that were not applicable, was largely a continuation of an
existing approach that already relied on the private sector but with
fewer providers. In contrast, the National Imagery and Mapping Agency's
(NIMA) Information Technology/ Information Services (IT/IS) project,
which fully or partially implemented all of the applicable practices,
involved a significant operational shift (e.g., functions previously
performed by NIMA staff are now performed by a contractor) and was
intended to result in substantial process improvements. In addition,
the three projects that implemented the largest percentage of practices
also used third-party assistance--including employing a contractor with
sourcing expertise--to help formulate their sourcing strategy, which
could account for the extent of their compliance.
Figure 4: Use of Leading Commercial Practices for Outsourcing of IT
Services, by Project:
[See PDF for image]
Note: Not applicable--The practice was not relevant to the project's
particular circumstances.
No--The agency did not implement the practice. Limited--The agency
fully implemented some but not all aspects of the practice and did not
take alternative actions that fully satisfied the practice. Yes--The
agency fully implemented the practice or took an alternative action
that fully satisfied its intent.
[End of figure]
In addition, figure 5 illustrates that project compliance extended to
each of the phases of our framework. Collectively, the projects fully
implemented from 70 to 97 percent of the practices in each phase. Phase
I, determine sourcing strategy, had the lowest percentage of practices
implemented by the projects (70 percent). This result is not
inconsistent with a recent Giga Information Group, Inc., survey, which
found that only half of the
respondent organizations had documented an IT sourcing
strategy.[Footnote 24] This approach carries risk since phase I sets
the tone for the outsourcing initiative within the client organization.
Figure 5: Percentage of Practices Implemented in Each Phase:
[See PDF for image]
Note: Not applicable--The practice was not relevant to the project's
particular circumstances.
No--The agency did not implement the practice. Limited--The agency
fully implemented some but not all aspects of the practice and did not
take alternative actions that fully satisfied the practice. Yes--The
agency fully implemented the practice or took an alternative action
that fully satisfied its intent.
[End of figure]
In addition, collectively the projects fully implemented 88 percent of
the practices (see table 3).[Footnote 25]
Table 3: Percentage of Practices Implemented, by Project:
Phase: Phase I: Determine sourcing strategy (6 practices).; Percentage
of practices implemented[A]: Air Force
C4 Services project: 50; Percentage of practices implemented[A]: Army
TAC-SWA project: 33; Percentage of practices implemented[A]: MHS/ITO
Help Desk project: 83; Percentage of practices implemented[A]: Navy
NMCI project: 83; Percentage of practices implemented[A]: NIMA IT/IS
project: 100; Percentage of practices implemented[A]: Overall: 70.
Phase: Phase II: Define operational model (13 practices).; Percentage
of practices implemented[A]: Air Force
C4 Services project: 92; Percentage of practices implemented[A]: Army
TAC-SWA project: 83; Percentage of practices implemented[A]: MHS/ITO
Help Desk project: 100; Percentage of practices implemented[A]: Navy
NMCI project: 77; Percentage of practices implemented[A]: NIMA IT/IS
project: 100; Percentage of practices implemented[A]: Overall: 90.
Phase: Phase III: Develop the contract (16 practices).; Percentage of
practices implemented[A]: Air Force
C4 Services project: 64; Percentage of practices implemented[A]: Army
TAC-SWA project: 83; Percentage of practices implemented[A]: MHS/ITO
Help Desk project: 100; Percentage of practices implemented[A]: Navy
NMCI project: 94; Percentage of practices implemented[A]: NIMA IT/IS
project: 100; Percentage of practices implemented[A]: Overall: 89.
Phase: Phase IV: Select provider(s) (7 practices).; Percentage of
practices implemented[A]: Air Force
C4 Services project: 86; Percentage of practices implemented[A]: Army
TAC-SWA project: 86; Percentage of practices implemented[A]: MHS/ITO
Help Desk project: 100; Percentage of practices implemented[A]: Navy
NMCI project: 100; Percentage of practices implemented[A]: NIMA IT/IS
project: 71; Percentage of practices implemented[A]: Overall: 89.
Phase: Phase V: Transition to provider(s) (11 practices).; Percentage
of practices implemented[A]: Air Force
C4 Services project: 82; Percentage of practices implemented[A]: Army
TAC-SWA project: 88; Percentage of practices implemented[A]: MHS/ITO
Help Desk project: 100; Percentage of practices implemented[A]: Navy
NMCI project: 82; Percentage of practices implemented[A]: NIMA IT/IS
project: 100; Percentage of practices implemented[A]: Overall: 90.
Phase: Phase VI: Manage provider(s) performance (11 practices).;
Percentage of practices implemented[A]: Air Force
C4 Services project: 73; Percentage of practices implemented[A]: Army
TAC-SWA project: 73; Percentage of practices implemented[A]: MHS/ITO
Help Desk project: 100; Percentage of practices implemented[A]: Navy
NMCI project: 91; Percentage of practices implemented[A]: NIMA IT/IS
project: 82; Percentage of practices implemented[A]: Overall: 84.
Phase: Phase VII: Ensure services are provided (6 practices).;
Percentage of practices implemented[A]: Air Force
C4 Services project: 83; Percentage of practices implemented[A]: Army
TAC-SWA project: 100; Percentage of practices implemented[A]: MHS/ITO
Help Desk project: 100; Percentage of practices implemented[A]: Navy
NMCI project: 100; Percentage of practices implemented[A]: NIMA IT/IS
project: 100; Percentage of practices implemented[A]: Overall: 97.
Phase: Overall; Percentage of practices implemented[A]: Air Force
C4 Services project: 76; Percentage of practices implemented[A]: Army
TAC-SWA project: 79; Percentage of practices implemented[A]: MHS/ITO
Help Desk project: 99; Percentage of practices implemented[A]: Navy
NMCI project: 89; Percentage of practices implemented[A]: NIMA IT/IS
project: 94; Percentage of practices implemented[A]: Overall: 88.
Source: GAO.
[A] These calculations do not include practices that were not
applicable to a particular project.
[End of table]
The following provides additional information on the projects'
implementation of each phase of our framework.
* Phase I: Determine sourcing strategy. In the first phase of our
outsourcing framework, the client organization determines whether
internal capability or external expertise can more effectively meet its
IT needs. The purpose of a sourcing strategy is to achieve the optimal
balance between internal and external capabilities, activities,
processes, and services to ensure the achievement of strategic business
objectives at the lowest risk.[Footnote 26] The five projects'
implementation of this phase was uneven. In particular, two of the six
practices in this phase were fully implemented by all five projects,
but the other four practices were not. Among the practices that were
implemented by all of the projects was determining the business reasons
for outsourcing. In addition, the three projects that implemented the
largest percentage of practices in our framework--the Military Health
System/Information Technology Organization (MHS/ITO) Help Desk, the
Department of the Navy's Navy and Marine Corps Intranet (NMCI), and
NIMA IT/IS projects--used the third-party assistance practice in this
phase to help formulate their sourcing strategy, which could account
for the extent of their compliance. For example, the MHS/ITO Help Desk
project, which implemented the largest percentage of practices, worked
with the Department of the Interior's GovWorks Program,[Footnote 27]
the Defense Acquisition University staff, and a private-sector
contractor to obtain expertise on sourcing strategies.
The practice in this phase that was the most unevenly implemented was
the benchmarking[Footnote 28] and baselining of existing internal
services. Of the five projects in our review, (1) one fully benchmarked
and baselined the productivity of the activity being outsourced before
making the final sourcing decision, (2) two partially baselined their
existing activities, and (3) two did not perform benchmark and baseline
analyses at all. The agencies' reasons for not fully implementing this
practice included that an executive decision had been made to conduct a
public/private competition following the OMB Circular A-76 policy, so
such an analysis would not have affected the sourcing decision, or that
available documentation to perform such an analysis was limited.
Leading research firms suggest benchmarking and baselining the entity's
current processes before outsourcing because only then would it be able
to determine whether the arrangement has been successful.[Footnote 29]
In addition, the risk of not fully baselining the existing environment
is illustrated by the NMCI project. Specifically, the NMCI project's
baseline of its existing environment was limited because it did not
include an assessment of its legacy applications since project
officials decided to rely on a preexisting inventory developed to
address the Year 2000 challenge. The Navy subsequently found that it
had substantially underestimated its number of legacy applications,
that, according to program officials, later contributed to the
transition period slipping from 2-˝ years to 3-˝ years. Appendix II
provides additional information on projects' implementation of the
practices comprising this phase.
* Phase II: Define operational model. The operational model is an
important mechanism for an organization to compare its plans with the
expectations that were set when the decision to outsource was made and
to ascertain whether these plans will enable the organization to meet
expectations. The five projects had largely implemented the 13
practices contained in this phase. Specifically, about 90
percent[Footnote 30] of the practices were implemented. For example,
all projects implemented the practice that executive leadership be
established to facilitate the outsourcing effort. NIMA, for instance,
formed a strategic sourcing office to oversee the IT/IS project.
Another practice--training the provider on the organization's business
environment and goals--was fully implemented by one project (in two
cases, the practice was not applicable). One project that did not fully
implement this practice was the Air Force's Kirtland Air Force Base's
Command, Control, Communications, and Computer (C4) Services project.
Although some training was provided (e.g., Kirtland held an orientation
session for potential bidders), provider officials stated that they did
not receive adequate training, which made the transition period more
difficult. Appendix III provides additional information on projects'
implementation of the practices comprising this phase.
* Phase III: Develop the contract. A well-written contract is necessary
for the outsourcing organization to meet its requirements while
allowing the service provider to make a fair profit. It sets the
expectations for service levels, delivery of essential services, and
continuous improvement and should protect the interests of all parties.
The five projects largely implemented the practices in this phase.
Specifically, about 89 percent[Footnote 31] of the practices were
implemented in this phase, and two projects (the MHS/ITO Help Desk and
NIMA IT/IS projects) implemented all of the practices. Several
practices in this phase address performance requirements. For example,
all five projects implemented the practices that called for basing
performance requirements on business requirements and reviewing and
updating them periodically. One practice that was not fully implemented
by two projects was including performance measures that address both
technical and end-user satisfaction aspects of performance. For
example, the Army TAC-SWA project included technical performance
measures in its contract but not measures related to end-user
satisfaction, even though the contract included help-desk services.
According to the project official that developed the performance work
statement in the contract, the command did not include customer
satisfaction measures because it did not think that it was necessary to
have a performance standard for the help-desk service. However, without
such measures, the agency does not have a contractual standard with
which to judge the provider's performance. Appendix IV provides
additional information on projects' implementation of the practices
comprising this phase.
* Phase IV: Select the provider(s). Critical to the success of any
outsourcing project for IT services is identifying potential providers
and ultimately selecting a provider(s) that will best meet the needs of
the agency. The five projects had largely implemented the seven
practices contained in this phase. About 89 percent of the practices
were implemented in this phase and two projects (the MHS/ITO Help Desk
and Navy NMCI projects) used each applicable practice. For example, the
five projects implemented the practice related to conducting due
diligence activities to verify provider capabilities before signing the
contract. In the case of the Army TAC-SWA project, the Network
Enterprise Technology Command (NETCOM) evaluated the provider's
financial and past performance information. The Department of the
Navy's NMCI project also evaluated bidders' past performance and
performed reference checks. Moreover, as part of its due diligence
activities, the Navy required all bidders to demonstrate that they had
experience in implementing large seat management contracts.[Footnote
32] A practice that was not implemented by two projects was using
third-party assistance when selecting the provider. Projects that did
not implement this practice believed that they had adequate in-house
expertise with outsourcing, making third-party assistance unnecessary.
However, because third-party assistance provides an independent
resource that can suggest options or processes that the client
organization may not be aware of, these projects may have missed an
opportunity to implement their outsourcing projects more effectively.
Appendix V provides additional information on projects' implementation
of the practices comprising this phase.
* Phase V: Transition to provider(s). This phase focuses on the client
organization's transfer of the IT function to one or more providers. As
part of this transition, the clear definition of responsibilities and
the careful consideration of employees' needs matched against the
client organization's needs enable both the client and provider to
focus on execution and give staff confidence in their future
employment. The five projects largely implemented the 11 practices
associated with this phase. Specifically, about 90 percent[Footnote 33]
of the practices were implemented in this phase, and two projects
implemented all of the practices (the MHS/ITO Help Desk and NIMA IT/IS
projects). Several practices in this phase address dealing with
employees affected by the outsourcing projects. For example, in the
four projects in which federal employees were affected, the projects
provided assistance to those who did not want to transfer to the
provider, including helping to place them in other positions and
helping with résumé writing. A related practice is to clearly
communicate to all employees what is going to happen and when it is
going to happen. Two projects did not fully implement this practice.
For example, the Navy used its normal chain of command to communicate
transition information, but found that implementation of this practice
was uneven. As a result, some staff did not know current information
about how NMCI would affect them until the provider was ready to
contact them regarding their possible transition to the contractor.
However, according to the NMCI's Director's office, this problem was
somewhat mitigated by the provider's Web site that provides transition
information to all NMCI customers/users. Appendix VI provides
additional information on projects' implementation of the practices
comprising this phase.
* Phase VI: Manage provider(s) performance. The effectiveness with
which the provider(s) performance is managed is critical to the
successful implementation of an outsourcing project. Indeed, according
to Gartner, Inc., a leading research firm, an outsourcing project can
be thwarted by poorly designed, funded, and delivered processes for
managing the delivery of services.[Footnote 34] The five projects
generally implemented the 11 practices contained in this phase, with
about 84 percent of the practices being implemented in this phase. For
example, the practices related to obtaining feedback on provider
performance were largely implemented. This is important because
different levels of an agency can have different perceptions about the
value of the outsourcing project. For example, an outsourcing project
may be considered successful by the agency's executive management if it
is focused on controlling costs, but be considered inadequate by
business managers and users who may be expecting higher levels of
service. Each of these viewpoints is valid and should be taken into
account when the provider's performance is evaluated. Two other
practices--including incentives and penalties in contracts--were fully
implemented by two projects. Incentives and penalties are important
because they can help motivate the provider to exceed or meet
performance requirements. Nonetheless, two projects did not include
monetary incentives and two projects did not include monetary penalties
in their contracts. For example, the Army TAC-SWA project did not
include monetary incentives, although the contracting officer stated
that incentives might have been useful to motivate the provider to
exceed performance requirements. Incentives can also help control
risks. According to a guide on performance-based services acquisition,
if the incentives in the contract are right and if the provider and
agency share the same goals, risk is largely controlled and effective
performance is "almost the
inevitable outcome."[Footnote 35] Appendix VII provides additional
information on projects' implementation of the practices comprising
this phase.
* Phase VII: Ensure services are provided. Although outsourcing
transfers responsibility for performing the service to the provider(s),
the client organization is ultimately responsible for ensuring that
services are provided and that end-user needs are met. Accordingly, it
is critical that the agency ensure that services are provided. The
projects had implemented 97 percent of these practices, and four
projects implemented all of them. For example, every project monitored
the providers' work. In the case of the Air Force's C4 Services
project, quality assurance evaluators monitored the provider's work to
identify problems or trends in accordance with the project's quality
assurance surveillance plan. The results were reported to the
contracting officer and to the functional area chief for resolution.
Another practice, using customer satisfaction surveys, was fully
implemented by four of the five projects. However, the Air Force
project did not conduct, or require its contractor to conduct, customer
satisfaction surveys. Although the provider surveys staff annually, the
Air Force is nevertheless relying on the provider to voluntarily
implement an important practice for determining how customers view the
services being delivered and whether changes need to be made. Appendix
VIII provides additional information on projects' implementation of the
practices comprising this phase.
Leveraging Lessons Learned DOD-wide Could Assist Other DOD Projects:
We have previously reported on the importance of collecting and
disseminating lessons learned.[Footnote 36] For example, a critical
activity in IT investment management is establishing a process for
developing and capturing lessons learned in a written product or
knowledge base and disseminating them to decision-makers.[Footnote 37]
In addition, one of the practices in our framework for outsourcing IT
services addresses incorporating lessons learned from peers who have
engaged in similar sourcing decisions. Use of lessons learned is a
principal component of an organizational culture committed to
continuous improvement. Sharing such information serves to communicate
acquired knowledge more effectively and to ensure that beneficial
information is factored into planning, work processes, and activities.
Lessons learned can be based on positive experiences or on negative
experiences that result in undesirable outcomes.
Although DOD has taken action to gather and disseminate lessons learned
and best practices on general acquisition issues, these efforts
generally do not focus on outsourcing or include sharing the lessons
learned from IT outsourcing projects across the department.
Specifically, a number of DOD Web sites provide guidance, lessons
learned, and best practices related to general acquisition issues.
However, using these sites to locate specific information on IT
outsourcing best practices and lessons learned can be time-consuming
and difficult because so many topics and information sources are
provided. Specifically, MHS/ITO Help Desk project officials said that
searching numerous Web sites to get relevant information to address
questions and concerns about outsourcing IT services can consume hours.
For example, when we entered the keywords "IT outsourcing" and "best
practices" into the search feature on the Office of the Undersecretary
for Defense of Acquisition, Technology, and Logistics site, ACQWeb
(www.acq.osd.mil) in early March, it provided us with links to 1,251
documents. Likewise, lessons learned covers so many topics that it is
difficult to search for an applicable IT lesson. For example, when we
used the phrase "lessons learned in IT outsourcing" no documents were
identified on the ACQWeb, but when we inserted "lessons learned" and
"IT outsourcing" links to more than 1,700 documents were produced.
One DOD Web site, Share A-76!, was established to address one of our
previous recommendations,[Footnote 38] that is, to establish a
framework for identifying and analyzing best practices and lessons
learned from competitive sourcing studies and disseminating them DOD-
wide. Share A-76! promotes the sharing of best practices and lessons
learned related to one form of outsourcing that was conducted under the
OMB Circular A-76 competitive sourcing process. Among other things, the
site contains guidance, links to other relevant sites, sample
documents, and a best practices library that communicates field staff
experiences and advice about the Circular A-76 process. A NIMA project
official said that NIMA's staff routinely accesses the Web site because
it contains a wealth of information on policies, procedures, lessons
learned, and links to other outsourcing sites. The DOD analyst
responsible for Share A-76! estimated that the site receives about
12,000 visits per month and said that on the basis of E-mails and
anecdotes, user satisfaction is favorable. However, this Web site is
specific to the OMB Circular A-76 process, which may not apply to other
types of outsourcing. For example, Circular A-76's policy pertains to
public/private competitions and requires that the final evaluation
between the government and the private sector be based exclusively on
cost.
DOD acquisition and IT officials acknowledged that there is no
mechanism in DOD to easily share and leverage lessons learned relating
to outsourcing IT services. However, these officials agreed that a
departmentwide effort to identify, capture, and disseminate lessons
learned and leading practices of projects with experience in carrying
out IT outsourcing could offer valuable insights and new ideas that
would benefit others. Moreover, officials from three of the projects in
our review told us that there is value in collecting and disseminating
the knowledge acquired from IT outsourcing projects in a systematic
manner across the department.
Each of the projects in our review identified knowledge and experience
gained from their approaches to outsourcing IT services that could
offer insights and practices for other ongoing and future projects to
consider. For example:
* MHS developed specific guidance and lessons learned for implementing
a performance-based incentive contract for help-desk operations.
* The Department of the Navy's NMCI project has developed a series of
lessons learned related to transitioning to the provider that is being
shared within the NMCI community; one example was that all personnel
should be available during scheduled testing and deployment.
* NIMA has experience in contracting techniques emphasizing a
partnering approach with providers to refine requirements and establish
a common understanding of costs.
In addition, a departmental IT outsourcing knowledge-sharing approach
could include links to information about other government agencies' IT
outsourcing projects. For example, our 2002 report on desktop
outsourcing includes an extensive discussion of lessons learned by
agencies that have implemented this type of IT services
outsourcing.[Footnote 39]
Developing an effective lessons learned activity is not easy. For
example, NMCI officials said that for a lessons learned initiative to
be effective, a process must exist that is clearly understood by
everyone and allows capturing and sharing of knowledge to occur with
minimum effort. Other challenges in developing an effective lessons
learned process were outlined by the Share A-76! analyst. The analyst
stated that only a small number of site users have contributed lessons
learned to the Share A-76! Web site, which she attributed, in part, to
the amount of time and effort needed to document and obtain agreement
by all levels of the organization on the lessons learned. In addition,
the analyst stated that there is reluctance to share negative lessons,
and often the review and approval process sanitizes best practice
information so that it becomes too general to be most helpful to users.
Such challenges can be overcome by executive-level support. Indeed, DOD
acquisition and IT officials stated that for lessons learning
activities to be effective, senior management must devote support and
resources to the effort. This is consistent with our prior work, which
showed that knowledge can be effectively shared only when employees are
given adequate time as well as established places where they can
actually transfer knowledge.[Footnote 40]
Last year, we outlined a generic lessons learned process that could be
used to guide the development of such a process for outsourcing IT
services.[Footnote 41] Although the mechanism or processes used to
collect, share, and disseminate lessons learned may vary, in general
such a process comprises four main elements: collection, verification,
storage, and dissemination. The collection process involves the capture
of information through structured and unstructured processes.
Verification serves to verify the correctness and applicability of
lessons submitted. The storage aspect of lessons learned usually
involves incorporating the lessons into an electronic database for the
dissemination and sharing of information, including the ability to
conduct information searches. The final element, and the most
important, is the dissemination of lessons learned, since lessons are
of little benefit unless they are distributed and used by people who
will benefit from them. Lessons can be "pushed," or automatically
delivered to a user, or "pulled" in situations where a user must
manually search for them. Lessons can also be disseminated with an
assigned priority descriptor, which denotes the risk, immediacy, and
urgency of the lessons learned content.
Conclusions:
The projects in our review substantially implemented leading commercial
practices for outsourcing IT services, which has increased each
project's probability of success. Capturing how these projects
operationalized leading commercial practices could help other IT
services outsourcing projects succeed. Although currently there is no
such DOD-wide mechanism, such as an electronic tool, to easily share
and leverage lessons learned, DOD IT and acquisition officials agreed
that a departmentwide effort to identify, capture, and disseminate
lessons learned could offer valuable insights and new ideas that would
benefit others. Lessons learned that are pragmatic and easily
accessible could give DOD managers a more informed understanding of the
important issues to be addressed when making outsourcing decisions, as
well as the factors to be considered to help ensure the success of
these endeavors. DOD managers can also benefit from lessons learned on
the basis of negative experiences. The projects in our review were well
into implementation, and therefore, at this late stage, we see little
advantage for them to revisit practices that were not implemented.
Nevertheless, an electronic tool for capturing and disseminating
lessons learned would allow the rest of DOD to benefit from the
negative consequences and increased risks associated with those
practices that the projects did not implement.
Developing a lessons learned mechanism is not easy; thus, senior
management support and resources are keys to success. Without such
support driving the capture and dissemination of lessons learned, DOD
is losing an opportunity for wider application of leading practices and
thus better ensuring that its IT outsourcing efforts are successful.
Recommendations:
To assist DOD organizations in planning and implementing outsourcing
projects for IT services, we recommend that the Secretary of Defense
direct the Undersecretary of Defense for Acquisition, Technology, and
Logistics, working in conjunction with the Assistant Secretary of
Defense for Command, Control, Communications, and Intelligence (C3I),
to provide senior management support and adequate resources to develop
and implement an electronic tool to capture and disseminate examples
and lessons learned from actual IT outsourcing projects. These examples
and lessons learned, at a minimum, should include the results of our
review of the five projects discussed in this report.
We also recommend that the Secretary of Defense direct the
Undersecretary of Defense for Acquisition, Technology, and Logistics,
working in conjunction with the Assistant Secretary of Defense for
Command, Control, Communications, and Intelligence (C3I), to ensure
that the method used to gather information for this electronic tool
incorporate the main elements of a lessons learned process--namely,
collection, verification, storage, and dissemination.
Agency Comments and Our Evaluation:
In written comments on a draft of our report, signed by DOD's Principal
Director, Deputy Assistant Secretary of Defense (Deputy Chief
Information Officer), the department partially concurred with our
recommendations. Specifically, DOD agreed that capturing lessons
learned in the development and implementation of its IT outsourcing
initiatives is important to continually improve its outsourcing methods
and results achieved. The department also stated that before deciding
on a specific method to achieve this aim, it intends to explore a
variety of mechanisms that could be used. In particular, the department
stated that it currently has several processes and communities of
interest that collect and disseminate lessons learned in other areas,
which are logical starting points for determining the best path
forward. DOD's written comments are reproduced in appendix IX.
We agree that it is prudent to explore various alternatives to
leveraging lessons learned from DOD's IT services outsourcing
experiences. Our recommendations are not prescriptive as to the
electronic method to be used to capture and disseminate lessons
learned. Therefore, the department's plan to explore various
alternatives is consistent with our recommendations.
:
We are sending copies of this report to the appropriate congressional
committees, the Secretary of Defense, and the Director of the Office of
Management and Budget. We will also provide copies to others upon
request. In addition, the report will be available at no charge on the
GAO Web site at http://www.gao.gov.
If you have any questions on matters discussed in this report, please
contact me at (202) 512-3439 or Linda J. Lambert, Assistant Director,
at (202) 512-9556. We can also be reached by E-mail at hiter@gao.gov
and lambertl@gao.gov, respectively. Other contacts and key contributors
to this report are listed in appendix X.
Randolph C. Hite
Director, Information Technology Architecture and Systems Issues:
Signed by Randolph C. Hite:
[End of section]
Appendixes :
Appendix I: Objectives, Scope, and Methodology:
Our objectives were to determine (1) the extent to which selected
Department of Defense (DOD) information technology (IT) services
outsourcing projects use leading commercial practices as specified in
our framework and (2) whether DOD is sharing lessons learned from its
IT outsourcing projects across the department.
To determine the extent to which selected DOD outsourcing projects for
IT services use leading commercial practices, we identified the
practices in our November 2001 report on leading commercial
practices[Footnote 42] that (1) are typically applied at the project
level and (2) were verifiable through documentation and interviews.
Because DOD did not centrally maintain a list of outsourcing projects
for IT services, we asked the department to identify candidate projects
for our evaluation. From this list, we selected the following five
projects for our review: (1) Air Force Kirtland Air Force Base's
Command, Control, Communications, and Computer (C4) Services project;
(2) Army Network Enterprise Technology Command's (NETCOM) Total Army
Communications - Southwest Asia (TAC-SWA) project; (3) Military Health
System/Information Technology Organization (MHS/ITO) Help Desk
project; (4) Department of the Navy's Navy and Marine Corps Intranet
(NMCI) project; and (5) National Imagery and Mapping Agency (NIMA)
Information Technology/Information Services (IT/IS) project. We chose
each project on the basis of the following criteria: (1) no more than
one project from each military service and two agencies, (2)
illustrative example of DOD IT outsourcing, (3) dollar value greater
than $10 million, and (4) enough time elapsed for services to have been
delivered and performance measured.
At our request, each project completed a self-assessment on whether and
how it implemented leading commercial practices. We reviewed the agency
self-assessments and accompanying documentation and interviewed the
appropriate agency project officials to verify whether the practices
were followed. In addition, we interviewed representatives from each of
the providers associated with these projects.
We also researched additional information on commercial practices in
our November 2001 guide[Footnote 43] and the challenges the federal
government faces in implementing them. Specifically, we performed a
literature search, which included reviewing reports issued by leading
research firms, such as Gartner, Inc., and Giga Information Group, Inc.
In addition, we interviewed representatives from industry organizations
that have an interest in outsourcing IT services, including the
Coalition for Government Procurement; the Information Technology
Association of America; the Professional Services Council; and
Acquisition Solutions, Inc.
To determine whether DOD is sharing lessons learned from its IT
outsourcing projects across the department, we identified and reviewed
various approaches that DOD currently uses to capture and disseminate
such information. This included identifying and reviewing various Web
sites and performing key word searches on these sites to identify
lessons learned for outsourcing IT services. We also interviewed
applicable DOD acquisition and IT officials. Finally, we identified and
reviewed a generic lessons learned process contained in our January
2002 report on the National Aeronautics and Space Administration's
lessons learned mechanisms.[Footnote 44]
We performed our work at the Army's NETCOM in Ft. Huachuca, Arizona;
the MHS/ITO Help Desk project office in Falls Church, Virginia; the
NMCI Director's office in Crystal City, Virginia; and NIMA's
headquarters in Bethesda, Maryland. We conducted our review between May
2002 and early March 2003 in accordance with generally accepted
government auditing standards.
[End of section]
Appendix II: Projects' Implementation of Phase I: Determine Sourcing
Strategy:
[See PDF for image]
[End of figure]
In the first phase of our outsourcing framework, the client
organization determines whether internal capability or external
expertise can more effectively meet its IT needs. The purpose of this
sourcing strategy is to achieve the optimal balance between internal
and external capabilities, activities, processes, and services to
ensure that strategic business objectives are achieved at the lowest
risk. Among the factors that an organization should evaluate in
crafting this strategy are technology, business, financial, and
personnel requirements and whether it has skilled business and IT
managers. In addition, according to Gartner, Inc., sound sourcing
decisions depend on whether IT organizations (1) know and understand
their business priorities, (2) are prepared to invest in some skills
and divest others, and (3) identify and assess trade-offs.[Footnote 45]
The six practices in this phase that we used to evaluate the five
projects are as follows:
* Use third-party assistance with experience in a variety of sourcing
arrangements when formulating a sourcing strategy.
* Incorporate lessons learned from peers who have engaged in similar
sourcing decisions.
* Estimate impact of sourcing decision on internal organization.
* Benchmark and baseline productivity of internal services before
making the final sourcing decision.
* Determine the business reasons for outsourcing IT.
* Determine reasons for outsourcing IT that can improve the
organization's ability to use and manage technology.
Figure 6 shows that the implementation of the practices by the five
projects in our review was uneven.
Figure 6: Project Implementation of Phase I: Determine Sourcing
Strategy:
[See PDF for image]
Note: Not applicable--The practice was not relevant to the project's
particular circumstances.
No--The agency did not implement the practice. Limited--The agency
fully implemented some but not all aspects of the practice and did not
take alternative actions that fully satisfied the practice. Yes--The
agency fully implemented the practice or took an alternative action
that fully satisfied its intent.
[End of figure]
Table 4 provides detailed information on whether and how each project
implemented each of the six practices in this phase.
Table 4: Summary of Projects' Use of Phase I Practices: Did The Project
Use the Practice?:
Practice: Use third-party assistance with experience in a variety of
sourcing arrangements when formulating a sourcing strategy.; Did the
project use the practice?: Air Force C4 Services: No--According to Air
Force project officials, they decided that they had sufficient in-house
expertise, and the project had no funding available to employ a
contractor. However, the former functional area chief told us that
using third-party assistance would have been beneficial because the
requirements would have been better written. According to this
official, the requirements had to be substantially rewritten 2 years
after contract award.; Did the project use the practice?: Army TAC-SWA:
No--According to a TAC-SWA project official, the Army decided that it
had sufficient expertise in-house since the TAC-SWA contract was a
consolidation of three existing contracts.; Did the project use the
practice?: MHS/ITO Help Desk: Yes--For example, MHS worked with the
Department of the Interior's GovWorks Program and the Defense
Acquisition University staff to obtain expertise on sourcing
strategies. Also, MHS employed a contractor with sourcing expertise.;
Did the project use the practice?: Navy NMCI: Yes--The Navy employed a
contractor with sourcing expertise and contacted other government
entities about their experiences.; Did the project use the practice?:
NIMA IT/IS: Yes--NIMA employed a contractor with sourcing expertise.
Practice: Incorporate lessons learned from peers who have engaged in
similar sourcing decisions.; Did the project use the practice?: Air
Force C4 Services: Yes--Primarily from other Department of Defense
(DOD) projects that implemented Office of Management and Budget (OMB)
Circular A-76 policies.; Did the project use the practice?: Army TAC-
SWA: Yes--According to a TAC-SWA project official, it used lessons
learned from its prior contracts for these services and another similar
Army contract.; Did the project use the practice?: MHS/ITO Help Desk:
Yes--Primarily from industry peers and MHS's prior help-desk function.;
Did the project use the practice?: Navy NMCI: Yes--Navy officials
stated that there were no peers that had engaged in similar sourcing
decisions because no other outsourcing project was of as large a scale
as NMCI. However, the Navy did talk to members of private industry and
the National Aeronautics and Space Administration on their more limited
efforts.; Did the project use the practice?: NIMA IT/IS: Yes--A NIMA
contractor provided the agency with a report on industry best
practices. In addition, NIMA held discussions with the National
Security Agency on its outsourcing effort.
Practice: Estimate impact of sourcing decision on internal
organization.; Did the project use the practice?: Air Force C4
Services: Yes--The Air Force estimated that there would be substantial
internal impact, such as to its staff, due to its decision to
outsource.; Did the project use the practice?: Army TAC-SWA: No--The
Army did not analyze the impact on its internal organization because,
according to the TAC-SWA program manager, NETCOM was outsourcing a
function that was already contracted out.[A]; Did the project use the
practice?: MHS/ITO Help Desk: Yes--MHS assessed the staff and financial
impact of its sourcing decision.; Did the project use the practice?:
Navy NMCI: Yes--The Navy performed an assessment of staffing and other
impacts, such as cost, related to its sourcing decision.; Did the
project use the practice?: NIMA IT/IS: Yes--NIMA performed an
assessment of staffing and other impacts, such as cost, related to its
sourcing decision.
Practice: Benchmark and baseline productivity of internal services
before making the final sourcing decision.; Did the project use the
practice?: Air Force C4 Services: No--The Air Force did not implement
this practice because, according to project officials, an executive
decision was made to outsource the C4 services following the policies
contained in OMB Circular A-76. Accordingly, Air Force officials stated
that such analyses would not have affected the final sourcing
decision.; Did the project use the practice?: Army TAC-SWA: No--The
Army did not perform a benchmark or baseline analysis because,
according to the TAC-SWA program manager, NETCOM was outsourcing a
function that was already contracted out.; Did the project use the
practice?: MHS/ITO Help Desk: Limited--MHS attempted to establish a
baseline of its prior environment but, according to project officials,
available documentation was limited to historical trouble ticket
workload data; therefore, this baseline was a best estimate. In
addition, MHS project officials stated that a contractor performed a
benchmark analysis, but they did not provide supporting documentation.;
Did the project use the practice?: Navy NMCI: Limited--The Navy had a
private-sector firm benchmark its environment against seven large
public and private organizations. In addition, the Navy and the Marine
Corps performed an analysis at a sample of representative locations to
obtain a baseline. However, this baseline did not include an assessment
of the Department of the Navy's legacy applications since project
officials decided to rely upon an inventory developed in addressing the
Year 2000 challenge. The Navy subsequently found that it had
substantially underestimated its number of legacy applications.
According to NMCI program officials, this underestimation contributed
to the transition period slipping from 2-˝ years to 3-˝ years.; Did the
project use the practice?: NIMA IT/IS: Yes--A NIMA contractor baselined
the existing NIMA environment and benchmarked it to peers.
Practice: Determine the business reasons for outsourcing IT.; Did the
project use the practice?: Air Force C4 Services: Yes--To achieve cost
savings and to shift military personnel to other work.; Did the project
use the practice?: Army TAC-SWA: Yes--To address shortages in military
personnel to perform its mission.; Did the project use the practice?:
MHS/ITO Help Desk: Yes--To achieve increased productivity and customer
satisfaction and to decrease costs.; Did the project use the practice?:
Navy NMCI: Yes--To have private industry capitalize infrastructure
improvements that were needed to quickly and securely share knowledge
around the globe.; Did the project use the practice?: NIMA IT/IS: Yes-
-To obtain improved customer services and decreased costs. Also, this
function was determined to be a commercial function under the Federal
Activities Inventory Reform Act.[B].
Practice: Determine reasons for outsourcing IT that can improve the
organization's ability to use and manage technology.; Did the project
use the practice?: Air Force C4 Services: No--According to Kirtland Air
Force Base officials, the Air Force decided to hold a public/private
competition following OMB Circular A-76 policies; therefore, improving
its ability to use and manage technology was not a factor in
determining its outsourcing strategy.; Did the project use the
practice?: Army TAC-SWA: No--According to TAC-SWA project officials,
improving its ability to use and manage technology was not a factor in
determining its outsourcing strategy.; Did the project use the
practice?: MHS/ITO Help Desk: Yes--To improve IT expertise and
knowledge.; Did the project use the practice?: Navy NMCI: Yes--To
improve, for example, system security, interoperability, reliability,
and network response.; Did the project use the practice?: NIMA IT/IS:
Yes--To achieve better IT management performance.
Source: GAO.
[A] Having had the activity previously performed by a contractor does
not obviate the need to estimate the impact of a sourcing decision on
the internal organization--there are still risks involved, such as the
potential disruption of services during the transfer to the new
contractor. In fact, the staff from the incumbent contractor did not
transition to the TAC-SWA provider, and the provider had difficulty
filling these slots within the schedule outlined in the contract.
[B] The Federal Activities Inventory Reform Act of 1998 requires
federal agencies to prepare and submit to OMB, by June 30 of each year,
inventories of the commercial activities performed by federal
employees.
[End of table]:
[End of section]
Appendix III: Project's Implementation of Phase II: Define Operational
Model:
[See PDF for image]
[End of figure]
Critical to the successful outsourcing relationship is an operational
model for guiding the structure of the contract and the plans for
transition. In defining the operational model, client organizations
formalize executive leadership, team composition, client
responsibilities, and operating relationships between the client and
provider. The operational model helps the organization to compare its
plans with the expectations that were set as the initial decision to
outsource was made and to ascertain whether these plans will enable the
organization to meet those objectives. An important aspect of the
operational model is an explicit understanding of how the client
organization plans to communicate its needs and provide feedback to the
provider. In addition, communication between the business and IT
offices within the client organization is always critical. This is
particularly true in the case of outsourcing because the IT service
provider is outside the client organization and disconnects are more
likely to occur. Therefore, organizational processes to facilitate good
communication are critical.
The 13 practices in this phase that we used to evaluate the five
projects are as follows:
* Establish executive leadership for IT to facilitate the outsourcing
initiative.
* Continually communicate/clarify outsourcing objectives, while
correcting misinformation that affects the organization.
* Establish a core group of people who will be involved in all phases
of outsourcing.
* Select a person involved in the negotiation of the contract to manage
the outsourcing relationship.
* Create and define a contract management structure with operational
points of contact and managers.
* Define the role of internal IT managers and business leaders.
* Ensure that the right skills are in place to support the outsourcing
relationship.
* Establish a point of contact high in the provider management
structure for elevating provider performance concerns.
* Have provider establish an on-site support team to serve as liaison
between client and provider.
* Train provider on client business environment and goals.
* Select or develop standard tools for managing the relationship.
* Use third-party assistance to take advantage of expertise from a
variety of outsourcing arrangements in defining the operational model.
* Ensure that the provider management team has prior experience in the
client's field of business.
Figure 7 illustrates that the five projects in our review largely
implemented the practices.
Figure 7: Project Implementation of Phase II: Define Operational Model:
[See PDF for image]
Note: Not applicable--The practice was not relevant to the project's
particular circumstances.
No--The agency did not implement the practice. Limited--The agency
fully implemented some but not all aspects of the practice and did not
take alternative actions that fully satisfied the practice. Yes--The
agency fully implemented the practice or took an alternative action
that fully satisfied its intent.
[End of figure]
Table 5 provides detailed information on whether and how each project
implemented each of the 13 practices in this phase.
Table 5: Summary of Projects' Use of Phase II Practices:
Practice: Establish executive leadership for IT to facilitate the
outsourcing initiative.; Did the project use the practice?: Air Force
C4 Services: Yes--The initiative was planned by a steering group made
up of representatives from various major offices at Kirtland Air Force
Base. Subsequent to contract award, the Air Force established a
functional area chief to manage the initiative, and a Lieutenant
Colonel was appointed to be responsible for this project.; Did the
project use the practice?: Army TAC-SWA: Yes--NETCOM used the existing
leadership in its offices of operations and logistics to provide
executive leadership.; Did the project use the practice?: MHS/ITO Help
Desk: Yes--MHS used its existing leadership structure and processes.
This structure includes a program executive office, steering committee,
and program review board.; Did the project use the practice?: Navy
NMCI: Yes--In 1999, the Navy established a program executive office for
IT primarily to support the NMCI outsourcing effort. The Congress later
directed the Navy to identify a single individual whose sole
responsibility would be to oversee and direct the NMCI program. As a
result, in February 2002, the Navy established the NMCI Director's
Office to take over responsibility for NMCI.; Did the project use the
practice?: NIMA IT/IS: Yes--The NIMA enterprise transformation
directorate formed a strategic sourcing office to provide executive
oversight of the effort.
Practice: Continually communicate/clarify outsourcing objectives,
while correcting misinformation that affects organization.; Did the
project use the practice?: Air Force C4 Services: Yes--The Air Force
provided communication through, for example, briefings and meetings
with employees.; Did the project use the practice?: Army TAC-SWA: Yes-
-According to TAC-SWA project officials, the applicable NETCOM
officials were informed about the initiative and provided comments on
the draft performance work statement. TAC-SWA project officials also
brief new commanders on the contract before they are transferred to the
military theater.; Did the project use the practice?: MHS/ITO Help
Desk: Yes--MHS provided updates to the Chief Information Officer (CIO)
and program review boards, provided briefings to the deputy surgeon
general, and published questions and answers.; Did the project use the
practice?: Navy NMCI: Yes--The Navy established an action collaboration
team structure to involve the Navy community in the NMCI communications
process. Also, the NMCI Information Bureau initiated press conferences,
briefings, site visits, informational pamphlets, and promotional
material. In addition, the NMCI Director and other staff provide
numerous briefings and presentations to commands, industry gatherings,
and government officials. Finally, Web sites and Web-based
collaboration sites were established to facilitate communications.; Did
the project use the practice?: NIMA IT/IS: Yes--NIMA created an
internal Web site to post information on the outsourcing project,
distributed periodic global electronic mails, and held town hall
meetings.
Practice: Establish a core group of people who will be involved in all
phases of outsourcing.; Did the project use the practice?: Air Force C4
Services: Yes--The core group included the contracting officer,
functional area chief, and manpower specialist.; Did the project use
the practice?: Army TAC-SWA: Yes--According to TAC-SWA project
officials, the core group included representatives from logistics,
operations, and resource management as well as the contracting
officer.; Did the project use the practice?: MHS/ITO Help Desk: Yes--
According to program officials, the core group included representatives
from the MHS program executive office, MHS IT program offices, military
department chief information officers' offices, and Interior's GovWorks
organization, which provided contracting services.; Did the project use
the practice?: Navy NMCI: Yes--The program executive office for IT and
the NMCI Director's organization comprise the core group responsible
for managing the outsourcing initiative.; Did the project use the
practice?: NIMA IT/IS: Yes--NIMA established the Strategic Sourcing
Office, which was dedicated to managing and facilitating the
outsourcing initiative.
Practice: Select a person involved in the negotiation of the contract
to manage the outsourcing relationship.; Did the project use the
practice?: Air Force C4 Services: Yes--The contracting officer.[A]; Did
the project use the practice?: Army TAC-SWA: Yes--The contracting
officer.; Did the project use the practice?: MHS/ITO Help Desk: Yes--
According to MHS program officials, the chair of the contract
evaluation committee and the individual in charge of transition are
responsible for managing the outsourcing relationship.; Did the project
use the practice?: Navy NMCI: Yes--The primary contracting officer.;
Did the project use the practice?: NIMA IT/IS: Yes--A lead contracting
officer.
Practice: Create and define contract management structure with
operational points of contact and managers.; Did the project use the
practice?: Air Force C4 Services: Yes--The structure was defined by the
roles and responsibilities of the contracting officer and her staff.;
Did the project use the practice?: Army TAC-SWA: Yes--The overall
responsibility for the contract rests with the contracting officer. The
contracting officer's representative acts as a liaison between the
government and the contractor.; Did the project use the practice?: MHS/
ITO Help Desk: Yes--Responsibility for contract management is jointly
held by the contracting officer, who is part of Interior's GovWorks
organization, and the MHS program office, which provides day-to-day
oversight of the provider.; Did the project use the practice?: Navy
NMCI: Yes--The primary contracting officer establishes procedures and
controls necessary for effective contractual oversight of the NMCI
initiative and has a matrix relationship with the NMCI Director. The
primary contracting officer certifies contracting officer
representatives for the NMCI contract to provide technical coordination
efforts.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA has
appointed an operational point of contact for each of the seven
functional areas being outsourced, which are documented in a "rules of
engagement" agreement between the government and the contractor.
Practice: Define the role of internal IT managers and business
leaders.; Did the project use the practice?: Air Force C4 Services:
Yes--Kirtland Air Force Base did not initially define the roles of its
internal IT and business managers. However, subsequent to contract
award, Kirtland Air Force Base established a CIO committee and
configuration control board comprising the functional area chief
(Kirtland's CIO) and representatives from various business areas.
Although not exclusively devoted to the C4 Services project, these
groups help provide direction to the project.; Did the project use the
practice?: Army TAC-SWA: Limited--TAC-SWA project officials explained
the roles of the principal staff involved with this project but
generally did not have supporting documentation defining these roles.;
Did the project use the practice?: MHS/ITO Help Desk: Yes--MHS's help-
desk performance assessment plan defines the roles of various groups
and individuals associated with the program.; Did the project use the
practice?: Navy NMCI: Yes--The NMCI execution plan describes the roles
of internal and external organizations that directly or indirectly
affect the management of NMCI and explains in detail the duties and
responsibilities of the program executive office for IT and the NMCI
program management offices.; Did the project use the practice?: NIMA
IT/IS: Yes--NIMA transformation teams for the activities being
outsourced define the roles of their managers and leaders.
Practice: Ensure that the right skills are in place to support the
outsourcing relationship, including those dealing with; ; * contract
management,; * financial management,; * IT management,; * negotiation
strategies,; * teaming and interpersonal relationships,; * project
management, and; * relationship management.; Did the project use the
practice?: Air Force C4 Services: Yes--According to Air Force
officials, the right skills are in place.; Did the project use the
practice?: Army TAC-SWA: Yes--According to the TAC-SWA project
officials, the Army has created several positions to ensure that the
right skills are in place to manage the outsourcing relationship,
including a contracting officer, legal advisor, operations and
logistics personnel as well as a NETCOM unit commander. According to
Army TAC-SWA officials, each position employs an individual with the
necessary skills to support the outsourcing relationship.; Did the
project use the practice?: MHS/ITO Help Desk: Yes--According to MHS
program officials, the Tri-Service Management Program Office, GovWorks
(the Interior organization that provides contracting assistance to
MHS), and a contractor collectively employ the skill sets needed to
support the outsourcing relationship.; Did the project use the
practice?: Navy NMCI: Limited--The Director, NMCI, has staff that
report directly or are matrixed with him that are responsible for
performing all but two of these functions. Specifically, at this time,
NMCI does not have staff assigned to support teaming and interpersonal
relationships and relationship management. The Navy recognizes the need
for these skills and is taking, or plans to take, various actions to
obtain these skills.; Did the project use the practice?: NIMA IT/IS:
Yes--According to NIMA, its transformation teams provide the skills
necessary to support the outsourcing relationship.
Practice: Establish a point of contact high in the provider management
structure for elevating provider performance concerns.; Did the project
use the practice?: Air Force C4 Services: Yes--According to the
contracting officer, concerns can be addressed to the senior vice
president at the provider headquarters.; Did the project use the
practice?: Army TAC-SWA: Yes--The provider established a program
manager as the point of contact for elevating concerns.; Did the
project use the practice?: MHS/ITO Help Desk: Yes--The provider
established an executive program manager as the point of contact for
elevating concerns.; Did the project use the practice?: Navy NMCI: Yes-
-The NMCI Director interacts directly with the provider's program
executive on a regularly scheduled basis; such meetings would include
any performance issues.; Did the project use the practice?: NIMA IT/IS:
Yes--The transformation team charters define procedures for addressing
issues, including possibly elevating concerns to the provider's general
manager.
Practice: Have provider establish an on-site support team to serve as a
liaison between client and provider.; Did the project use the
practice?: Air Force C4 Services: Yes--The provider has established an
on-site support team, which is led by the on-site manager.; Did the
project use the practice?: Army TAC-SWA: Yes--Site managers have been
established in accordance with the contract.; Did the project use the
practice?: MHS/ITO Help Desk: Yes--The provider's transition plan
identifies the on-site support team.; Did the project use the
practice?: Navy NMCI: Yes--The provider establishes site leads that
remain on-site as the location goes through the cutover to NMCI.; Did
the project use the practice?: NIMA IT/IS: Yes--The rules of engagement
define the on-site support team for each major NIMA location.
Practice: Train provider on client business environment and goals.; Did
the project use the practice?: Air Force C4 Services: Limited--Prior to
contract award, Kirtland Air Force Base provided an orientation session
in which they discussed their operations, business environment, and
goals. Subsequent to contract award, specific Kirtland Air Force Base
government personnel were responsible for aiding the provider during
the transition period. However, provider officials stated that they did
not receive adequate training, which made the transition period more
difficult.; Did the project use the practice?: Army TAC-SWA: Not
applicable--The Army contract was a consolidation of prior contracts
for this activity, and the winning bidder was one of the incumbent
contractors.; Did the project use the practice?: MHS/ITO Help Desk:
Yes--MHS officials stated that they provided some training to the
provider on their business environment. In particular, MHS trained and
certified the provider's staff on the agency's application systems. In
addition, the provider had previous MHS experience.; Did the project
use the practice?: Navy NMCI: Limited--The Navy did not provide
training on its business environment and goals; instead it relied on
the NMCI design reference mission document, which was included with the
request for proposals (RFP). This document defined the NMCI operational
environment. However, a provider official stated that although the
document was useful, it did not identify Navy enterprisewide
operations. According to NMCI program officials, the provider somewhat
mitigated this problem by hiring several highly knowledgeable staff
from the Department of the Navy shortly after contract award.; Did the
project use the practice?: NIMA IT/IS: Not applicable--According to the
project's contracting officer, such training was not necessary because
the senior officers of the provider are all recent NIMA management
employees and were already knowledgeable of these areas. For example,
among those who transitioned to the provider was a former director of
the hydrographic topographic center of the Defense Mapping Agency, a
predecessor agency to NIMA.
Practice: Select or develop standard tools for managing the
relationship (e.g., performance scorecards, enterprise resource
management system).; Did the project use the practice?: Air Force C4
Services: Yes--For example, the revised contract outlines the use of
specific software to help manage the provider's performance.; Did the
project use the practice?: Army TAC-SWA: Yes--NETCOM uses a monthly
report from the contracting officer's representative to evaluate the
provider.; Did the project use the practice?: MHS/ITO Help Desk: Yes--
According to MHS officials, the program office and the provider have
selected, for example, an enterprise resource management system.; Did
the project use the practice?: Navy NMCI: Yes--The Navy is using a
balanced scorecard process to provide Navy and Marine Corps leadership
with information to judge how well NMCI is supporting the missions and
strategies of the department.; Did the project use the practice?: NIMA
IT/IS: Yes--NIMA used various tools, including twice-a-day performance
reports for the operational help-desk function and monthly performance
reports.
Practice: Use third-party assistance to take advantage of expertise
from a variety of outsourcing arrangements in defining the operational
model (i.e., defining roles and responsibilities).; Did the project use
the practice?: Air Force C4 Services: Yes--Although third-party
assistance was not used at the onset of the C4 Services project, a
person with prior experience in managing IT services contracts for the
Air Force was brought in to provide advice. He was subsequently put in
charge of the project.; Did the project use the practice?: Army TAC-
SWA: No--The Army believed that it had sufficient expertise in-house.;
Did the project use the practice?: MHS/ITO Help Desk: Yes--For example,
MHS contacted the Interior's GovWorks Program, Defense Acquisition
University, and others. Also, MHS employed a contractor with sourcing
expertise.; Did the project use the practice?: Navy NMCI: No--The Navy
believed it had sufficient internal experience and expertise.; Did the
project use the practice?: NIMA IT/IS: Yes--NIMA employed contractors
to develop contractual roles and assist in evaluating the contract.
Practice: Ensure that the provider management team has prior experience
in the client's field of business.; Did the project use the practice?:
Air Force C4 Services: Yes--Provider management capabilities and
experience in providing IT services were evaluation factors in the
technical evaluation process.; Did the project use the practice?: Army
TAC-SWA: Yes--Vendor past performance was part of the evaluation
criteria during source selection and the provider was an incumbent
contractor with NETCOM.; Did the project use the practice?: MHS/ITO
Help Desk: Yes--The request for quote laid out MHS's evaluation
criteria for selecting a provider, which included prior experience in
health care.; Did the project use the practice?: Navy NMCI: Yes--
According to NMCI program officials, the Navy required bidders to
provide information on their prior experience in related business
fields. For example, the solicitation required bidders to demonstrate
experience managing a similar effort of 100,000 or more seats at one
time.[B]; Did the project use the practice?: NIMA IT/IS: Yes--It was
expected that most of the provider's workforce would be composed of
former NIMA employees. Among those who transitioned to the provider was
a former director of the hydrographic topographic center of the Defense
Mapping Agency.
Source: GAO.
[A] The Air Force's initial contract was not negotiated because
Kirtland Air Force Base used an OMB Circular A-76 2-step, sealed bid
process. However, the contracting officer was involved in the bid
process as well as in managing the winning contractor.
[B] This threshold could be met through multiple efforts, but at least
one had to include at least 20,000 seats.
[End of table]:
[End of section]
Appendix IV: Project's Implementation of Phase III: Develop the
Contract:
[See PDF for image]
[End of figure]
Phase III focuses on the development of the contract, which defines the
legal terms of the relationship between client and provider. While
other phases in the outsourcing process describe the need for mutual
trust and a close, flexible working relationship, this phase focuses on
the development of the contract, which is the foundation on which a
working relationship will be built. A well-written contract helps the
outsourcing organization meet its requirements while allowing the
service provider to make a fair profit. It sets the expectations for
service levels, delivery of essential services, and continuous
improvement and is intended to protect the interests of all parties.
The 16 practices in this phase that we used to evaluate the five
projects are as follows:
* Base performance requirements on business outcomes.
* Include measures that reflect end-user satisfaction as well as
technical IT performance.
* Review and update performance requirements periodically.
* Require the provider to meet the minimum performance in each category
of service.
* Require the provider to achieve escalating performance standards at
agreed-upon intervals.
* Incorporate sufficient flexibility so that minimum acceptable
performance can be adjusted as conditions change, as the provider
becomes more adept at satisfying customer demands, and as improvement
goals are achieved.
* Use service-level agreements (SLA)[Footnote 46] to clearly articulate
all aspects of performance, including management, processes, and
requirements.
* Client and provider work together to define the appropriate number of
SLAs and appropriate structure for each.
* Specify circumstances under which the provider is excused from
performance levels mandated by master service agreements.
* Client and provider work together to identify SLAs for which
compensation is based, while additional SLAs may be defined to manage
performance.
* The contract should include clauses for (1) determining pricing
structures; (2) performing customer satisfaction surveys and using the
results to redefine performance levels; (3) terminating the contract;
(4) resolving disputes in a timely manner; (5) taking work away,
without penalty, from provider for nonperformance; (6) declaring a
significant event that can lead to a change in the contract; (7)
defining performance requirements; and (8) conducting regularly
scheduled meetings.
* Consider setting up a master services agreement under which all
arrangements between client and provider operate.
* Include the appropriate representation from each major organizational
unit on the contract negotiation team.
* Specify the use of volume purchases to obtain optimal discounts.
* Use third-party assistance in negotiating and developing the
contract.
* Sign the contract after contract negotiations and final vendor
selections.
As figure 8 shows, the five projects largely implemented the practices.
Figure 8: Project Implementation of Phase III: Develop the Contract:
[See PDF for image]
Note: Not applicable--The practice was not relevant to the project's
particular circumstances.
No--The agency did not implement the practice. Limited--The agency
fully implemented some but not all aspects of the practice and did not
take alternative actions that fully satisfied the practice. Yes--The
agency fully implemented the practice or took an alternative action
that fully satisfied its intent.
[End of figure]
Table 6 provides detailed information on whether and how each project
implemented each of the 16 practices in this phase.
Table 6: Summary of Projects' Use of Phase III Practices:
Practice: Base performance requirements on business outcomes.; Did
the project use the practice?: Air Force C4 Services: Yes--The
performance requirements are contained in the contract and are based on
business outcomes. For example, the requirements define the percentage
of system availability needed to support users.; Did the project use
the practice?: Army TAC-SWA: Yes--The performance requirements are
outcome based and stated in terms of the level of operation and
maintenance services required. According to TAC-SWA officials, multiple
trips to overseas locations were made to identify the performance
requirements.; Did the project use the practice?: MHS/ITO Help Desk:
Yes--Industry standards and metrics were used to base performance
requirements on business outcomes.; Did the project use the practice?:
Navy NMCI: Yes--The performance requirements are included in the
contract and are based on business outcomes. SLAs establish the
performance standards and service quality for all types of NMCI seats.;
Did the project use the practice?: NIMA IT/IS: Yes--Performance-based
requirements are in the contract, including those related to customer
satisfaction, process improvement, quality control, and timely and
accurate completion of requirements.
Practice: Include measures that reflect end-user satisfaction as well
as technical IT performance.; Did the project use the practice?: Air
Force C4 Services: Limited--The Air Force measures include technical IT
performance. However, end-user satisfaction measures are limited to the
percentage of complaints received and do not measure overall customer
satisfaction. Project officials did not know why overall end-user
satisfaction measures were not established.; Did the project use the
practice?: Army TAC-SWA: Limited--The Army's contract includes
technical IT performance measures but not end-user satisfaction
performance measures. For example, the contract includes a requirement
for help-desk services, but does not include measures related to
customer satisfaction associated with such services. According to the
project official that developed the performance work statement in the
contract, NETCOM did not include such customer satisfaction measures
because it did not think that it was necessary to have a performance
standard for that service.; Did the project use the practice?: MHS/ITO
Help Desk: Yes--MHS measures the service provider's technical IT
performance and uses an electronic customer satisfaction survey to
assess the quality of help-desk services. Satisfaction rates and number
of survey responses are tracked and used for both evaluation and
incentive payments (positive and negative).; Did the project use the
practice?: Navy NMCI: Yes--The SLAs include measures for technical IT
performance and customer satisfaction. In addition, the contract
requires the service provider to measure and report on overall customer
satisfaction with NMCI services.; Did the project use the practice?:
NIMA IT/IS: Yes--The SLAs contain technical IT performance measures. In
addition, after a function is transitioned, the contractor must provide
a plan that includes measures for customer satisfaction, which the
provider is contractually obligated to meet.
Practice: Review and update performance requirements periodically.;
Did the project use the practice?: Air Force C4 Services: Yes--The
performance requirements were revised about 2 years after contract
award.; Did the project use the practice?: Army TAC-SWA: Yes--The
contract is reviewed and modified as needed on the basis of additions
or changes to such requirements. According to TAC-SWA and provider
officials, changes are coordinated between the government and the
contractor before they are finalized.; Did the project use the
practice?: MHS/ITO Help Desk: Yes--According to MHS program officials,
performance requirements are updated as required and reviewed on an
annual basis before the decision is made to execute the option year on
the contract.; Did the project use the practice?: Navy NMCI: Yes--
Performance reviews and adjustments are ongoing. For example, the Navy
is in the process of refining its SLAs to ensure that they adequately
reflect technical performance and customer satisfaction needs.; Did the
project use the practice?: NIMA IT/IS: Yes--According to NIMA,
requirements are updated as needed or once a year when the government
is assessing whether to exercise its annual option.
Practice: Require the provider to meet the minimum performance in
each category of service.; Did the project use the practice?: Air Force
C4 Services: Yes--The minimum performance is defined in the contract.;
Did the project use the practice?: Army TAC-SWA: Yes--The contract
defines minimum performance requirements.; Did the project use the
practice?: MHS/ITO Help Desk: Yes--The minimal acceptable performance
criteria the contractor must meet are based upon commercial industry
standards and are defined in the contract.; Did the project use the
practice?: Navy NMCI: Yes--The SLAs provide the basic level of service
the contractor must deliver for every NMCI seat.; Did the project use
the practice?: NIMA IT/IS: Yes--The provider must meet the performance
standards as specified in the contract, SLAs, and NIMA's performance
assessment plan.
Practice: Require the provider to achieve escalating performance
standards at agreed-upon intervals.; Did the project use the practice?:
Air Force C4 Services: No--According to Kirtland Air Force Base
officials, they did not include escalating performance standards
because the focus of this contract was on meeting requirements at the
least cost.[A]; Did the project use the practice?: Army TAC-SWA: Not
applicable--According to a TAC-SWA project official, this practice was
not applicable because NETCOM believed that the performance standards
in the contract were already high. Provider officials also stated that
the performance requirements in the contract were high. The Army
expected, and the contractor agreed, to meet these standards
immediately.; Did the project use the practice?: MHS/ITO Help Desk:
Yes--The contract defines negative, acceptable, and positive incentive
ranges associated with escalating performance standards.; Did the
project use the practice?: Navy NMCI: No--According to NMCI program
officials, the Navy expected vendors to develop their pricing proposal
assuming 8 years[B] of providing service in accordance with the SLAs
included in the RFP.; Did the project use the practice?: NIMA IT/IS:
Yes--The contract allows for the redefinition of service levels. In
addition, according to NIMA program officials, the need for escalating
performance standards will be evaluated on an as-needed basis as well
as during the semiannual award fee analysis period and during the
annual review on whether to exercise the contract option. However, such
escalating standards have not yet been identified.
Practice: Incorporate sufficient flexibility so that minimum
acceptable performance can be adjusted as conditions change, as the
provider becomes more adept at satisfying customer demands, and as
improvement goals are achieved.; Did the project use the practice?: Air
Force C4 Services: No--The contracting officer said that the
requirements should be built into the contract, which can be modified
if necessary.; Did the project use the practice?: Army TAC-SWA: Not
applicable--According to a TAC-SWA project official, NETCOM believed
that the performance standards in the contract were already high, and
provider officials agreed. According to the contracting officer, the
military theater where performance is delivered is not an area where
anything less than meeting the defined standards is acceptable.; Did
the project use the practice?: MHS/ITO Help Desk: Yes--According to MHS
program officials, the contract incorporates sufficient flexibility to
adjust minimal acceptable performance as conditions change. For
example, the contract includes incentives for the service provider to
introduce new capabilities and new services within the scope of the
performance work statement without further competition.; Did the
project use the practice?: Navy NMCI: Yes--The contract includes a
provision requiring the provider to submit an annual plan for
technology refreshment and deployment. This plan is to include proposed
revisions to the contract and an estimate of changes in performance
that would result.; Did the project use the practice?: NIMA IT/IS: Yes-
-According to NIMA, the performance measures are designed to first
stabilize NIMA's IT environment and then to be adjusted to bring
minimum performance into alignment with industry best practices. Also,
the contract indicates that the SLAs may be further defined throughout
the contract life.
Practice: Use SLAs to clearly articulate all aspects of performance,
including management, processes, and requirements.; Did the project use
the practice?: Air Force C4 Services: Yes--The SLA defines the
requirements, processes, and who is responsible for meeting the
requirements.; Did the project use the practice?: Army TAC-SWA: Yes--
The TAC-SWA project did not use SLAs, but the contract addresses the
management, processes, and requirements associated with the project.;
Did the project use the practice?: MHS/ITO Help Desk: Yes--The contract
defines the requirements, service performance standards, thresholds,
objectives, and metrics as well as the requirements for quarterly
management reviews and quality assurance plans.; Did the project use
the practice?: Navy NMCI: Yes--According to NMCI officials, SLAs cover
all aspects of provider performance. For example, the E-mail services
SLA includes location and frequency of the service, performance
categories, the performance measures, and methods of measurement.; Did
the project use the practice?: NIMA IT/IS: Yes--The contract
incorporates SLAs, which articulate performance requirements and take
effect when the function is transitioned to the provider.
Practice: Client and provider work together to define the appropriate
number of SLAs and appropriate structure for each.; Did the project use
the practice?: Air Force C4 Services: Yes--Although the original SLA
was prepared by the Air Force, the revised SLA included contractor
input and concurrence.; Did the project use the practice?: Army TAC-
SWA: Yes--SLAs are not used but, according to TAC-SWA and provider
officials, they work together on any contract modifications dealing
with requirements and performance levels. In addition, the partnering
clause in the contract emphasizes a mutual commitment between
government and industry to work as a team.; Did the project use the
practice?: MHS/ITO Help Desk: Yes--MHS did not work with the service
provider to determine the number and structure of these requirements.
However, according to MHS and provider officials, they have worked
together on revisions to these requirements.; Did the project use the
practice?: Navy NMCI: Yes--The SLAs were developed by the Navy with the
help of a third-party contractor. According to the NMCI Deputy Director
for Enterprise Operations, it would have been inappropriate to work
with individual competing contractors before the contract was awarded.
However, NMCI staff and the service provider are now working together
on SLA revisions.; Did the project use the practice?: NIMA IT/IS: Yes-
-According to NIMA and service provider officials, they used a
partnering contracting approach to jointly define, develop, and
structure the SLAs.
Practice: Specify circumstances under which the provider is excused
from performance levels mandated by master service agreements.; Did the
project use the practice?: Air Force C4 Services: Yes--The contract
includes a clause that the contractor is excused in the event of
government delay of work.; Did the project use the practice?: Army TAC-
SWA: Yes--The contract includes clauses that the provider is excused
from performance levels if there are government delays or factors
beyond its control.; Did the project use the practice?: MHS/ITO Help
Desk: Yes--For example, the provider's proposal, which is incorporated
as part of the contract, states that during transition periods, metrics
will not be reported for incentive and penalty purposes. As a result,
according to program officials, the provider would be excused from
required performance levels when baseline requirements are being
established for new applications.; Did the project use the practice?:
Navy NMCI: Yes--Contract terms and conditions specify the situations
where the service provider does not have to meet the SLAs. For example,
during transition, the service provider does not have to meet the
performance levels set by the SLAs.; Did the project use the practice?:
NIMA IT/IS: Yes--The contractor can be excused from meeting performance
levels with the permission of the contracting officer if circumstances
occur beyond the service provider's control.
Practice: Client and provider work together to identify SLAs for
which compensation is based, while additional SLAs may be defined to
manage performance.; Did the project use the practice?: Air Force C4
Services: Yes--According to Kirtland Air Force Base officials, the
service provider had input on the revised SLAs, including those for
compensation.; Did the project use the practice?: Army TAC-SWA: Not
applicable--SLAs are not used.; Did the project use the practice?: MHS/
ITO Help Desk: Yes--Before the contract was awarded, MHS did not work
with the contractor to identify SLAs for which compensation is based.
However, according to MHS and provider officials, after award, there
have been instances in which they have worked together on refining the
SLAs, including those affecting compensation.; Did the project use the
practice?: Navy NMCI: Yes--Meetings were held with potential bidders to
obtain their input in defining the SLAs, including discussions of
compensation. In addition, Navy and the provider are now refining the
SLAs, including those affecting compensation.; Did the project use the
practice?: NIMA IT/IS: Yes--NIMA and the service provider work together
in transformation teams to develop SLAs, including those affecting
compensation.
Practice: The contract should include clauses for; ; * determining
pricing structures;; * performing customer satisfaction surveys and
using the results to redefine performance levels;; * terminating the
contract, including early terminations;; * resolving contract disputes
in a timely manner;; * taking work away, without penalty, from the
provider for nonperformance;; * declaring a significant event that can
lead to a change in the contract;; * defining performance requirements;
and; * conducting regularly scheduled meetings.; Did the project use
the practice?: Air Force C4 Services: Limited--All clauses are included
in the contract, except a clause pertaining to customer satisfaction
surveys.; Did the project use the practice?: Army TAC-SWA: Yes--All
contract clauses are included.; Did the project use the practice?: MHS/
ITO Help Desk: Yes--All contract clauses are included.; Did the project
use the practice?: Navy NMCI: Yes--All contract clauses are included.;
Did the project use the practice?: NIMA IT/IS: Yes--All contract
clauses are included.
Practice: Consider setting up master services agreement under which
all arrangements between client and provider operate.; Did the project
use the practice?: Air Force C4 Services: Yes--According to Kirtland
Air Force Base officials, the contract, which incorporates the
contractor proposal, governs all arrangements between the government
and the contractor and is considered the master services agreement.;
Did the project use the practice?: Army TAC-SWA: Yes--According to TAC-
SWA project officials, the contract is considered the master services
agreement.; Did the project use the practice?: MHS/ITO Help Desk: Yes-
-According to MHS program officials, the contract is considered the
master services agreement.; Did the project use the practice?: Navy
NMCI: Yes--According to NMCI program officials, the contract is
considered to be a master services agreement.; Did the project use the
practice?: NIMA IT/IS: Yes--According to NIMA, the contract is
considered the master services agreement.
Practice: Include appropriate representation from each major
organizational unit on contract negotiation team.; Did the project use
the practice?: Air Force C4 Services: Not applicable--The C4 Services
contract was awarded under a sealed bid process and was not
negotiated.; Did the project use the practice?: Army TAC-SWA: Yes--
Representatives from the field and headquarters commands and the
contracting office participated in contract negotiation.; Did the
project use the practice?: MHS/ITO Help Desk: Yes--Staff from affected
program management offices and the chief information offices were on
the negotiation team.; Did the project use the practice?: Navy NMCI:
Yes--The contracting team was built with experts from each of the major
systems commands. Also, the source selection evaluation board consisted
of more than 50 people from various commands.; Did the project use the
practice?: NIMA IT/IS: Yes--The NIMA contract negotiating team included
representatives from each major organizational unit.
Practice: Specify the use of volume purchases to obtain optimal
discounts.; Did the project use the practice?: Air Force C4 Services:
Not applicable--According to Kirtland Air Force Base officials, such
volume purchases were not relevant to this contract.; Did the project
use the practice?: Army TAC-SWA: Not applicable--According to TAC-SWA
project officials, such volume purchases were not relevant to this
contract.; Did the project use the practice?: MHS/ITO Help Desk: Yes--
Call bands[C] are used to obtain optimal discounts on the number of
calls being fielded to the help desk.; Did the project use the
practice?: Navy NMCI: Yes--The contract includes volume discounts. For
example, seat moves, adds, and changes are purchased in bulk only,
because prices for these actions are lower when purchased in groups of
250.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA has
authorized the provider to use government sources, such as the Federal
Supply Schedule, in procuring products, services, and supplies related
to this contract. According to the contracting officer, this
authorization was made so that the provider could take advantage of
discounts available to the government.
Practice: Use third-party assistance in negotiating and developing
the contract.; Did the project use the practice?: Air Force C4
Services: No--This was not done because the Air Force staff believed
that it had adequate expertise available in-house.; Did the project use
the practice?: Army TAC-SWA: No--NETCOM officials believed that the
Army's designated contracting agency had adequate expertise and
experience.; Did the project use the practice?: MHS/ITO Help Desk: Yes-
-Private-sector and other government organizations provided
assistance.; Did the project use the practice?: Navy NMCI: Yes--The
Navy used private-sector firms to assist in developing the overall NMCI
concept and negotiation strategies as well as in drafting the contract
documents.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA
obtained help from (1) private contractors to help develop the
performance work statement, SLAs, and award fee plan;
(2) another agency on a particular contract technique; and (3) a
private-sector firm to compare the vendor's proposal with industry best
practices and trends and to attend some negotiation meetings to answer
questions.
Practice: Sign contract after contract negotiations and final vendor
selections.; Did the project use the practice?: Air Force C4 Services:
Yes--The contract was signed after negotiations and final vendor
selection.; Did the project use the practice?: Army TAC-SWA: Yes--The
contract was signed after negotiations and final vendor selection.; Did
the project use the practice?: MHS/ITO Help Desk: Yes--The contract was
signed after negotiations and final vendor selection.; Did the project
use the practice?: Navy NMCI: Yes--The contract was signed after
negotiations and final vendor selection.; Did the project use the
practice?: NIMA IT/IS: Yes--The contract was signed after negotiations
with the selected vendor.
Source: GAO.
[A] Requiring a provider to meet escalating requirements and focusing
on achieving results at the least cost are not mutually exclusive
goals. As the provider becomes more familiar with the client
organization, it may be able to exceed the original performance
requirements at the same, or possibly lower, cost.
[B] The NMCI contract was subsequently extended to 7 years, with an
option for an additional 3 years.
[C] Call bands are call volume ranges used to determine contractor
pricing.
[End of table]:
[End of section]
Appendix V: Projects' Implementation of Phase IV: Select the
Provider(s):
[See PDF for image]
[End of figure]
Critical to the success of any project to outsource IT services is the
identification of potential providers and the ultimate selection of a
provider(s) that will best meet the needs of the organization.
Developing a strategy that will lead to the selection of the "right
contractor" is especially important in a performance-based
acquisition.[Footnote 47] The overall success of the outsourcing
project requires the contractor to understand the performance-based
approach, know or develop an understanding of the organization's
requirement, have a history of performing exceptionally in the field,
and have the processes and resources in place to support the mission.
The seven practices in this phase that we used to evaluate the five
projects are as follows:
* Conduct research on the state of the market, vendors, and technology
before defining vendor selection criteria.
* Identify and evaluate various sourcing solutions (e.g., single
vendor, multivendor, and alliance).
* Define a process for selecting vendors to be providers.
* Define vendor selection and evaluation (acceptance) criteria at the
outset.
* When issuing an RFP, identify services with expected performance
levels and define client and provider roles and responsibilities.
* Use third-party assistance with expertise in a variety of outsourcing
arrangements when selecting provider(s), including developing the RFP.
* Conduct due diligence activities to verify vendor capabilities before
signing the contract.
As figure 9 illustrates, all five projects largely implemented the
practices.
Figure 9: Project Implementation of Phase IV: Select the Provider(s):
[See PDF for image]
Note: Not applicable--The practice was not relevant to the project's
particular circumstances.
No--The agency did not implement the practice. Limited--The agency
fully implemented some but not all aspects of the practice and did not
take alternative actions that fully satisfied the practice. Yes--The
agency fully implemented the practice or took an alternative action
that fully satisfied its intent.
[End of figure]
Table 7 provides detailed information on whether and how the projects
in our review implemented each of the seven practices in this phase.
Table 7: Summary of Projects' Use of Phase IV Practices:
Practice: Conduct research on state of the market, vendors, and
technology before defining vendor selection criteria.; Did the project
use the practice?: Air Force C4 Services: Yes--The Air Force advertised
in the Commerce Business Daily and interested parties submitted
statements of capabilities. This was done before the Air Force
identified the vendor selection criteria.; Did the project use the
practice?: Air Force C4 Services: Yes--A Commerce Business Daily notice
was published identifying TAC-SWA requirements to interested parties.
In addition, a draft RFP was issued before the final solicitation to
solicit comments from industry that might affect the requirements.
According to the contracting officer, comments received from industry
were incorporated into the final solicitation, where applicable.
Additionally, a preproposal conference was conducted in the overseas
operations location to inform industry representatives about issues and
the procurement strategy.; Did the project use the practice?: Army TAC-
SWA: Yes--MHS had a third-party contractor perform market research.;
Did the project use the practice?: MHS/ITO Help Desk: Yes--The Navy and
a private-sector firm performed market research to, for example, help
define the market conditions and vendor selection criteria. The Navy
also held meetings with leading companies that had outsourced IT
services on an enterprisewide level. Finally, more than 200 companies
participated in the July 1999 NMCI Industry Day Conference, which
informed companies about the NMCI vision, requirements, and procurement
strategy.; Did the project use the practice?: NIMA IT/IS: Limited--NIMA
staff and a contractor researched preferential providers before
defining vendor selection criteria. NIMA program officials noted that
they limited their analysis to such providers because they had
previously decided on a strategy to directly convert their IT/IS
activities to a preferential provider.
Practice: Identify and evaluate various sourcing solutions (e.g.,
single-vendor, multivendor, and alliance).; Did the project use the
practice?: Air Force C4 Services: Yes--As outlined in OMB Circular A-
76, the Air Force evaluated private-sector and internal government
staff proposals to satisfy the C4 services requirements.; Did the
project use the practice?: Air Force C4 Services: Yes--A multivendor
approach has been used in the past, which the Army opted not to
continue. Instead, NETCOM chose to consolidate its requirements and
contractor oversight by choosing a single-vendor solution.; Did the
project use the practice?: Army TAC-SWA: Yes--According to an MHS
program official, MHS evaluated continuing with the status quo, using
another federal agency's help-desk services as part of a cross-services
agreement, and contracting with a commercial firm. According to the MHS
program office, MHS chose a single-vendor solution to consolidate
requirements and performance accountability.; Did the project use the
practice?: MHS/ITO Help Desk: Yes--The Navy evaluated single-vendor and
multivendor approaches. A joint Navy and Marine Corps team determined
that single-vendor point of contact for accountability and
responsibility was critical to mission success.; Did the project use
the practice?: NIMA IT/IS: Yes--NIMA considered the sourcing solutions
allowed under OMB Circular A-76, including the direct conversion to a
preferential provider, holding a public/private competition, or
obtaining an agency cost comparison waiver. NIMA decided that directly
converting the selected IT functions to a preferential procurement
source[A] was the lowest risk to its mission and NIMA employees.
Practice: Define a process for selecting vendors to be providers (e.g.,
issuing an RFP and prequalifying vendors).; Did the project use the
practice?: Air Force C4 Services: Yes--As explained in the invitation
to bid, the Air Force followed the two-step process outlined in OMB
Circular A-76 for public/private competitions.; Did the project use the
practice?: Air Force C4 Services: Yes--The contract was awarded on
"best value" considerations of technical and management capabilities,
past performance, and price, as defined in the solicitation.; Did the
project use the practice?: Army TAC-SWA: Yes--MHS used a selection
process that included the prequalification of 13 industry leaders in
both medical and commercial help-desk operations. MHS also issued a
request for comment and a request for quote, which defined vendor
evaluation and selection criteria on the basis of technical approach,
past performance, key personnel qualifications, organizational
experience, and price.; Did the project use the practice?: MHS/ITO Help
Desk: Yes--The NMCI RFP used a negotiated commercial items evaluation
process. The vendor evaluation criteria included technical approach,
management plan, small business utilization, past performance, and
price.; Did the project use the practice?: NIMA IT/IS: Limited--NIMA's
decision to use a phased direct conversion of its IT/IS functions to a
statutory, preferential procurement vendor limited its vendor selection
process to the identification of potential Alaska Native Corporation
companies capable of performing the requirements. NIMA decided to take
this approach because it believed that it would avoid schedule delays
and mission risk that it thought would result from taking other
approaches. In addition, NIMA program officials believed that the
preferential provider approach would maximize the retention of
institutional knowledge whether employees stayed at NIMA or
transitioned to the provider.
Practice: Define vendor selection and evaluation (acceptance) criteria
at the outset.; Did the project use the practice?: Air Force C4
Services: Yes--The criteria were included in the invitation to bid.;
Did the project use the practice?: Air Force C4 Services: Yes--The
solicitation included vendor evaluation and selection criteria. The
source selection evaluation plan describes the evaluation process.; Did
the project use the practice?: Army TAC-SWA: Yes--Vendor evaluation and
selection were based upon industry-defined help-desk criteria. MHS used
a multistep process that included an assessment of minimum
qualifications, evaluation of written technical and cost proposals, and
oral presentations for qualified vendors.; Did the project use the
practice?: MHS/ITO Help Desk: Yes--The final RFP required bidders to
have implemented and provided service to at least 100,000 seats, of
which 20,000 were to be on the same contract. In addition, bidders were
evaluated on their technical approach, management plan, small business
utilization, past performance, and price. Finally, vendors in the
competitive range had to provide demonstrations of technical network
operations centers to verify their ability to achieve required service
levels set forth in technical proposals.; Did the project use the
practice?: NIMA IT/IS: Yes--NIMA evaluated the past performance and
financial capabilities of Alaska Native Corporation companies.
According to NIMA, only two companies had demonstrated successful
performance in service environments similar to NIMA's requirements.
However, to be able to fully meet NIMA's requirements, these vendors
formed a joint venture, which was awarded the contract.
Practice: When issuing an RFP, identify services with expected
performance levels and define client and provider roles and
responsibilities.; Did the project use the practice?: Air Force C4
Services: Yes--The services and performance levels were included in the
original performance work statement included in the invitation to bid.;
Did the project use the practice?: Air Force C4 Services: Yes--The RFP
identified the government and contractor roles and responsibilities,
and system operational and availability requirements.; Did the project
use the practice?: Army TAC-SWA: Yes--The request for quote identified
services and expected performance levels and specified a performance-
based, incentivized, shared-risk relationship with the service
provider. It also defined client and provider roles and
responsibilities.; Did the project use the practice?: MHS/ITO Help
Desk: Yes--The NMCI solicitation documents identified the required
services and expected performance levels. These documents, along with
the contract, define the client and provider roles and
responsibilities.; Did the project use the practice?: NIMA IT/IS: Yes-
-Because this was a sole source contract, an RFP was not issued.
However, according to NIMA and provider officials, they worked jointly
to refine the requirements and expected performance levels and to
define client and provider roles and responsibilities in the contract.
Practice: Use third-party assistance with expertise in a variety of
outsourcing arrangements when selecting provider(s), including
developing the RFP.; Did the project use the practice?: Air Force C4
Services: No--This was not done because the Air Force believed that
adequate in-house expertise was available.; Did the project use the
practice?: Air Force C4 Services: No--The Army believed that it had
adequate government experience and expertise covering technical,
resource management, and contracting areas.; Did the project use the
practice?: Army TAC-SWA: Yes--A private-sector contractor, the Defense
Acquisition University, and GovWorks (an Interior organization offering
procurement services to government agencies) provided assistance in
this area.; Did the project use the practice?: MHS/ITO Help Desk: Yes-
-A private-sector firm assisted with market research and developing
vendor pass/fail criteria. Another contractor assisted the Navy with
developing SLAs and the technical evaluation of vendors.; Did the
project use the practice?: NIMA IT/IS: Yes--NIMA used a private-sector
firm and the NIMA Acquisition Center to support the evaluation of the
vendor proposal.
Practice: Conduct due diligence activities to verify vendor
capabilities before signing the contract.; Did the project use the
practice?: Air Force C4 Services: Yes--The Air Force performed a
technical evaluation of the vendor's capabilities. Also, as part of its
due diligence activities, the Air Force reviewed the debarred list and
DOD's Central Contractor Registry.; Did the project use the practice?:
Air Force C4 Services: Yes--NETCOM evaluated the contractor's financial
and past performance information before signing the contract.; Did the
project use the practice?: Army TAC-SWA: Yes--Minimum vendor
qualifications were established and evaluated early in the process for
selecting the provider.; Did the project use the practice?: MHS/ITO
Help Desk: Yes--As part of due diligence, the Navy ensured that all
bidders had relevant experience implementing large seat management
contracts. In addition, past performance, including reference checks,
was a source-selection evaluation factor.; Did the project use the
practice?: NIMA IT/IS: Yes--NIMA researched both parent companies of
the new joint venture corporation to ensure that they had financial and
technical backing.
Source: GAO.
[A] Preferential procurement programs are special commercial source
programs, such as Federal Prison Industries and the workshops
administered by the Committee for the Purchase from the Blind and Other
Severely Handicapped under the Javis-Wagner O'Day Act.
[End of table]:
[End of section]
Appendix VI: Projects' Implementation of Phase V: Transition to
Provider(s):
[See PDF for image]
[End of figure]
In this phase, client organizations transfer responsibility of IT
functions to one or more providers. A clear definition of
responsibilities among the various parties and the careful
consideration of employees' needs matched against the organization's
needs enable both the client and provider to focus on execution and
give staff confidence in their future employment. If the contractor is
assuming responsibility for functions previously performed by federal
employees, it is especially important that the organization communicate
a clear transition process. Without such communication, an outsourcing
project can be negatively affected if misinformation and mistrust
ensues.
The 11 practices in this phase that we used to evaluate the five
projects are as follows:
* Communicate a clear transition process to all key players from both
client and provider organizations.
* Clearly communicate to employees what is going to happen and when it
is going to happen.
* Establish a client transition team with representatives from across
the organization to facilitate the transition.
* Place the transition under a single program manager.
* Create client/provider transition teams to address short-term
transition tasks as required.
* Recognize that it takes time to effect transition and plan
accordingly.
* Encourage the transition of staff to the provider, where appropriate,
using bonuses, stock options, and other appropriate methods.
* Develop employee-retention programs and offer bonuses to keep key
people, where appropriate.
* When consistent with organizational objectives, assist employees who
do not want to transfer in finding other jobs, either within an
organization or at another organization.
* Document key information to preserve organizational knowledge in the
event that one or more providers change.
* Use change management strategies to help client employees deal with
the transition.
Figure 10 demonstrates that the five projects in our review were
largely using the practices.
Figure 10: Project Implementation of Phase V: Transition to
Provider(s):
[See PDF for image]
Note: Not applicable--The practice was not relevant to the project's
particular circumstances.
No--The agency did not implement the practice. Limited--The agency
fully implemented some but not all aspects of the practice and did not
take alternative actions that fully satisfied the practice. Yes--The
agency fully implemented the practice or took an alternative action
that fully satisfied its intent.
[End of figure]
Table 8 provides information on whether and how the projects
implemented each of the 11 practices in this phase.
Table 8: Summary of Projects' Use of Phase V Practices:
Practice: Communicate a clear transition process to all key players
from both client and provider organizations.; Did the project use the
practice?: Air Force C4 Services: No--There were two transition plans
prepared, one by the government and one by the provider. However, the
government's plan contained conflicting direction. In addition,
provider officials said that the two separate transition plans were not
well coordinated and that the Air Force did not have a strong advocate
to ensure that the transition process was well planned and executed.
Both provider and Kirtland Air Force Base officials acknowledged that
there were problems with the transition process, including incorrect
information about upgrades that would be made before the transition and
staff morale problems that hampered knowledge transfer to the
provider.; Did the project use the practice?: Army TAC-SWA: Yes--The
provider developed a phase-in plan that addressed personnel issues, the
integration of added locations, updating plans, and finalizing new
subcontract/
sponsorship arrangements. This phase-in plan was discussed at a
postaward conference, attended by representatives from both the client
and provider.; Did the project use the practice?: MHS/ITO Help Desk:
Yes--The contract delivery order and the provider's transition plan
laid out a transition process. In addition, a kickoff meeting was held
between the client and provider to communicate the transition process
to all key players.; Did the project use the practice?: Navy NMCI: Yes-
-The Navy developed numerous guides and Web sites to communicate with
the NMCI community about site, technical, and personnel transition
processes, procedures, and tasks.; Did the project use the practice?:
NIMA IT/IS: Yes--Several NIMA transformation teams were formed,
consisting of both client and provider personnel, to develop a single
transition plan. The plan was developed to formalize communications
with all key players.
Practice: Clearly communicate to employees what is going to happen and
when it is going to happen.; Did the project use the practice?: Air
Force C4 Services: Yes--The Air Force kept its employees informed of
the outsourcing project through town hall meetings and internal
briefings. As part of this process, employees were informed about the
reduction-in-force process and the procedure for those interested in
being transitioned or reassigned.; Did the project use the practice?:
Army TAC-SWA: Limited--According to TAC-SWA officials, the Army
communicated with its employees, but they did not provide documentation
supporting any communication efforts. However, the provider was
required to submit a phase-in plan, which included a section on
communicating with the incumbent contractors' staff.; Did the project
use the practice?: MHS/ITO Help Desk: Yes--The government task manager
for the Tri-Service Infrastructure Management Office held regular
meetings with MHS program officials to discuss transition issues. The
MHS program office also periodically sent global E-mails to MHS staff
on transition activities and MHS leadership made on-site visits to San
Antonio, the location of the help-desk function.; Did the project use
the practice?: Navy NMCI: Limited--The Navy has used its normal chain
of command to communicate transition information, but found that the
implementation was uneven. As a result, some staff did not know current
information about how NMCI would affect them until the provider was
ready to contact them regarding their possible transition to the
contractor. However, according to the NMCI Director's office, this
problem was somewhat mitigated by the provider's Web site that provides
transition information to all NMCI customers/users.; Did the project
use the practice?: NIMA IT/IS: Yes--With help from a contractor, NIMA
developed a communications plan. The plan included town hall meetings,
global
E-mail messages, and a Web site posting that provided employees with
access to transition information.
Practice: Establish client transition team with representatives from
across the organization to facilitate the transition.; Did the project
use the practice?: Air Force C4 Services: Yes--The government's
transition team included personnel from the functional areas being
outsourced (i.e., communications and IT services) as well as personnel
from other offices.; Did the project use the practice?: Army TAC-SWA:
Yes--According to TAC-SWA officials, the contracting officer's
representative assisted the provider with the transition. The officials
also stated that Army technical points of contact at each site also
helped facilitate the transition to the new contract.; Did the project
use the practice?: MHS/ITO Help Desk: Yes--MHS program officials for
the Tri-Service Infrastructure Management Office established
transition teams with representatives from the former Tri-Service
Medical System Support Center contract.; Did the project use the
practice?: Navy NMCI: Yes--The headquarters-level transition team
consists of representatives from the NMCI Director's Office and the
program management office. Also, every site has a transition team made
up of customer and provider personnel.; Did the project use the
practice?: NIMA IT/IS: Yes--NIMA's transition team included
representatives from various offices within NIMA to help facilitate the
transition.
Practice: Place transition under single program manager.; Did the
project use the practice?: Air Force C4 Services: Yes--The Air Force
established a single program manager, who is referred to as the
functional area chief.; Did the project use the practice?: Army TAC-
SWA: Yes--The Army designated the contracting officer's representative
to oversee the transition.; Did the project use the practice?: MHS/ITO
Help Desk: Yes--The transition was assigned to the government task
manager in the Tri-Service Infrastructure Management Office.; Did the
project use the practice?: Navy NMCI: Yes--NMCI's Director has overall
responsibility for the transition.; Did the project use the practice?:
NIMA IT/IS: Yes--The transition process is under a single program
manager.
Practice: Create client/provider transition teams to address short-term
transition tasks as required.; Did the project use the practice?: Air
Force C4 Services: Yes--During the transition phase, management from
both the government and the provider met weekly to track the progress
of the transition.; Did the project use the practice?: Army TAC-SWA:
Yes--Personnel from both the Army and provider formed three transition
teams, one for each site location. These teams worked with the
contracting officer's representative and the provider's project manager
to deal with transition issues.; Did the project use the practice?:
MHS/ITO Help Desk: Yes--Client/provider transition teams were
established to address short-term transition tasks. The teams met
weekly to address any transition issues.; Did the project use the
practice?: Navy NMCI: Yes--NMCI and the provider designated staff to
work together on short-term transition issues.; Did the project use the
practice?: NIMA IT/IS: Yes--An integrated NIMA transformation team
comprising NIMA and provider staff, was responsible for addressing
short-term tasks, including (1) transition planning, (2) resources and
recruitment, (3) program budget, and (4) contract development and
costing.
Practice: Recognize that it takes time to effect transition and plan
accordingly.; Did the project use the practice?: Air Force C4 Services:
Yes--The Air Force allowed 60 days for the transition.; Did the project
use the practice?: Army TAC-SWA: Yes--The contract allowed a 60-day
phase-in period.; Did the project use the practice?: MHS/ITO Help Desk:
Yes--The contract provided for a transition period of 30 to 90 days.
According to MHS program officials, the transition period lasted 90
days.; Did the project use the practice?: Navy NMCI: Limited--The Navy
said that it had initially underestimated the scope of the project and
the magnitude of the problems brought on by legacy applications and
associated information assurance and cyber-security issues. According
to NMCI program officials, addressing these problems and completing the
operational testing that was mandated by the Congress subsequent to
contract award led to the transition period being lengthened from 2-1/
2 to
3-1/2 years. NMCI officials also said the extension will allow the
Department of the Navy to have time to operate NMCI as a fully
transitioned enterprise before having to decide whether to exercise the
contract option.; Did the project use the practice?: NIMA IT/IS: Yes--
According to NIMA, it has scheduled its seven functional areas to be
transitioned to the provider over a 3-to 5-year period. The provider
will not assume responsibility for a given functional area until the
NIMA transition review board has given its approval to the provider's
turnover plan.
Practice: Encourage transition of staff to provider, where appropriate,
using bonuses, stock options, and other appropriate methods.; Did the
project use the practice?: Air Force C4 Services: Yes--The Air Force
staff were given the right to transfer to the contractor. It was left
up to the contractor to decide whether to offer incentives or not,
which it chose not to do.; Did the project use the practice?: Army TAC-
SWA: Yes--The Army had no personnel expected to transition to the
provider since the activity being outsourced had previously been
contracted out. However, the provider was interested in retaining
existing contractor staff and had a program to recruit them.; Did the
project use the practice?: MHS/ITO Help Desk: Yes--According to
provider officials, they extended offers to employees from the prior
contractor and some MHS staff.; Did the project use the practice?: Navy
NMCI: Yes--The NMCI contract provides that displaced federal employees
that transition to the provider under employment openings as a result
of NMCI will be given guaranteed 3 years of employment with the
provider, a 15 percent salary increase, and a sign-on bonus.; Did the
project use the practice?: NIMA IT/IS: Yes--The contract requires the
provider to actively recruit, hire, and make reasonable efforts to
retain NIMA staff. If the provider meets certain target thresholds for
recruiting NIMA staff, it receives a monetary incentive. According to
the provider's general manager, the company offered signing bonuses to
NIMA employees that wanted to transition.
Practice: Develop employee-retention programs and offer bonuses to keep
key people, where appropriate.; Did the project use the practice?: Air
Force C4 Services: No--According to Kirtland Air Force Base officials,
they did not develop an employee-retention program because they did not
believe that they could offer incentives.; Did the project use the
practice?: Army TAC-SWA: Not applicable--Since this activity was
previously contracted out, employee retention of Army civilian
personnel was not relevant.; Did the project use the practice?: MHS/ITO
Help Desk: Yes--The government task manager persuaded key government
employees from the Tri-Service Medical System Support Center to
temporarily work at the Tri-Service Infrastructure Management Office
until positions became available with a contractor that provides
support to this office. In addition, according to a program official,
to keep expertise in particular applications, key employees were
transferred to the MHS program executive office and Tri-Service
Infrastructure Management Office.; Did the project use the practice?:
Navy NMCI: Yes--NMCI program officials noted that the Navy has human
resources programs that would allow its organizations a range of
options, including bonuses, to help keep key staff. Additionally, the
Navy directed all Navy and Marine Corps commands affected by NMCI to
develop civilian personnel transition plans to mitigate the impact of
NMCI implementation on employees.; Did the project use the practice?:
NIMA IT/IS: Yes--According to NIMA program officials, NIMA categorized
affected employees into three tiers, on the basis of whether they would
be allowed to transition to the provider or whether there would be
restrictions on their activities if they did transition. This
distinction was made to comply with government conflict of interest
rules. According to the program manager, NIMA also sought personnel who
wanted to remain at the agency to help monitor the provider's
performance and considered the qualifications of those who wished to
serve in this role prior to deciding who to retain.
Practice: When consistent with organizational objectives, assist
employees who do not want to transfer in finding other jobs, either
within an organization or at another organization.; Did the project use
the practice?: Air Force C4 Services: Yes--Kirtland Air Force Base
officials stated that they have reduction-in-force procedures to help
place civilian employees in other Air Force jobs. Employees also
receive priority status for other DOD job openings. Further, the base
education office provided assistance with outplacement; résumé writing;
and, for those nearing retirement, planning advice.; Did the project
use the practice?: Army TAC-SWA: Not applicable--According to the TAC-
SWA contracting officer, no Army personnel were displaced by the
contract.; Did the project use the practice?: MHS/ITO Help Desk: Yes--
According to MHS program officials, the government task manager
assisted government employees with the Tri-Service Medical System
Support Center staff in finding new positions with other MHS
organizations. For example, he located open positions and contacted the
organizations.; Did the project use the practice?: Navy NMCI: Yes--NMCI
program officials said that existing Navy civilian personnel programs
are in place to assist employees. They noted that two Navy
organizations found other jobs within their command for their small
number of employees affected by NMCI. Other Navy organizations are also
reassigning affected personnel to other government jobs, where
applicable.; Did the project use the practice?: NIMA IT/IS: Yes--For
staff that did not want to transition, NIMA provided various types of
assistance, such as retirement planning, résumé writing, and
interviewing skills. In addition, NIMA reported that if an employee did
not want to transition to the provider, that it would consider re-
adjusting the employee's work assignment or provide training to support
the individual's placement within other areas.
Practice: Document key information to preserve organizational knowledge
in the event that one or more providers change.; Did the project use
the practice?: Air Force C4 Services: Yes--Kirtland Air Force Base's
work functions and workload size are documented in the contract.; Did
the project use the practice?: Army TAC-SWA: Yes--Documentation
produced by the provider, including maintenance logs, become the
property of the government.; Did the project use the practice?: MHS/ITO
Help Desk: Yes--The Tri-Service Medical System Support Center processes
and procedures were captured during transition. The provider's
processes and procedures are currently being captured. The contract
also includes requirements on documenting key information.; Did the
project use the practice?: Navy NMCI: Yes--NMCI maintains a Web portal
with organizational knowledge information. In addition, a new Intranet
site is being developed with up-to-date interactive organizational
knowledge relevant to the many varied NMCI communities.; Did the
project use the practice?: NIMA IT/IS: Yes--The executing NIMA
transformation team was charged with documenting the activities for
each functional area that is being outsourced. This includes developing
and capturing operational processes and procedures for each of the
functional areas.
Practice: Use change management strategies to help client employees
deal with the transition.; Did the project use the practice?: Air Force
C4 Services: Yes--Kirtland Air Force Base held town hall meetings and
internal briefings to communicate the objectives of the outsourcing
project and the changes that would take place internally. It also had
one-on-one meetings with each employee that would be displaced.; Did
the project use the practice?: Army TAC-SWA: Not applicable--The
activity had been previously contracted out.; Did the project use the
practice?: MHS/ITO Help Desk: Yes--According to MHS program officials,
the government task manager for the Tri-Service Infrastructure
Management Office assisted government employees with finding new
positions and kept them informed of transition issues. Weekly
E-mails were also sent out to all affected employees.; Did the project
use the practice?: Navy NMCI: Yes--The Navy has provided employees
affected by NMCI with a range of employment options and incentives.;
Did the project use the practice?: NIMA IT/IS: Yes--NIMA's change
management strategies included training seminars offered by its human
resources office on résumé writing, interviewing skills, career
transition workshops, and employment trends.
Source: GAO.
[End of table]:
[End of section]
Appendix VII: Projects' Implementation of Phase VI: Manage Provider(s)
Performance:
[See PDF for image]
[End of figure]
The effectiveness with which the performance of the provider(s) is
managed--the focus of this phase--is critical to the successful
implementation of an outsourcing project. Indeed, according to Gartner,
Inc., an outsourcing project can be thwarted by poorly designed,
funded, and delivered processes for managing the delivery of
services.[Footnote 48] This firm also points out that an enterprise
needs to retain the resources to oversee the planning and
implementation of the IT services being delivered by the provider to
ensure that the contractor meets the client's business needs throughout
the life of the agreement.[Footnote 49] Moreover, frequent and clear
communication between the client and provider ensures that potential
problems are resolved before they cause disruptions. In addition,
performance reviews should take place regularly to keep the project on
course, measure performance levels, and make adjustments as
necessary.[Footnote 50]
The 11 practices in this phase that we used to evaluate the five
projects are as follows:
* Consider incentives to motivate provider(s) to exceed performance
requirements.
* Use penalties to motivate provider(s) to meet performance
requirements.
* Periodically undertake studies to assess (1) how the provider's
performance compares with the value being delivered to similar clients
and (2) the extent to which the provider's performance is improving
over time.
* Schedule periodic working-level meetings with both the end-user
groups and the provider to review the provider's performance.
* Conduct executive-level oversight meetings with the provider's senior
management to review provider's performance.
* Distribute performance data to stakeholders.
* Reserve audit rights on performance data supplied by the provider.
* Ensure that the provider measures and reports on performance.
* Work with the provider to redefine service levels, as appropriate.
* Sample performance data frequently enough to perform trend analysis
and to permit extrapolation based on historical data.
* Allow employees and possibly stakeholders to rate the provider on a
regular basis.
As shown by figure 11, the five projects in our review generally
implemented the practices.
Figure 11: Project Implementation of Phase VI: Managing Provider
Performance:
[See PDF for image]
Note: Not applicable--The practice was not relevant to the project's
particular circumstances.
No--The agency did not implement the practice. Limited--The agency
fully implemented some but not all aspects of the practice and did not
take alternative actions that fully satisfied the practice. Yes--The
agency fully implemented the practice or took an alternative action
that fully satisfied its intent.
[End of figure]
Table 9 depicts whether and how the five projects in our review
implemented each of the 11 practices in this phase.
Table 9: Summary of Projects' Use of Phase VI Practices:
Practice: Consider incentives to motivate provider(s) to exceed
performance requirements.; Did the project use the practice?: Air Force
C4 Services: No--The Air Force did not include incentives in the
contract because the agency expects the contractor to meet the
contract's requirements without incentives. Specifically, according to
the former functional area chief, because the contract was issued under
the rules established by OMB Circular A-76, the focus of the project
was on achieving cost savings, and incentives were not included in the
contract. As a result, he noted that the provider did not have any
incentive to be innovative or creative.; Did the project use the
practice?: Army TAC-SWA: No--There are no monetary incentives, but,
according to TAC-SWA officials, the fact that the agency will prepare a
performance evaluation report can help motivate a provider to meet
requirements. However, the contracting officer said that incentives
might have been useful to motivate the provider to exceed performance
requirements.; Did the project use the practice?: MHS/ITO Help Desk:
Yes--The contract includes incentives linked to each performance
requirement. Each requirement has a positive, negative, and acceptable
performance range that provides a basis for monetary incentives (as
well as penalties).; Did the project use the practice?: Navy NMCI: Yes-
-The NMCI contract includes a one-time payment of $10 million if the
provider successfully completes full operational capability. The
contract also has incentives for customer satisfaction, information
assurance, and small business and small disadvantaged business
participation.; Did the project use the practice?: NIMA IT/IS: Yes--The
contract includes monetary incentives in accordance with an award fee
plan. It also includes share-in-savings provisions to encourage process
improvements.
Practice: Use penalties to motivate provider(s) to meet performance
requirements, such as; ; * assess penalties for failure to perform at
required individual service as well as aggregate service levels;; *
apply penalties in the form of credit to the client;; * increase
penalty for recurring deficient performance;; * hold back a percentage
of provider's pay for a particular service until performance
requirements are met;; * refund a penalty if the provider returns to
agreed-upon performance levels within a designated period of time; and;
* ensure that the provider will cover costs, but not profit, when a
particular performance requirement is not met.; Did the project use the
practice?: Air Force C4 Services: Yes--The contract contains monetary
penalties that are linked to each of the performance requirements. If
imposed, they would reduce the amount of the payment owed to the
provider.; Did the project use the practice?: Army TAC-SWA: Limited--
The contract does not have specific monetary penalties linked to
performance requirements. However, the contract states that
unacceptable work must be redone at the provider's expense, and, if the
defects and services cannot be corrected, the government may reduce the
contract's price to reflect the reduced value of the services
performed. TAC-SWA project officials stated that monetary penalties
were not included in the contract because the Army was concerned that
contractors might not bid on the contract, and administrating this type
of contract is more difficult.; Did the project use the practice?: MHS/
ITO Help Desk: Yes--The contract includes monetary penalties linked to
each performance requirement. Each requirement has positive, negative,
and acceptable performance ranges that provide a basis for monetary
penalties as well as incentives.; Did the project use the practice?:
Navy NMCI: Yes--The contract includes monetary penalties in the form of
credits to the agency if the provider fails to perform to the levels
specified in the SLAs.; Did the project use the practice?: NIMA IT/IS:
Limited--NIMA did not include monetary penalties in its contract.
However, if the provider does not meet certain minimum performance
standards, only its costs would be covered.
Practice: Periodically undertake studies to assess: (1) how the
provider's performance compares with the value being delivered to
similar clients and (2) the extent to which the provider's performance
is improving over time.; Did the project use the practice?: Air Force
C4 Services: Limited--(1) This has not been done because, according to
a project official, the Air Force was not aware of similar clients.;
(2) This is done through periodic monitoring of performance by quality
assurance evaluators and through monthly review meetings held by the
functional area chief to identify and address any problems that are
starting to occur.; Did the project use the practice?: Army TAC-SWA:
No--TAC-SWA officials stated that they address performance issues at
the time that they occur and that they have not performed any studies.;
Did the project use the practice?: MHS/ITO Help Desk: Yes--The
provider's performance is reviewed monthly by the government task
manager and in quarterly management reviews of the provider's
performance. According to MHS program officials, the results are
compared with peers and reviewed for how the provider has improved over
time.; Did the project use the practice?: Navy NMCI: Limited--(1) NMCI
only has performance data since October 2002. The Navy intends to do
such a study but has not established a schedule for it.; (2) The
provider's improvements over time are being monitored monthly.; Did the
project use the practice?: NIMA IT/IS: Limited--(1) At this time, NIMA
has not undertaken such studies because it only recently transitioned
functions to the provider. However, NIMA program officials stated that
they intend to use a private-sector firm to periodically compare the
provider's performance with those of similar organizations.; (2) The
provider's improvements over time are being monitored monthly.
Practice: Schedule periodic working-level meetings with both the end-
user groups and the provider to review the provider's performance.; Did
the project use the practice?: Air Force C4 Services: Yes--Monthly
meetings are held at which the contractor briefs the functional area
chief and other Kirtland Air Force Base representatives.; Did the
project use the practice?: Army TAC-SWA: Yes--According to the provider
and TAC-SWA officials, they hold daily meetings to discuss any issues.;
Did the project use the practice?: MHS/ITO Help Desk: Yes--Periodic
working-level meetings are held with the government task manager and
the program management office representatives to obtain help-desk
feedback.; Did the project use the practice?: Navy NMCI: Yes--At each
implemented site, the provider and government managers meet on a
frequent, as-needed basis to review performance and resolve any
issues.; Did the project use the practice?: NIMA IT/IS: Yes--Quarterly
performance management review meetings are held and are attended by
NIMA and the provider. In addition, the provider hired an ombudsman to
obtain anonymous comments from NIMA staff and management on its
performance.
Practice: Conduct executive-level oversight meetings with the
provider's senior management to review provider's performance.; Did the
project use the practice?: Air Force C4 Services: Yes--Air Force
schedules executive-level oversight meetings whenever they are
necessary. A recent meeting included the president of the provider.;
Did the project use the practice?: Army TAC-SWA: Yes--According to
provider and TAC-SWA officials, quarterly in-progress reviews are held
with the provider's corporate management and the battalion commander.;
Did the project use the practice?: MHS/ITO Help Desk: Yes--Executive
oversight meetings are held quarterly with the provider's senior
management to review performance.; Did the project use the practice?:
Navy NMCI: Yes--The NMCI Director and the provider executive meet
weekly to review performance and discuss other NMCI implementation
issues. In addition, the Department of the Navy recently established an
operations advisory board consisting of Department of the Navy
leadership and the provider. The goal of this board is to focus senior
leadership on issues affecting NMCI in order to establish priorities
and make decisions.; Did the project use the practice?: NIMA IT/IS:
Yes--This is done through the quarterly review meetings.
Practice: Distribute performance data to stakeholders.; Did the project
use the practice?: Air Force C4 Services: Yes--Performance data were
not distributed to stakeholders at the beginning of the project, but
began after complaints from stakeholders about the need for such
information. Metrics, such as system reliability or "uptime," are now
provided to stakeholders on line.; Did the project use the practice?:
Army TAC-SWA: Yes--The contracting officer's representative
distributes monthly performance reports to Army stakeholders for
review. Performance issues or problems identified are discussed with
the contracting officer's representative and the contracting officer,
and corrective actions are planned to prevent reoccurrence.; Did the
project use the practice?: MHS/ITO Help Desk: Yes--The help-desk
monthly reports, including results against performance metrics, are e-
mailed to stakeholders monthly.; Did the project use the practice?:
Navy NMCI: Yes--The NMCI Navy and Marine Corps program managers are
responsible for providing SLA performance data to their commands.; Did
the project use the practice?: NIMA IT/IS: Yes--Performance data are
distributed at quarterly review meetings, which according to the
contracting officer, are attended by stakeholders.
Practice: Reserve audit rights on performance data supplied by the
provider.; Did the project use the practice?: Air Force C4 Services:
Yes--Addressed in the contract.; Did the project use the practice?:
Army TAC-SWA: Yes--Addressed in the contract.; Did the project use the
practice?: MHS/ITO Help Desk: Yes--Addressed in the contract.; Did the
project use the practice?: Navy NMCI: Yes--Addressed in the contract.;
Did the project use the practice?: NIMA IT/IS: Yes--Addressed in the
contract.
Practice: Ensure that provider measures and reports on performance.;
Did the project use the practice?: Air Force C4 Services: Yes--As
required by the quality control plan in the contract, the provider
makes information available on product or service quality and any
actions needed to correct decreases in quality. In addition, according
to the former functional area chief, the provider provides performance
information during monthly meetings.; Did the project use the
practice?: Army TAC-SWA: Yes--The contracting officer's representative
receives performance information in daily meetings with the provider's
project manager.; Did the project use the practice?: MHS/ITO Help Desk:
Yes--The contract requires the provider to submit monthly and quarterly
performance reports that are then reviewed and verified.; Did the
project use the practice?: Navy NMCI: Yes--The contract specifies that
the provider must measure and report on whether its SLA performance
goals are being met.; Did the project use the practice?: NIMA IT/IS:
Yes--The contract requires the provider to submit data, including
measurements of service, quarterly.
Practice: Work with provider to redefine service levels, as
appropriate.; Did the project use the practice?: Air Force C4 Services:
Yes--This is done on an as-needed basis. For example, the service
levels were redefined in April 2002.; Did the project use the
practice?: Army TAC-SWA: Yes--According to TAC-SWA and provider
officials, they have worked together to redefine performance
requirements in the contract.; Did the project use the practice?: MHS/
ITO Help Desk: Yes--MHS and the provider have worked together to
redefine SLAs. For example, the first call closure performance metric
was redefined to be more realistic.; Did the project use the practice?:
Navy NMCI: Yes--The Navy and service provider have and continue to
refine NMCI SLAs to ensure more precise performance measures and to
more accurately capture user satisfaction with the system. For example,
a contract modification standardized many of the performance categories
that dealt with service availability.; Did the project use the
practice?: NIMA IT/IS: Yes--The redefinition of service levels is
provided for in the contract. According to NIMA, the redefinition of
service levels is expected to occur in the future, generally at the
time that the annual contract option is exercised.
Practice: Sample performance data frequently enough to perform trend
analysis and to permit extrapolation based on historical data.; Did the
project use the practice?: Air Force C4 Services: Yes--Air Force
quality assurance evaluators monitor performance in accordance with the
quality assurance surveillance plan and the performance requirements
summary. Any trends identified are addressed in monthly status
meetings.; Did the project use the practice?: Army TAC-SWA: Yes--The
provider's proposal and quality control plan state that it will provide
various trend analyses to the government. According to TAC-SWA project
officials, these data are submitted to the contracting officer's
representative for analysis.; Did the project use the practice?: MHS/
ITO Help Desk: Yes--Performance data are sampled and reported monthly
and quarterly. They are analyzed and verified, including any supporting
data. In addition, the provider's knowledge management system provides
analysis and trend data to MHS.; Did the project use the practice?:
Navy NMCI: Yes--The NMCI Director monitors provider performance to
identify trends by assessing provider-supplied information as well as
information from Navy independent verification and validation testing
and customer satisfaction survey results. In addition, NMCI program
officials said that they have requested funding to develop a
performance measurement data repository to support trend analyses.; Did
the project use the practice?: NIMA IT/IS: Yes--The contract requires
data to be provided periodically for NIMA to use for trend and other
types of analyses.
Practice: Allow employees and possibly stakeholders to rate the
provider on a regular basis (e.g., scorecards and quarterly report
cards).; Did the project use the practice?: Air Force C4 Services:
Limited--The provider conducts annual customer satisfaction surveys,
but it is not required to submit the results to the Air Force. Kirtland
Air Force Base also obtains ad hoc feedback from employees who, after
they report a problem, are asked to provide information about how well
the problem was addressed and their overall satisfaction level.; Did
the project use the practice?: Army TAC-SWA: Yes--The contracting
officer's representative and on-site points of contact obtain feedback
from employees on a regular basis.; Did the project use the practice?:
MHS/ITO Help Desk: Yes--The customer satisfaction survey is one method
used to rate the provider. In addition, according to MHS program
officials, stakeholder input is also obtained from program management
reviews and meetings held by the government task manager.; Did the
project use the practice?: Navy NMCI: Yes--User surveys are used to
measure satisfaction with specific services such as E-mail access, help
desk, etc., and overall user satisfaction with the service provider's
performance.; Did the project use the practice?: NIMA IT/IS: Yes--The
provider surveys NIMA employees on their satisfaction with the help-
desk function. The results of these surveys are reviewed by NIMA during
quarterly meetings. In addition, stakeholders are also responsible for
providing monthly assessments of contractor performance. Moreover,
customer satisfaction is a factor in determining the amount of the
semiannual award fee earned by the provider.
Source: GAO.
[End of table]:
[End of section]
Appendix VIII: Projects' Implementation of Phase VII: Ensure Services
are Provided:
[See PDF for image]
[End of figure]
Although outsourcing focuses on the provider's ability to perform, the
client organization is ultimately responsible for ensuring that
services are provided and that end-user needs are met. The previous
phases addressed the extensive preparation that must precede the
provider's assuming responsibility for the client organization's
services. This phase encompasses practices designed to ensure that an
effective oversight approach is established. In addition, over the
course of the outsourcing project, the client learns more about the
capabilities of the provider, and market conditions may change. As a
result, it is important to monitor service levels internally as well as
maintain an external view of the performance of other providers in
order to identify opportunities to improve and ensure that the
outsourcing arrangement maintains its value to the client.
The six practices in this phase that we used to evaluate the five
projects are as follows:
* Monitor the provider's work to anticipate issues for resolution.
* Make sure that the provider uses the standard tools and processes
defined as part of the operational model.
* Use provider performance data to continuously improve processes.
* Pursue improvement based on customer satisfaction surveys.
* Ensure that an appropriately empowered individual from the client
organization oversees the work.
* Set realistic time frames that are agreed to by the provider.
As illustrated by figure 12, the five projects in our review largely
implemented the practices.
Figure 12: Project Implementation of Phase VII: Ensure Services Are
Provided:
[See PDF for image]
Note: Not applicable--The practice was not relevant to the project's
particular circumstances.
No--The agency did not implement the practice. Limited--The agency
fully implemented some but not all aspects of the practice and did not
take alternative actions that fully satisfied the practice. Yes--The
agency fully implemented the practice or took an alternative action
that fully satisfied its intent.
[End of figure]
Table 10 provides details on whether and how each project implemented
each of the six practices associated with this phase.
Table 10: Summary of Projects' Use of Phase VII Practices:
Practice: Monitor the provider's work to anticipate issues for
resolution.; Did the project use the practice?: Air Force C4 Services:
Yes--Quality assurance evaluators monitor the quality of the provider's
work and identify problems or trends. The results and any problems are
reported to the contracting officer and the functional area chief.; Did
the project use the practice?: Army TAC-SWA: Yes--The contracting
officer's representative reports monthly on contractor performance. In
addition, he meets daily with provider officials to discuss performance
results.; Did the project use the practice?: MHS/ITO Help Desk: Yes--
The provider's work is monitored through monthly reports and meetings
as well as through on-site meetings and readiness assessments.; Did the
project use the practice?: Navy NMCI: Yes--Navy representatives at each
implemented site (including headquarters) monitor the provider's work
and identify issues. According to NMCI program officials, the NMCI
Enterprise Management System enables the government to monitor the
performance of the network and associated service delivery.; Did the
project use the practice?: NIMA IT/IS: Yes--NIMA reviews the provider's
work in quarterly program management reviews and as needed in response
to issues raised by NIMA's performance monitoring officials.
Practice: Make sure the provider uses the standard tools and processes
defined as part of the operational model.; Did the project use the
practice?: Air Force C4 Services: Yes--The provider uses software tools
to help monitor system performance against performance standards to
monitor and manage the help-desk function.; Did the project use the
practice?: Army TAC-SWA: Yes--According to TAC-SWA project officials,
the provider uses various standard tools outlined in its proposal.; Did
the project use the practice?: MHS/ITO Help Desk: Yes--According to MHS
program officials, the provider uses the standard tools outlined in the
contract.; Did the project use the practice?: Navy NMCI: Yes--The Navy
has monitored the service provider's use of standard tools and
processes. Tools are being used to support legacy migration, client
installation, and help-desk procedures.; Did the project use the
practice?: NIMA IT/IS: Yes--The tools are documented in the quality
management plan. For example, the provider is using a specific quality
and process improvement methodology.
Practice: Use provider performance data to continuously improve
processes.; Did the project use the practice?: Air Force C4 Services:
Yes--Although Kirtland Air Force Base does not rely on data provided by
its provider, its quality assurance evaluators continually monitor
provider performance.; Did the project use the practice?: Army TAC-SWA:
Yes--The contracting officer's representative monitors provider
performance data, and any potential areas for improvement are
incorporated in the monthly report.; Did the project use the practice?:
MHS/ITO Help Desk: Yes--For example, MHS used provider performance data
to improve the customer satisfaction survey process.; Did the project
use the practice?: Navy NMCI: Yes--The Navy uses the provider's data to
help determine ways to improve processes, such as to improve NMCI
implementation procedures and the timeliness of help-desk problem
resolution.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA's
performance monitors use provider data to assess progress and ways to
improve performance.
Practice: Pursue improvement based on customer satisfaction surveys.;
Did the project use the practice?: Air Force C4 Services: Limited--The
Air Force does not perform, or require the provider to perform,
customer surveys. However, the provider has initiated such surveys but
is not required to distribute the results to the Air Force.; Did the
project use the practice?: Army TAC-SWA: Yes--Surveys are received by
the contracting officer's technical representatives and provided as
input for the contracting officer's representative report.; Did the
project use the practice?: MHS/ITO Help Desk: Yes--Negative feedback
that is received based on the results of the customer satisfaction
surveys is reviewed by the MHS help-desk manager.; Did the project use
the practice?: Navy NMCI: Yes--Surveys of user satisfaction with
specific services and overall satisfaction with provider performance
are used to identify areas needing improvement.; Did the project use
the practice?: NIMA IT/IS: Yes--To pursue improvement, NIMA uses both
customer satisfaction surveys and interviews with senior-level
customers.
Practice: Ensure that an appropriately empowered individual from the
client organization oversees the work.; Did the project use the
practice?: Air Force C4 Services: Yes--The functional area chief
oversees the work, and the quality assurance evaluators support this
oversight effort.; Did the project use the practice?: Army TAC-SWA:
Yes--The contracting officer's representative is the empowered
individual from NETCOM.; Did the project use the practice?: MHS/ITO
Help Desk: Yes--The government task manager is the individual empowered
to oversee the work of the provider.; Did the project use the
practice?: Navy NMCI: Yes--The NMCI Director oversees the work and
discusses performance with the NMCI provider executive during weekly
meetings.; Did the project use the practice?: NIMA IT/IS: Yes--The
contracting officer's representative and a staff of technical monitors
oversee the work.
Practice: Set realistic time frames that are agreed to by the
provider.; Did the project use the practice?: Air Force C4 Services:
Yes--Kirtland Air Force Base and the provider agree that reasonable
time frames for performance have now been set. Some revisions were made
to them in April 2002.; Did the project use the practice?: Army TAC-
SWA: Yes--Time frames for new tasks are established by contract
modifications that are agreed to by the provider.; Did the project use
the practice?: MHS/ITO Help Desk: Yes--The monthly performance reports
and quarterly program reviews set time frames that have been agreed to
by the provider.; Did the project use the practice?: Navy NMCI: Yes--
According to NMCI program officials, Navy's practice is to work with
the provider in establishing schedules.; Did the project use the
practice?: NIMA IT/IS: Yes--NIMA and the provider have agreed on dates
associated with specific actions.
Source: GAO:
[End of table]
[End of section]
Appendix IX: Comments from the Department of Defense:
DEPARTMENT OF DEFENSE:
6000 DEFENSE PENTAGON WASHINGTON, DC 20301-6000:
April 8, 2003:
CHIEF INFORMATION OFFICER:
Mr. Randolph C. Hite Director:
Information Technology Architecture and Systems Issues United States
General Accounting Office:
Washington, DC 20548:
Dear Mr. Hite:
This is the Department of Defense (DoD) response to GAO draft Report
No. GAO-03-371, "INFORMATION TECHNOLOGY: DoD Needs to Leverage Lessons
Learned from its Outsourcing Projects," dated March 11, 2002 (GAO Code
310239).
The Department partially concurs with the report. The DoD agrees that
capturing lessons learned in the development and implementation of its
IT outsourcing initiatives is important to continually improving the
methods used and the results achieved. However, specifying a particular
method of accession is premature at this time. The DoD currently has
several processes and communities of interest that collect and
disseminate lessons learned in other areas; one noteworthy example is
"Share A-76." The processes used by these groups would be a logical
starting point to determine the best path forward. Before the
Department commits to a specific means of provision, we intend to
explore a variety of mechanisms by which we can exploit lessons learned
in IT outsourcing initiatives.
One minor correction to the matrix on page 57 is requested. The
National Imagery and Mapping Agency (NIMA) has authorized its
contractor to procure supplies, but not products or services under the
Federal Supply Schedule. Therefore, under the NIMA IT/IS column, in the
third box (at the bottom), please change the sentence from:
"Yes-NIMA has authorized the provider to use government sources, such
as the Federal Supply Schedule, in procuring products, services, and
supplies related to this contract.":
to:
"Yes-NIMA has authorized the provider to use government sources, such
as the Federal Supply Schedule, in procuring supplies related to this
contract.":
If you require further information, please contact the action officer,
Leo Milanowski, at (703) 602-2720 x142 or leo.milanowskiAosd.mil.
Sincerely,
Margaret E. Myers,
Principal Director
Deputy Assistant Secretary of Defense (Deputy CIO):
Signed by Margaret E. Myers:
Attachment:
Response to Recommendations:
GAO DRAFT REPORT DATED MARCH 11, 2003 GAO-03-371 (GAO CODE 310239):
"INFORMATION TECHNOLOGY: DoD Needs to Leverage Lessons Learned from Its
Outsourcing Projects:
DEPARTMENT OF DEFENSE COMMENTS TO THE GAO RECOMMENDATIONS:
RECOMMENDATION 1: The GAO recommended that the Undersecretary of
Defense for Acquisition, Technology, and Logistics, working in
conjunction with the Assistant Secretary of Defense for Command,
Control, Communications, and Intelligence, provide senior management
support and adequate resources to develop and implement an electronic
tool to capture and disseminate examples and lessons learned from
actual IT outsourcing projects.
DOD RESPONSE:
Partially concur; DoD agrees that capturing lessons learned in the
development and implementation of its IT outsourcing initiatives is
important to continually improving the methods used and the results
achieved. However, specifying a particular method of accession is
premature at this time. DoD currently has several processes and
communities of interest that collect and disseminate lessons learned.
Before the Department commits to a specific means of provision, we
intend to explore a variety of mechanisms by which we can exploit
lessons learned in IT outsourcing initiatives.
RECOMMENDATION 2: The GAO recommended that the DoD ensure that the
method used to gather information for this electronic tool incorporate
the main element of a lessons learned process, namely, collection,
verification, storage, and dissemination.
DOD RESPONSE:
Partially concur; see above:
The following are GAO's comments on the Department of Defense's letter
dated April 8, 2003.
GAO Comments:
1. Addressed in the Agency Comments and Our Evaluation section of this
report.
2. On January 29, 2003, NIMA granted the provider authorization to use
government sources in performing the IT/IS contract that was limited to
"products, services, and supplies that support the performance of the
specific functional areas and miscellaneous items required under this
contract." Accordingly, we did not modify this report.
3. Addressed in the Agency Comments and Our Evaluation section of this
report.
[End of section]
Appendix X: GAO Contact and Staff Acknowledgments:
GAO Contact:
Linda J. Lambert, (202) 512-9556:
:
Staff Acknowledgments:
Season Dietrich, James Houtz, Anjalique Lawrence, Patricia Slocum, and
Thomas Wright made key contributions to this report.
(310239):
FOOTNOTES
[1] This figure is from the Federal Procurement Data System, which
contains detailed information on contract actions over $25,000.
[2] IT services outsourcing is a type of acquisition in which a client
organization transfers responsibility for performance of one or more IT
services to one or more external providers.
[3] U.S. General Accounting Office, Information Technology: Leading
Commercial Practices for Outsourcing of Services, GAO-02-214
(Washington, D.C.: Nov. 30, 2001).
[4] We asked the military services and other DOD components to identify
candidate projects because DOD does not maintain a central list of IT
services outsourcing projects. We chose each project on the basis of
the following criteria: (1) no more than one project from each military
service and two agencies, (2) illustrative example of DOD IT
outsourcing, (3) dollar value greater than $10 million, and (4) enough
time elapsed for services to have been delivered and performance
measured.
[5] We identified 70 practices in our November 2001 report on leading
commercial practices that (1) are typically applied at the project
level and (2) were verifiable through documentation and interviews.
[6] This calculation does not include practices that were not
applicable to a particular project.
[7] This consensus view was based on interviews with managers in
leading commercial organizations, discussions with academic and
professional authorities, and extensive research on IT acquisition
practices.
[8] This figure is from the Federal Procurement Data System, which
contains detailed information on contract actions over $25,000.
[9] GAO-02-214.
[10] Giga Information Group, Inc., IT Trends 2003: IT Services (Dec.
19, 2002). We did not independently verify these data.
[11] INPUT, The Federal IT Outsourcing Market View (December 2002). We
did not independently verify these data.
[12] GAO-02-214.
[13] This consensus view was based on interviews with managers in
leading commercial organizations, discussions with academic and
professional authorities, and extensive research on IT acquisition
practices.
[14] Office of Management and Budget, Circular A-76, Performance of
Commercial Activities (Washington, D.C.: June 14, 1999). In November
2002, OMB issued proposed revisions that would substantially change
this circular. As of April 8, 2003, these revisions have not yet been
made final.
[15] Final report of the Commercial Activities Panel, Improving the
Sourcing Decisions of the Government (Washington, D.C.: Apr. 30, 2002).
[16] Section 5 of P.L. 105-270, codified at 31 U.S.C. 501 note (1998),
defines an inherently governmental function as a "function that is so
intimately related to the public interest as to require performance by
Federal Government employees."
[17] U.S. General Accounting Office, Proposed Revisions to OMB Circular
A-76, GAO-03-391R (Washington, D.C.: Jan. 16, 2003).
[18] U.S. General Accounting Office, Desktop Outsourcing: Positive
Results Reported, but Analyses Could Be Strengthened, GAO-02-329
(Washington, D.C.: Mar. 29, 2002).
[19] The Industrial College of the Armed Forces, Case Study: Complex
Business Management for Competitive Sourcing (2001).
[20] PricewaterhouseCoopers Endowment for the Business of Government,
Contracting for the 21ST Century: A Partnership Model (January 2002).
[21] GAO-02-329.
[22] U.S. General Accounting Office, Contract Management: Taking a
Strategic Approach to Improving Service Acquisitions, GAO-02-499T
(Washington, D.C.: Mar. 7, 2002).
[23] This calculation does not include practices that were not
applicable to a particular project.
[24] Giga Information Group, Inc., Optimizing IT Sourcing Strategy: Key
Stages and Phases of the IT Sourcing Process (Jan. 31, 2003).
[25] This calculation does not include practices that were not
applicable to a particular project.
[26] Gartner, Inc., Strategic Analysis Report, How to Build a Sourcing
Strategy, Research Note R-18-1099 (Sept. 23, 2002).
[27] The Department of the Interior's GovWorks Program is a federal
fee-for-service acquisition center that helps other federal agencies
acquire supplies and services for their programs on a project-by-
project basis.
[28] Gartner, Inc., defines benchmarking as a method to compare the
cost or price of an IT environment to peer groups with the same
workload characteristics.
[29] Giga Information Group, Inc., Payment and Incentives for
Outsourcing Management (July 27, 2000) and Gartner, Inc., Benchmarking
Helps Outsourcing Deals Stay Competitive, Research Note COM-16-8055
(June 14, 2002).
[30] This calculation does not include practices that were not
applicable to a particular project.
[31] This calculation does not include practices that were not
applicable to a particular project.
[32] Seat management generally refers to service provision arrangements
in which contractor-owned desktop and other computing hardware,
software, and related services are bundled and provided to a client
organization at a fixed price per unit (or seat).
[33] This calculation does not include practices that were not
applicable to a particular project.
[34] Gartner, Inc., Retain Enough Resources to Manage Outsourcing
Deals, Research Note COM-16-8425 (June 17, 2002).
[35] An Interagency-Industry Partnership in Performance, Seven Steps to
Performance-Based Services Acquisition, Benchmark Version (January
2002).
[36] U.S. General Accounting Office, NASA: Better Mechanisms Needed for
Sharing Lessons Learned, GAO-02-195 (Washington, D.C.: Jan. 30, 2002).
[37] U.S. General Accounting Office, Information Technology Investment
Management: A Framework for Assessing and Improving Process Maturity,
GAO/AIMD-10-1.23, Exposure Draft (Washington, D.C.: May 2000).
[38] U.S. General Accounting Office, DOD Competitive Sourcing: Lessons
Learned System Could Enhance A-76 Study Process, GAO/NSIAD-99-152
(Washington, D.C.: July 21, 1999).
[39] GAO-02-329.
[40] GAO-02-195.
[41] GAO-02-195.
[42] U.S. General Accounting Office, Information Technology: Leading
Commercial Practices for Outsourcing of Services, GAO-02-214
(Washington, D.C.: Nov. 30, 2001).
[43] GAO-02-214.
[44] U.S. General Accounting Office, NASA: Better Mechanisms Needed for
Sharing Lessons Learned, GAO-02-195 (Washington, D.C.: Jan. 30, 2002).
[45] Gartner, Inc., Five Tough Questions About Skill Sourcing, Research
Note SPA-13-2537 (Mar. 28, 2001).
[46] SLAs define the agency's expectations and are used to track and
measure a contractor's performance.
[47] An Interagency-Industry Partnership in Performance, Seven Steps to
Performance-Based Services Acquisition, Benchmark Version (January
2002).
[48] Gartner, Inc., Retain Enough Resources to Manage Outsourcing
Deals, Research Note COM-16-8425 (June 17, 2002).
[49] Gartner, Inc., Successful Outsourcing Means Retaining Some Staff,
Research Note COM-18-9692 (Dec. 18, 2002).
[50] An Interagency-Industry Partnership in Performance, Seven Steps to
Performance-Based Services Acquisition, Benchmark Version (January
2002).
GAO's Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to daily E-mail alert for newly
released products" under the GAO Reports heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW,
Room LM Washington,
D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.
General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.
20548:
