Defense Logistics
Improved Analysis and Cost Data Needed to Evaluate the Cost-effectiveness of Performance Based Logistics Gao ID: GAO-09-41 December 19, 2008In 2001, the Department of Defense (DOD) identified performance based logistics (PBL) as the preferred weapon system support strategy. Within DOD, PBL is the purchase of performance outcomes, such as system availability, rather than the purchase of individual elements of logistics support--such as parts, repairs, and engineering support. Although PBL initially arose from efforts to reduce support costs, questions have arisen about whether PBL has reduced support costs as originally intended. GAO was asked to evaluate the extent to which DOD has used business case analyses to guide decisions related to PBL arrangements and the impact PBL arrangements have had on weapon system support costs. In conducting the review, GAO analyzed the implementation of PBL arrangements for 29 weapon system programs. GAO also looked at the use and characteristics of performance-based contracting in the United Kingdom's Ministry of Defence.
Although DOD's guidance recommends that business case analyses be used to guide decision making regarding the implementation of PBL to provide weapon system support, the services are not consistent in their use of such analyses. About half of the DOD program offices responsible for the 29 PBL arrangements GAO reviewed either did not use a business case analysis or could not provide documentation for significant parts of their analyses. Almost all of the remaining analyses were missing one or more of the recommended elements in DOD's instruction for economic analysis. Finally, business case analyses were often not updated in accordance with service policies and guidance. Program office use of these analyses is inconsistent because DOD only recommends, but does not require, that they be prepared and because DOD's guidance on preparing a business case analysis is not comprehensive and does not adequately specify the criteria to be included. Also, most of the services have not established effective internal controls to ensure that the analyses are prepared or that they provide a consistent and comprehensive assessment. As a result, DOD has implemented PBL arrangements without the benefit of sound analyses that ensure that the chosen approach will provide the most cost-effective support option. While one of DOD's goals in moving toward the use of PBL arrangements was to reduce weapon system support costs, the ability of these arrangements to reduce costs remains unclear 7 years after DOD first identified PBL as the preferred weapon system support strategy. Many DOD program offices that implemented PBL arrangements have limited cost data, and various other factors--such as the lack of business case analyses--further limit an evaluation of the costs of this support strategy. Available data from the programs GAO reviewed indicated mixed results. Although a few programs in GAO's sample provided evidence of some cost reductions, GAO's analysis of the only two systems in its sample that are managed using both a PBL arrangement and a more traditional, non-PBL arrangement indicated that in both cases the PBL arrangement had higher costs. Also, GAO found that certain characteristics of DOD's PBL arrangements--contract length, funding stability, ownership of inventory, and the lack of cost metrics and effective incentives--could limit the ability of and incentive for contractors to reduce support costs. Neither DOD nor the services require detailed cost reporting for PBL arrangements and the lack of detailed cost data hinders DOD's ability to determine whether PBL has reduced support costs as intended. GAO describes the use of performance-based arrangements for weapon system support in the United Kingdom's Ministry of Defence, which the ministry refers to as contracting for availability. The Ministry of Defence began awarding availability contracts as an approach to reduce weapon system support costs, and officials believe that support cost reductions have been achieved as a result of using availability contracts. In general, the availability contracts used are significantly longer than those used by DOD, and the ministry uses an "open book accounting" arrangement to gain visibility into the contractors' costs to provide support.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-09-41, Defense Logistics: Improved Analysis and Cost Data Needed to Evaluate the Cost-effectiveness of Performance Based Logistics
This is the accessible text file for GAO report number GAO-09-41
entitled 'Defense Logistics: Improved Analysis and Cost Data Needed to
Evaluate the Cost-effectiveness of Performance Based Logistics' which
was released on December 22, 2008.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Subcommittee on Readiness, Committee on Armed Services,
House of Representatives:
United States Government Accountability Office:
GAO:
December 2008:
Defense Logistics:
Improved Analysis and Cost Data Needed to Evaluate the Cost-
effectiveness of Performance Based Logistics:
GAO-09-41:
GAO Highlights:
Highlights of GAO-09-41, a report to the Subcommittee on Readiness,
Committee on Armed Services, House of Representatives.
Why GAO Did This Study:
n 2001, the Department of Defense (DOD) identified performance based
logistics (PBL) as the preferred weapon system support strategy. Within
DOD, PBL is the purchase of performance outcomes, such as system
availability, rather than the purchase of individual elements of
logistics support”such as parts, repairs, and engineering support.
Although PBL initially arose from efforts to reduce support costs,
questions have arisen about whether PBL has reduced support costs as
originally intended. GAO was asked to evaluate the extent to which DOD
has used business case analyses to guide decisions related to PBL
arrangements and the impact PBL arrangements have had on weapon system
support costs. In conducting the review, GAO analyzed the
implementation of PBL arrangements for 29 weapon system programs. GAO
also looked at the use and characteristics of performance-based
contracting in the United Kingdom‘s Ministry of Defence.
What GAO Found:
Although DOD‘s guidance recommends that business case analyses be used
to guide decision making regarding the implementation of PBL to provide
weapon system support, the services are not consistent in their use of
such analyses. About half of the DOD program offices responsible for
the 29 PBL arrangements GAO reviewed either did not use a business case
analysis or could not provide documentation for significant parts of
their analyses. Almost all of the remaining analyses were missing one
or more of the recommended elements in DOD‘s instruction for economic
analysis. Finally, business case analyses were often not updated in
accordance with service policies and guidance. Program office use of
these analyses is inconsistent because DOD only recommends, but does
not require, that they be prepared and because DOD‘s guidance on
preparing a business case analysis is not comprehensive and does not
adequately specify the criteria to be included. Also, most of the
services have not established effective internal controls to ensure
that the analyses are prepared or that they provide a consistent and
comprehensive assessment. As a result, DOD has implemented PBL
arrangements without the benefit of sound analyses that ensure that the
chosen approach will provide the most cost-effective support option.
While one of DOD‘s goals in moving toward the use of PBL arrangements
was to reduce weapon system support costs, the ability of these
arrangements to reduce costs remains unclear 7 years after DOD first
identified PBL as the preferred weapon system support strategy. Many
DOD program offices that implemented PBL arrangements have limited cost
data, and various other factors”such as the lack of business case
analyses”further limit an evaluation of the costs of this support
strategy. Available data from the programs GAO reviewed indicated mixed
results. Although a few programs in GAO‘s sample provided evidence of
some cost reductions, GAO‘s analysis of the only two systems in its
sample that are managed using both a PBL arrangement and a more
traditional, non-PBL arrangement indicated that in both cases the PBL
arrangement had higher costs. Also, GAO found that certain
characteristics of DOD‘s PBL arrangements”contract length, funding
stability, ownership of inventory, and the lack of cost metrics and
effective incentives”could limit the ability of and incentive for
contractors to reduce support costs. Neither DOD nor the services
require detailed cost reporting for PBL arrangements and the lack of
detailed cost data hinders DOD‘s ability to determine whether PBL has
reduced support costs as intended.
GAO describes the use of performance-based arrangements for weapon
system support in the United Kingdom‘s Ministry of Defence, which the
ministry refers to as contracting for availability. The Ministry of
Defence began awarding availability contracts as an approach to reduce
weapon system support costs, and officials believe that support cost
reductions have been achieved as a result of using availability
contracts. In general, the availability contracts used are
significantly longer than those used by DOD, and the ministry uses an
’open book accounting“ arrangement to gain visibility into the
contractors‘ costs to provide support.
What GAO Recommends:
GAO makes five recommendations, including that DOD require the
development of business case analyses, clearly define specific criteria
for their development, and require that the services improve their
internal controls to ensure that the analyses are performed. GAO also
recommends that DOD require program offices to collect and report
detailed support cost data for their PBL arrangements. DOD generally
concurred with these recommendations.
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-09-41]. For more
information, contact William M. Solis at (202) 512-8365 or
solisw@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
Business Case Analyses Have Not Been Used Consistently or Effectively
to Influence Decision Making regarding Use of PBL:
Supporting Documentation Not Fully Available for Some Programs That
Conducted Business Case Analyses:
Most Remaining Programs Used Business Case Analyses That Were Not
Comprehensive:
Other Factors Influencing the Quality of Business Case Analyses:
Impact of PBL Arrangements on Reducing Weapon System Support Costs Is
Unclear:
Use and Characteristics of Availability Contracts by the United
Kingdom's Ministry of Defence:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Comments from the Department of Defense:
Appendix III: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Weapon System Programs Supported by PBL Arrangements:
Table 2: Number of PBL Arrangements Initiated with and without the Use
of a Business Case Analysis:
Table 3: Number of PBL Arrangements with and without Supporting
Documentation of a Business Case Analysis:
Table 4: Service-Issued Business Case Analysis Policies and Guidance:
Table 5: Current Service PBL and Business Case Analysis Policies and
Guidance:
Figures:
Figure 1: Elements Missing from the Business Case Analyses We Reviewed:
Figure 2: Comparison of Navy PBL Costs with Air Force Government/
Traditional Costs:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
December 19, 2008:
The Honorable Solomon P. Ortiz:
Chairman:
The Honorable J. Randy Forbes:
Ranking Member:
Subcommittee on Readiness:
Committee on Armed Services:
House of Representatives:
In the 1990s, the Department of Defense (DOD) predicted that it would
save billions of dollars by reengineering defense logistics to increase
its reliance on the private sector for the support of military weapon
systems.[Footnote 1] In 1999, DOD set a goal to reduce weapon system
operating and support costs by 20 percent by 2005. Around the same
time, a DOD study identified 30 pilot programs on which to test various
logistics reengineering initiatives, and DOD began to consider the
concept of long-term contractor logistics weapon system support with
performance metrics, which DOD later called performance based logistics
(PBL). By 2001, DOD had identified PBL as its preferred weapon system
support strategy. DOD defines PBL as the purchase of performance
outcomes (such as the availability of functioning weapon systems)
through long-term support arrangements rather than the purchase of
individual elements of support--such as parts, repairs, and engineering
support. The concept of PBL evolved from the more general concept of
performance-based contracting, which has been used in both the
commercial and government sectors for over 20 years. While the use of
PBL was first applied to weapon system platforms, DOD now also uses it
to purchase support for subsystems and components. Under DOD guidance,
[Footnote 2] prior to entering into a PBL arrangement, program offices
should conduct a business case analysis as part of their decision-
making process. A business case analysis is an analytical tool for
assessing the projected costs and benefits of a PBL arrangement
compared with other alternative weapon system support options. However,
since DOD first implemented PBL arrangements, questions have arisen
about whether they have reduced the costs of weapon system support as
intended.
In 2004 and 2005, we reported that DOD's use of PBL arrangements did
not reflect the best practices of private-sector companies and that DOD
could not demonstrate that PBL arrangements had reduced costs and
improved performance.[Footnote 3] Specifically, in 2004, we recommended
that DOD incorporate into guidance the private sector's practice of
using performance-based contracts at the component level (for example,
to support engines or auxiliary power units) rather than only at the
weapon system platform level (for example, to support aircraft). In
response, DOD issued policy memorandums emphasizing that PBL may be
applied at the weapon system, subsystem, or component level and revised
its guidance to implement our recommendation. In 2005, we recommended
that DOD, in conjunction with the military services, develop procedures
to track whether DOD program offices update their business case
analyses as called for under DOD guidance. We also recommended that DOD
verify the reliability of contractor cost and performance data. DOD
concurred with our recommendations and has issued limited guidance to
the services regarding the verification of the reliability of
contractor cost and performance data. However, DOD and the services,
with the exception of the Army, have not developed procedures to track
whether program offices update business case analyses after PBL
implementation.
Given concerns about the cost of programs using PBL arrangements, you
requested that we examine DOD's use of PBL for providing weapon system
support. In response to your request, we evaluated the (1) extent to
which DOD has used business case analyses to guide decisions related to
PBL arrangements and (2) impact PBL arrangements have had on weapon
system support costs. In addition, we describe the use and
characteristics of "availability contracts," a concept similar to PBL
arrangements that the United Kingdom's Ministry of Defence uses for
weapon system support. We looked at the Ministry of Defence's use of
availability contracts to identify potential promising practices
because reducing costs was a clear goal of the implementation of this
concept in the United Kingdom and a National Audit Office report stated
that the Ministry of Defence has reduced support costs as a result of
its implementation of this concept, and related initiatives, for two
aircraft systems.
We collected and analyzed data on 27 PBL arrangements--specifically 9
from the Army, 8 from the Navy, and 10 from the Air Force--that were
initiated from 1996 through 2007 to support DOD weapon systems.
[Footnote 4] In addition, we reviewed 2 additional weapon systems that
the Marine Corps was considering as candidates for PBL arrangements.
[Footnote 5] We selected these programs from lists of PBL arrangements
provided by the services and chose both component-and system-level
arrangements from each of the services on the basis of the dollar value
of the arrangements, prior audit findings, and location.[Footnote 6]
While we focused on the costs of PBL arrangements, program officials
said that performance generally met or exceeded requirements
established in the contracts or other agreements.
To evaluate the extent to which DOD has used business case analyses to
guide decisions regarding PBL arrangements, we obtained and examined
the analyses prepared by program offices. In assessing the quality of
these analyses, we used applicable criteria in DOD Instruction 7041.3,
Economic Analysis for Decisionmaking.[Footnote 7] This economic
analysis instruction is generally consistent with DOD's guidance on
conducting a business case analysis; in both cases, DOD recommends that
costs and benefits (both qualitative and quantitative) be considered
and documented. However, the economic analysis instruction contains
more specific information for evaluation of program costs. This
guidance is also consistent with Office of Management and Budget
guidelines for the benefit-cost analysis of federal programs.[Footnote
8] DOD's economic analysis instruction also states that analytical
studies that evaluate the cost and effectiveness of weapon system
support are considered to be "economic analyses" even if not titled as
such.
To evaluate the impact that PBL arrangements have had on weapon system
support costs, we asked program officials to identify and document
savings attributed to PBL implementation. In addition, if a program had
renewed a PBL arrangement or had finalized contract options that were
not priced, we analyzed the contracts for trends in PBL support costs.
We also compared PBL contract costs with estimated PBL support costs in
business case analyses to determine how closely the estimates matched
the actual PBL arrangement costs, where available. We also relied on
our previously issued reports and testimonies on DOD's implementation
of PBL.
To provide information regarding the characteristics and use of
availability contracts for weapon system support in the United
Kingdom's Ministry of Defence, we met Ministry of Defence and National
Audit Office officials and officials from defense contractors having
availability contracts with the Ministry of Defence. We also reviewed a
2007 National Audit Office report regarding the Ministry of Defence's
use of availability contracting to support Harrier and Tornado fast
jets.[Footnote 9] We obtained these data for informational purposes and
to identify promising practices and did not independently verify the
statements or data provided by Ministry of Defence and National Audit
Office officials.
We conducted this performance audit from February 2007 through December
2008 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objectives.
We believe that the evidence obtained provides a reasonable basis for
our findings and conclusions based on our audit objectives. A detailed
description of our scope and methodology is contained in appendix I.
Results in Brief:
Although DOD's guidance recommends that business case analyses be used
to guide decision making regarding the implementation of PBL to provide
weapon system support, the services are not consistent in their use of
such analyses. In general, for the PBL arrangements we reviewed
business case analyses either were not done, were not fully documented,
or were not comprehensive or sound. About half of the DOD program
offices responsible for the 29 PBL arrangements we reviewed either did
not use a business case analysis (or economic analysis) or could not
provide documentation for significant parts of their analyses. Almost
all of the remaining analyses were missing one or more of the
recommended elements in DOD's economic analysis instruction. Finally,
business case analyses were often not updated in accordance with
service policies and guidance. Program office use of these analytical
tools is inconsistent because (1) DOD's guidance did not require that
they be prepared and updated and (2) DOD did not provide specific
criteria for conducting the analyses. Additionally, most of the
services have not established effective internal controls to ensure
that the analyses are prepared or that they provide a consistent and
comprehensive assessment of weapon system support options. As a result,
DOD has implemented PBL arrangements to provide weapon system support
without sound analyses that ensure that the chosen approach will
provide the most cost-effective support option for the department.
Although the PBL concept was an initiative in a DOD logistics
reengineering pilot program intended to reduce weapon system support
costs, the ability of these arrangements to reduce costs remains
unclear 7 years after DOD first identified PBL as the preferred support
strategy. Many DOD program offices that implemented PBL arrangements
have limited cost data, and various other factors, such as the lack of
business case analyses and changing operational and materiel
conditions, further limit an evaluation of the costs of this support
strategy. Available data from the programs we reviewed indicated mixed
results. Although a few programs in our sample provided evidence of
some cost reductions, our analysis of the only two systems in our
sample that were managed using both a PBL arrangement and a more
traditional, non-PBL arrangement indicated that the PBL arrangement had
higher costs in both cases. We also found few examples of decreasing
support costs after PBL contract renewal, as the PBL concept suggests
should occur. Finally, we found that certain characteristics of DOD's
PBL arrangements--such as short-term contracts, unstable requirements
and funding, DOD ownership of inventory, and the lack of cost metrics
and effective incentives--could limit the ability of and incentive for
contractors to reduce support costs. Neither DOD nor the services
require detailed cost reporting for PBL (or for any type of contractor
logistics support). Further, in its implementation of PBL arrangements,
DOD has emphasized the potential to improve performance and not the
potential to reduce support costs. The lack of detailed cost data for
evaluating the impact of PBL arrangements on support costs, compounded
with deficiencies in DOD's use of business case analyses for decision
making, hinders DOD's ability to determine whether most PBL
arrangements have reduced support costs as intended.
We describe the use of performance-based arrangements for weapon system
support in the United Kingdom's Ministry of Defence, which is referred
to as contracting for availability. The Ministry of Defence began
awarding availability contracts--generally for aviation systems but
also for some maritime equipment--as an approach to reduce weapon
system support costs after a 1999 cost reduction goal of 20 percent was
set. Ministry of Defence officials believe that support cost reductions
have been achieved as a result of using availability contracts, and the
National Audit Office reported savings for the Tornado and Harrier fast
jets, although it is unclear to what extent the availability contracts
generated the cost reductions because other cost reduction initiatives
were implemented at the same time. In general, the availability
contracts used by the ministry are significantly longer than those used
by DOD, and the ministry uses an "open book accounting" arrangement to
gain visibility into the contractors' costs to provide support.
We are making recommendations in this report aimed at documenting and
improving the analysis of the cost-effectiveness of PBL arrangements.
We recommend, for example, that DOD require the development of business
case analyses, specifying the elements to be included in them so that
they are comprehensive and sound and that the services improve their
internal controls to ensure that the analyses are performed, and that
program offices collect and report detailed support cost data for their
PBL arrangements. Improvements in the collection and reporting of
detailed cost data could go a long way toward providing required
information for determining if PBL provides the most cost-effective
approach for supporting DOD weapon systems. In written comments on a
draft of this report, DOD generally concurred with our recommendations
and discussed actions it has planned to implement some of our
recommendations, such as revising DOD's acquisition directive to
require the development of a business case analysis. DOD also stated
that it would review existing policy and guidance in other areas as
part of an ongoing study. While we view DOD's actions as positive steps
to improve the implementation of PBL, it is unclear to what extent the
results of the study, and any related actions, will affect the
evaluation of the cost-effectiveness of DOD's PBL arrangements. DOD's
comments are reprinted in appendix II.
Background:
PBL is a method of providing support for weapon systems by designating
what system performance is required, such as a given level of system
availability, and placing the responsibility for how it is accomplished
on the support provider, which manages resources to achieve performance
objectives. Logistics support for almost all of DOD's weapon systems,
such as materiel management, maintenance, and engineering, is provided
by a combination of government and private-sector sources. In the past,
under traditional support arrangements, the government generally
managed the provision of weapon system support, using a combination of
support providers from the government and the private sector. PBL
support arrangements often use a private-sector support integrator to
manage support providers from both the public and private sectors to
meet specified performance requirements.
PBL evolved from performance-based service contracting, which has been
used in both the public and private sectors. The Federal Acquisition
Regulation defines performance-based contracting as structuring all
aspects of an acquisition around the purpose of the work to be
performed. The Federal Acquisition Regulation further defines the
statement of work for a performance-based acquisition as describing the
required results in clear, specific, and objective terms with
measurable outcomes.[Footnote 10] Performance-based service contracting
has been referenced in regulation, guidance, and policy for more than
two decades, and federal agencies have used it to varying degrees for
acquiring a range of services. In 1991 the Office of Management and
Budget issued a policy letter establishing the use of a performance-
based approach for service contracting, and in 1994 it initiated a
governmentwide pilot project to encourage the use of performance-based
service contracts in federal agencies, including DOD. In October 1997,
the Federal Acquisition Regulation was amended to incorporate the
Office of Management and Budget's 1991 policy. The Federal Acquisition
Regulation currently establishes a policy that agencies use performance-
based contracting methods to the maximum extent practicable for the
acquisition of services, with certain exceptions.[Footnote 11]
Using performance-based service contracts is intended to offer a number
of potential benefits, such as encouraging contractors to be innovative
and to find cost-effective ways of delivering services for a fixed
level of funding. By shifting the focus from process to results, these
contracts can potentially produce better outcomes and reduced costs.
The Office of Management and Budget reported that the agencies
participating in the pilot reduced contract prices and improved
customer satisfaction with contractor work after introducing
performance-based contracting.
PBL Implemented as a Strategy to Reduce or Minimize Costs and Improve
Performance:
As an approach for supporting military weapon systems, PBL emerged from
a 1999 DOD study to test logistics reengineering concepts that placed
greater reliance on the private sector for providing weapon system
support to reduce support costs and improve weapon system performance.
The goal was for the military departments to reduce costs and improve
efficiency by pursuing logistics support "reengineering" efforts using
contractors.[Footnote 12] The fiscal years 2001-2005 Defense Planning
Guidance advanced this cost reduction effort by establishing a goal
that directed each military department to reduce the operation and
support costs of its fielded systems by 20 percent by the year 2005.
During this time, the Under Secretary of Defense (Acquisition and
Technology) directed the services to use an existing pilot program
containing 30 weapon systems to demonstrate the type of cost savings
depicted in the fiscal years 2001-2005 Defense Planning Guidance. The
areas identified for potential cost savings were reducing demand on the
supply chain by improving the reliability and maintainability of the
equipment, reducing supply chain response time, and increasing
competitive sourcing of product support. Some of the 30 pilot programs
involved performance-type arrangements that the services subsequently
converted to, or designated as, PBL arrangements.
This emphasis on reducing costs through PBL implementation was
reiterated in DOD's 2001 Quadrennial Defense Review Report,[Footnote
13] which advocated the implementation of PBL to compress the supply
chain by removing steps in the warehousing, distribution, and order
fulfillment processes; to reduce inventories; and to decrease overhead
costs while improving the readiness of major weapon systems and
commodities. In November 2001, DOD identified PBL as the preferred
weapons system support strategy.[Footnote 14] In May 2003, DOD further
strengthened this emphasis on PBL by stating in a DOD policy directive
that acquisition managers shall use performance-based strategies for
sustaining products and services whenever feasible and PBL strategies
shall optimize total system availability while minimizing cost and the
logistics footprint.[Footnote 15]
In concept, a properly structured PBL arrangement is supposed to
provide a level of performance and also reduce costs over time.
According to the DOD/Defense Acquisition University PBL guide,[Footnote
16] a key aspect of PBL is the inclusion of an incentive for the
support provider to reduce costs through increased reliability.
Further, PBL arrangements can inherently motivate support providers to
improve component and system reliability, since such improvements can
provide the foundation for increased profit over the long term. In
other words, the support provider should have the incentive to make
reliability improvements to ensure that performance metrics are met and
also to increase profit by earning a performance incentive tied to the
metrics (an award fee or award term) and by reducing costs while still
being paid the agreed-upon, fixed price for the remainder of the
contract. The DOD/Defense Acquisition University PBL guide also states
that a critical element of the PBL arrangement that facilitates this
incentive and motivation is contract length. Further, long-term
contracts provide the support provider with confidence in continuing
cash flows and provide sufficient time for receiving an adequate return
on any investments made to improve reliability.
DOD Recommended That Analyses Be Conducted to Guide Decisions regarding
Weapon System Support Options:
In 1995, before DOD identified PBL as the preferred weapons system
support strategy, DOD's economic analysis instruction recommended using
an economic analysis for evaluating options for weapon system support.
The instruction stressed the importance of considering in the analysis
both qualitative and quantitative factors. With respect to quantitative
factors, the instruction recommended that costs and benefits be
expressed in terms of net present value to account for the time value
of money. These were also to be expressed as life cycle costs and
benefits that were to be calculated and compared for each feasible
alternative for meeting a given weapon system support objective.
Specifically, the economic analysis instruction identified and
characterized the following seven elements that should be present in an
economic analysis:
* objectives--to clearly identify the function to be accomplished and
not to assume a specific means of achieving a desired result;
* assumptions--to incorporate both actual data and future
uncertainties;
* alternatives--to comprise a comprehensive list of the feasible and
infeasible options followed by a discussion of the infeasible options
and comparisons of only the feasible options;
* costs and benefits--to compare the quantitative (expressed in terms
of net present value) and qualitative factors for each option;
* sensitivity and uncertainty (risk) analyses--to determine the effect
of uncertainties on the results of the analysis and to provide a range
of costs and benefits;
* summary of the results of the analysis; and:
* summary of the analysis's recommendations.
In DOD's 2001 PBL guide, in which DOD identified PBL as the preferred
weapon system support strategy, the department recommended that for all
new systems and fielded acquisition category I and II systems,[Footnote
17] program offices use an analytical tool called a business case
analysis to support the decision to use PBL arrangements for weapon
system support.
In 2004 and 2005, DOD guidance on conducting business case analyses
described this tool in less specific terms than those used to describe
the criteria laid out in DOD's economic analysis instruction. However,
there are some common themes in the guidance and instruction. For
example, a January 2004 Under Secretary of Defense (Acquisition,
Technology and Logistics) memorandum on PBL business case analysis
calls for an assessment of "best value," or the greatest overall
benefit.[Footnote 18] The DOD/Defense Acquisition University PBL guide
repeats the characterization of the business case analysis as a tool
with the goal of determining a best-value solution and suggests that at
a minimum a business case analysis should include an introduction
outlining the purpose and objectives of the program, an explanation of
the methods and assumptions used in the analysis, calculations of the
relative costs and benefits of each weapon system support option, the
financial and nonfinancial impacts of each, a risk assessment, and a
section with conclusions and recommendations. Finally, both DOD's
economic analysis instruction and the DOD/Defense Acquisition
University PBL guide recommend documenting the results of the analysis,
including all calculations and sources of data, down to the most basic
inputs, to provide an auditable and stand-alone document. According to
the guidance, a business case analysis must stand on its own and be
able to withstand rigorous analysis and review by independent agencies.
DOD's 2004 and 2005 guidance on conducting business case analyses also
recommended that program offices:
* update their business case analyses at key decision points both to
validate the approach taken and to support future plans and:
* use certain criteria, such as the capability of a PBL arrangement to
reduce the cost per operational unit of performance (i.e., cost per
flight hour), to assess all acquisition I and II programs without plans
for a PBL arrangement for the potential application of a PBL strategy
at the system, subsystem, or major assembly level. If the assessment
showed potential for a PBL arrangement, a business case analysis should
be conducted and completed by the September 30, 2006, deadline required
by DOD's Strategic Planning Guidance.
PBL Arrangements Included in Our Review:
In our review, we looked at PBL arrangements initiated as early as 1996
(when performance-based contracting was encouraged governmentwide) and
as recently as 2007 (by which time, at the DOD level, PBL arrangements
had moved from being encouraged to being required whenever feasible).
These PBL arrangements represent contract values totaling approximately
$12.1 billion and range from a low of $10.5 million to a high of $4.9
billion.[Footnote 19] Table 1 lists, by service, the weapon system
programs supported by the 29 PBL arrangements we reviewed.
Table 1: Weapon System Programs Supported by PBL Arrangements:
Service: Army;
Weapon system: Tube-launched Optically-tracked Wire-guided missile-
Improved Target Acquisition System.
Service: Army;
Weapon system: Javelin antitank missile.
Service: Army;
Weapon system: High Mobility Artillery Rocket System.
Service: Army;
Weapon system: RQ-7B Shadow Tactical Unmanned Aircraft System.
Service: Army;
Weapon system: Sentinel Radar.
Service: Army;
Weapon system: Patriot Air Defense System.
Service: Army;
Weapon system: AH-64D Apache Helicopter - Sensors.
Service: Army;
Weapon system: AH-64D Apache Helicopter - Airframe.
Service: Army;
Weapon system: AN/TSQ-179AV(2) Common Ground Station.
Service: Navy;
Weapon system: F-16 engines.
Service: Navy;
Weapon system: KC-130J[A].
Service: Navy;
Weapon system: V-22 engine.
Service: Navy;
Weapon system: T-45 engine.
Service: Navy;
Weapon system: Phalanx Close-In Weapon System.
Service: Navy;
Weapon system: AV-8B Harrier.
Service: Navy;
Weapon system: F/A-18 E/F.
Service: Navy;
Weapon system: Consolidated Automated Support System.
Service: Marine Corps;
Weapon system: Expeditionary Fighting Vehicle[B].
Service: Marine Corps;
Weapon system: Assault Breacher Vehicle[B].
Service: Air Force;
Weapon system: C-17 Globemaster III.
Service: Air Force;
Weapon system: T-6A Joint Primary Air Training System.
Service: Air Force;
Weapon system: F-117A Nighthawk.
Service: Air Force;
Weapon system: F-22 Raptor.
Service: Air Force;
Weapon system: B-2 Spirit.
Service: Air Force;
Weapon system: Secondary Power Systems.
Service: Air Force;
Weapon system: E8-C Joint Surveillance Target Attack Radar System.
Service: Air Force;
Weapon system: LITENING Advanced Airborne Targeting and Navigation Pod.
Service: Air Force;
Weapon system: Sniper Advanced Targeting Pod.
Service: Air Force;
Weapon system: C-130J Hercules.
Source: GAO analysis of DOD data.
[A] Although the KC-130J is a Marine Corps aircraft, the support
contracts are managed by the Navy.
[B] A PBL arrangement has not yet been implemented.
[End of table]
Business Case Analyses Have Not Been Used Consistently or Effectively
to Influence Decision Making regarding Use of PBL:
DOD has generally not used business case analyses consistently or
effectively to influence decision making regarding the use of PBL.
Although DOD guidance recommended that these analyses be used to guide
decisions on the cost-effectiveness of weapon system support
arrangements, about half of the programs we reviewed either did not
conduct such an analysis or did not retain adequate supporting
documentation.[Footnote 20] Further, most of the remaining programs in
our sample used analyses that were not comprehensive. For example, some
analyses did not evaluate alternative support options and most did not
contain all of the elements recommended in DOD's economic analysis
instruction. Additionally, analyses were often not updated to support
decision making after PBL implementation in accordance with service
policies and guidance. The key reasons for DOD's ineffective use of
business case analyses to support PBL decision making are that (1) DOD
has not required such analyses or provided specific criteria for
conducting them and (2) the services' internal controls have been
inadequate to ensure that the analyses are performed and updated. As a
result, DOD cannot ensure that decisions regarding weapon system
support options are guided by comprehensive, consistent, and sound
analytical tools. Further, the department cannot be sure that the
support arrangements being implemented will result in the most cost-
effective support program.
About Half of Programs Reviewed Either Did Not Use Business Case
Analyses to Guide Decisions to Enter into PBL Arrangements or Did Not
Retain Details of the Analysis:
For 9 of the 29 PBL arrangements we reviewed, a business case analysis
had not been completed. Additionally, for 6 others, program officials
could not provide supporting details of the business case analysis they
told us that they had conducted. When program offices did not conduct
business case analyses as part of their PBL decision making, as we
found for many of the programs we reviewed, the department cannot be
sure that the support arrangements implemented will result in the most
cost-effective support program.
Programs That Did Not Conduct a Business Case Analysis Before Entering
into PBL Arrangements:
Table 2 provides the number of PBL arrangements we reviewed by service
that were initiated with and without the use of a business case
analysis.
Table 2: Number of PBL Arrangements Initiated with and without the Use
of a Business Case Analysis:
Service: Army;
Prepared an analysis: 7;
Did not prepare an analysis: 2;
Total: 9.
Service: Navy;
Prepared an analysis: 6;
Did not prepare an analysis: 2;
Total: 8.
Service: Marine Corps;
Prepared an analysis: 2;
Did not prepare an analysis: 0;
Total: 2.
Service: Air Force;
Prepared an analysis: 5;
Did not prepare an analysis: 5;
Total: 10.
Service: Total;
Prepared an analysis: 20;
Did not prepare an analysis: 9; Total: 29.
Source: GAO analysis of DOD data.
[End of table]
Although both of the Marine Corps programs we reviewed conducted a
business case analysis, about 50 percent of Air Force programs, 22
percent of Army programs, and 25 percent of Navy programs did not. In
general, the Air Force programs only developed a source-of-repair
analysis, which evaluated only the repair element of weapon system
support. For the two PBL arrangements for which the Army did not
conduct an analysis--the Apache sensor and airframe arrangements--the
Deputy Assistant Secretary of the Army (Integrated Logistics Support)
approved the Apache program office's request for a waiver from Army
business case analysis policy based on prior analyses and program
decisions. However, the U.S. Army Audit Agency reported that the prior
analyses did not consider all components included in the two PBL
arrangements, other support strategies, performance metrics, and
relative costs.[Footnote 21] According to an Army official, a business
case analysis for the airframe components is being conducted and is
expected to be completed in December 2008, and efforts to develop a
business case analysis for the program's sensors are expected to begin
in November 2008.
The F-22A Raptor and KC-130J are examples of programs where the Air
Force and Navy did not conduct a business case analysis as part of the
PBL decision-making process. When DOD recommended in 2001 that program
offices fielding new systems base PBL arrangement decisions on business
case analyses, the F-22 was beginning low-rate initial production. In
2002, the Assistant Secretary of the Air Force (Acquisition) and the
Air Force Deputy Chief of Staff (Logistics, Installations and Mission
Support) directed the F-22 program office to develop a long-term
support strategy and manage life cycle product support through a PBL
arrangement that includes government-contractor partnerships as
necessary. From 2003 through 2007, the program office acquired support
as part of the aircraft's production contract, and in 2008 the office
signed a separate PBL support contract with Lockheed Martin, one of the
original equipment manufacturers, to support the aircraft from 2008 to
2012. In March 2008, program officials said that they did not conduct a
business case analysis before awarding the 2008 contract because
current program data, such as material usage rates and costs, are
immature. Officials planned to conduct an analysis in 2010 or 2011 when
it could be completed using more meaningful data. However, program
officials subsequently decided that the available data were sufficient
and in July 2008 awarded a contract to develop a business case
analysis. Completion of the analysis is expected in late 2009.
In 2002, the Navy contracted for a PBL arrangement to support the
Marine Corps' KC-130J engines without first preparing a business case
analysis. Program officials explained that the decision was made not to
develop an analysis because the technical data needed to repair the
engines were not available and the Marine Corps did not have the
infrastructure necessary to support the system. Officials also said
that a market analysis was conducted prior to implementing the PBL
arrangement, but they could not provide a copy of the analysis.
Nonetheless, a business case analysis that documented the results of
the market analysis and the program's negotiated costs versus expected
flight hours and anticipated repairs and maintenance could have been
used to monitor the actual results and cost-effectiveness of the
performance-based approach, especially since support for the engines is
obtained under a commercial contract and the contractor does not
provide detailed cost data to the government.
Supporting Documentation Not Fully Available for Some Programs That
Conducted Business Case Analyses:
Program officials from 20 of the PBL arrangements we reviewed told us
that they had conducted business case analyses before implementing the
arrangements; however, officials for 6 programs could not provide all
or some of the data to support the analyses. According to DOD's
economic analysis instruction, the results of the analysis, including
all calculations and sources of data, must be documented down to the
most basic inputs to provide an auditable and stand-alone document.
Table 3 lists by service the number of PBL arrangements for which all
of the business case analysis documentation was retained and those for
which it was not.
Table 3: Number of PBL Arrangements with and without Supporting
Documentation of a Business Case Analysis:
Service: Army;
Programs that did retain supporting documentation: 3;
Programs that did not retain all or some supporting documentation: 4;
Total: 7.
Service: Navy;
Programs that did retain supporting documentation: 5;
Programs that did not retain all or some supporting documentation: 1;
Total: 6.
Service: Marine Corps;
Programs that did retain supporting documentation: 2;
Programs that did not retain all or some supporting documentation: 0;
Total: 2.
Service: Air Force;
Programs that did retain supporting documentation: 4;
Programs that did not retain all or some supporting documentation: 1;
Total: 5.
Service: Total;
Programs that did retain supporting documentation: 14;
Programs that did not retain all or some supporting documentation: 6;
Total: 20.
Source: GAO analysis of DOD data.
[End of table]
In general, program officials for six programs said that they were
unable to locate all the details of their analyses; however, the amount
of data that was missing varied. For example:
* Although officials for the Army's Common Ground Station said that an
analysis was performed in 2002, they were unable to provide any details
of the analysis or the results.
* While program officials for the Army's Shadow Tactical Unmanned
Aircraft System were able to provide the results of their 2002
analysis, they did not retain the details regarding the assumptions,
data sources, or calculations used to develop the analysis. However,
program officials said that the analysis was developed early in the
life cycle of the program and was not based on historical cost and
maintenance data, and therefore they did not consider it to be very
accurate based on actual program results that have occurred since the
analysis was developed.
* For the Army's Javelin PBL arrangement, the DOD Office of the
Inspector General reported in 2005 that it was unable to validate the
program office's 2001 analysis because the program office was not able
to provide adequate documentation.[Footnote 22] The documentation has
not been located, and program officials were only able to provide a
summary of the results of the analysis.
Most Remaining Programs Used Business Case Analyses That Were Not
Comprehensive:
Service program officials could provide documentation of the business
case analyses conducted for 14 of the PBL arrangements we reviewed, but
all but 1 of the 14 analyses were missing one or more of the elements
recommended in DOD's economic analysis instruction. As a result,
decisions regarding weapon system support options for many of the
programs we reviewed were not guided by comprehensive, consistent, and
sound economic analysis. Further, the department cannot be sure that
the support arrangements implemented will result in the most cost-
effective support programs. Figure 1 shows which elements were missing
from the 14 business case analyses.
Figure 1: Elements Missing from the Business Case Analyses We Reviewed:
[Refer to PDF for image]
This figure is a table depicting the following data:
Service: Army;
Weapon system: Tube-launched Optically-tracked Wire-guided
missile”Improved Target Acquisition System;
Life cycle costs: [Empty];
Net present value: [Empty];
Objectives: Missing element;
Assumptions: [Empty];
Documentation of feasible and infeasible alternatives considered:
Missing element;
Comparison of feasible alternatives: [Empty];
Costs and benefits: [A];
Sensitivity and uncertainty analysis: Missing element;
Results and recommendations: Missing element.
Service: Army;
Weapon system: Sentinel Radar;
Life cycle costs: [Empty];
Net present value: [Empty];
Objectives: [B];
Assumptions: [Empty];
Documentation of feasible and infeasible alternatives considered:
Missing element;
Comparison of feasible alternatives: [Empty];
Costs and benefits: [Empty];
Sensitivity and uncertainty analysis: [Empty];
Results and recommendations: [Empty].
Service: Army;
Weapon system: Patriot Air Defense System;
Life cycle costs: Missing element;
Net present value: [Empty];
Objectives: [Empty];
Assumptions: [Empty];
Documentation of feasible and infeasible alternatives considered:
[Empty];
Comparison of feasible alternatives: [Empty];
Costs and benefits: [Empty];
Sensitivity and uncertainty analysis: [C];
Results and recommendations: [D].
Service: Navy;
Weapon system: V-22 engine;
Life cycle costs: [Empty];
Net present value: Missing element;
Objectives: [Empty];
Assumptions: [Empty];
Documentation of feasible and infeasible alternatives considered:
Missing element;
Comparison of feasible alternatives: [Empty];
Costs and benefits: [A];
Sensitivity and uncertainty analysis: Missing element;
Results and recommendations: [D].
Service: Navy;
Weapon system: Phalanx Close-In Weapon System;
Life cycle costs: Missing element;
Net present value: Missing element;
Objectives: Missing element;
Assumptions: [Empty];
Documentation of feasible and infeasible alternatives considered:
Missing element;
Comparison of feasible alternatives: [Empty];
Costs and benefits: [A];
Sensitivity and uncertainty analysis: Missing element;
Results and recommendations: Missing element.
Service: Navy;
Weapon system: AV-8B Harrier;
Life cycle costs: Missing element;
Net present value: Missing element;
Objectives: Missing element;
Assumptions: [Empty];
Documentation of feasible and infeasible alternatives considered:
Missing element;
Comparison of feasible alternatives: [Empty];
Costs and benefits: [A];
Sensitivity and uncertainty analysis: Missing element;
Results and recommendations: Missing element.
Service: Navy;
Weapon system: F/A-18 E/F[E];
Life cycle costs: [Empty];
Net present value: Missing element;
Objectives: [Empty];
Assumptions: [Empty];
Documentation of feasible and infeasible alternatives considered:
Missing element;
Comparison of feasible alternatives: [Empty];
Costs and benefits: [Empty];
Sensitivity and uncertainty analysis: Missing element;
Results and recommendations: [Empty].
Service: Navy;
Weapon system: Consolidated Automated Support System;
Life cycle costs: Missing element;
Net present value: Missing element;
Objectives: [Empty];
Assumptions: [Empty];
Documentation of feasible and infeasible alternatives considered:
Missing element;
Comparison of feasible alternatives: [Empty];
Costs and benefits: [A];
Sensitivity and uncertainty analysis: Missing element;
Results and recommendations: Missing element.
Service: Marine Corps;
Weapon system: Expeditionary Fighting Vehicle;
Life cycle costs: [Empty];
Net present value: [Empty];
Objectives: [B];
Assumptions: [Empty];
Documentation of feasible and infeasible alternatives considered:
[Empty];
Comparison of feasible alternatives: [Empty];
Costs and benefits: [Empty];
Sensitivity and uncertainty analysis: [Empty];
Results and recommendations: [Empty].
Service: Marine Corps;
Weapon system: Assault Breacher Vehicle;
Life cycle costs: [Empty];
Net present value: Missing element;
Objectives: [B];
Assumptions: [Empty];
Documentation of feasible and infeasible alternatives considered:
[Empty];
Comparison of feasible alternatives: [Empty];
Costs and benefits: [Empty];
Sensitivity and uncertainty analysis: [Empty];
Results and recommendations: [Empty].
Service: Air Force;
Weapon system: C-130J Hercules;
Life cycle costs: [Empty];
Net present value: [Empty];
Objectives: [Empty];
Assumptions: [Empty];
Documentation of feasible and infeasible alternatives considered:
[Empty];
Comparison of feasible alternatives: [Empty];
Costs and benefits: [Empty];
Sensitivity and uncertainty analysis: [Empty];
Results and recommendations: [Empty].
Service: Air Force;
Weapon system: C-17 Globemaster III;
Life cycle costs: [Empty];
Net present value: [Empty];
Objectives: [B];
Assumptions: [Empty];
Documentation of feasible and infeasible alternatives considered: [F];
Comparison of feasible alternatives: [F];
Costs and benefits: [Empty];
Sensitivity and uncertainty analysis: [Empty];
Results and recommendations: [Empty].
Service: Air Force;
Weapon system: B-2 Spirit;
Life cycle costs: Missing element;
Net present value: Missing element;
Objectives: [B];
Assumptions: [Empty];
Documentation of feasible and infeasible alternatives considered: [F];
Comparison of feasible alternatives: [F];
Costs and benefits: [Empty];
Sensitivity and uncertainty analysis: [C];
Results and recommendations: [Empty].
Service: Air Force;
Weapon system: Joint Surveillance and Target Attack Radar System;
Life cycle costs: [Empty];
Net present value: Missing element;
Objectives: [B];
Assumptions: [Empty];
Documentation of feasible and infeasible alternatives considered: [F];
Comparison of feasible alternatives: [F];
Costs and benefits: [Empty];
Sensitivity and uncertainty analysis: [Empty];
Results and recommendations: [Empty].
Source: GAO analysis of DOD data.
[A] Included costs but no analysis of nonquantifiable benefits.
[B] Included objective of document, not objective of warfighter or
warfighter need.
[C] Included either sensitivity or risk analysis, but did not include
both.
[D] Included results but no recommendation.
[E] Both the Navy Inventory Control Point and the Naval Air Systems
Command prepared a business case analysis for the F/A-18 E/F PBL
arrangement. We summarize the elements missing from both analyses.
[F] Did not analyze alternative support strategies.
[End of figure]
For three PBL arrangements, the business case analyses did not compare
alternative support options and either evaluated only a single option
for weapon system support or evaluated contracting strategies instead
of alternative support arrangements. For example, the 2007 business
case analysis for the Joint Surveillance and Target Attack Radar System
did not analyze the costs and benefits of alternative support
strategies for the program. The business case analysis was developed in
response to a 2006 recommendation from the DOD Office of the Inspector
General after a review found that the program office had not evaluated
alternative support strategies prior to implementing a PBL arrangement
in 2000.[Footnote 23] However, the 2007 analysis covered only the
potential impacts of changing the type of contract used to obtain
support for the program from cost plus award fee to firm fixed price.
[Footnote 24] The business case analysis for the B-2 also did not
analyze alternative support strategies but focused on potential
efficiencies available to the current program through funding
consolidation,[Footnote 25] funding stability, and long-term
contracting. According to program officials, the only assumption in the
analysis that actually occurred to some extent after PBL implementation
was funding consolidation. Finally, although the C-17 program office
developed a business case analysis in 2003, the DOD Office of the
Inspector General reported in 2006 that the analysis focused only on
one support option and did not evaluate the costs and benefits of
multiple support options.[Footnote 26] In 2007, a contract was awarded
for development of another business case analysis planned for
completion prior to awarding the next C-17 support contract.
Two other important elements missing from some analyses were an
evaluation of costs over the remaining life cycle of the program and
the calculation of net present value to account for the time value of
money.[Footnote 27] For example, the analysis for the Patriot PBL
arrangement only evaluated the costs and benefits over a 3-year period.
On the other hand, while the business case analysis for the Assault
Breacher Vehicle evaluated costs over the 20-year life cycle of the
program, the net present value of the alternatives was not calculated.
Four other business case analyses--those prepared for the F/A-18 E/F,
AV-8B Harrier, Close-In Weapon System, and Consolidated Automated
Support System--did not include these two elements and several others,
such as sensitivity or risk analysis. These analyses were prepared in a
similar format by the Naval Inventory Control Point, an organization
that provides supply support to the Navy, Marine Corps, and others. We
conducted a net present value analysis on the amounts contained in the
Naval Inventory Control Point's business case analysis for the F/A-18
E/F PBL arrangement and found that the PBL option it chose was about
$1.7 million more expensive than the alternative option. Its analysis,
which did not use net present value, found that the PBL option was
about $277,000 less expensive. The Naval Inventory Control Point's
philosophy is that if the costs of PBL are equal to or less than the
costs for government-provided support, a PBL arrangement will be used.
Therefore, if Naval Inventory Control Point officials had conducted a
net present value analysis, based on this decision criterion, they
would not have implemented the PBL arrangement. According to Naval
Inventory Control Point officials, there is confusion in the Navy
regarding the purpose of the analyses they prepare. Officials said that
the analyses were conducted for internal decision making and were not
intended to satisfy Navy PBL policy, which places responsibility for
development of a life cycle business case analysis on the weapon system
program office.[Footnote 28] However, program officials for the Close-
In Weapon System, Harrier, and Consolidated Automated Support System
did not develop business case analyses that evaluated PBL
implementation over the remaining life cycle of the system.
Other Factors Influencing the Quality of Business Case Analyses:
Several other factors affected the quality of the business case
analyses we reviewed. Most of the analyses we reviewed did not identify
and quantify the benefits that could be expected from contractor
incentives to increase reliability or improve processes to reduce
support costs. The only business case analysis that specifically
identified cost savings based on contractor incentives to reduce costs
was the 2005 life cycle business case analysis for the F/A-18 E/F. The
life cycle analysis included an estimate of future cost savings
associated with investments the contractor was willing to make under
the arrangement to reduce supply chain costs. In addition, most of the
analyses did not recognize or quantify the costs associated with the
transfer of risk that occurs under a performance-based support
arrangement. According to the DOD/Defense Acquisition University PBL
guide, PBL arrangements transfer responsibility for making support
decisions--and corresponding risk--to the support provider, and risk is
one of the major cost drivers for contractors. Therefore, the use of
performance metrics could introduce a large element of risk for the
contractor that may be built into the costs of such an arrangement. In
general, many of the business case analyses we reviewed simply
estimated the costs for contract logistics support and government-
provided support. One exception was the business case analysis for the
Marine Corps' Assault Breacher Vehicle, which attempted to address the
costs of risk transfer and the benefits of contractor incentive to
reduce costs by estimating the costs for a traditional contractor
logistics support arrangement and a performance-based contractor
logistics support arrangement, in addition to estimates for government-
provided support and performance-based, government-managed support.
Another business case analysis was based on questionable assumptions.
The 2002 business case analysis for the Sentinel program estimated that
the cost for the government depot to overhaul the system was 50 percent
of the total cost of the system. The business case analysis estimated
that for the alternative option--a partnership between the government
and the contractor with the government depot performing enough work to
meet the system's core requirements--the cost for an overhaul was only
25 percent of the system's cost. The analysis assumed that under the
partnership option the overhaul cost less because, instead of complete
disassembly and parts replacement, the system would be inspected and
repaired only as necessary. However, according to an official at the
Army depot that would perform the overhaul, the depot also could have
used the inspect and repair concept as the basis for its maintenance
work. Therefore, this assumption in the business case analysis may have
overstated the costs for the government depot to perform the work.
Lastly, the Naval Air Systems Command's 2005 life cycle business case
analysis for the F/A-18 E/F estimated that over the 28-year life cycle
of the program, PBL support costs were $76 million more expensive than
costs for government-provided support. However, the business case
analysis estimated that Naval Inventory Control Point surcharges would
be $325 million less for the PBL arrangement. The Naval Inventory
Control Point adds a surcharge to the cost of goods sold to its
customers (including weapon system program offices) to recoup its
expenses, but according to officials, they do not adjust their
resources based on PBL implementation and would still need to recoup
their expenses through surcharges to other customers. Therefore, while
the F/A-18 program office may realize a $325 million benefit from the
reduced surcharge, the overall costs to the Navy may remain the same.
Including this reduced amount in the analysis is inconsistent with
DOD's economic analysis instruction, which states that all measurable
costs and benefits to the federal government should be included in the
analysis. In addition, in the 2006 business case analysis prepared by
the Naval Inventory Control Point, which estimated supply chain
management costs for both PBL and government-provided support for a 5-
year period from 2006 through 2011, the Naval Inventory Control Point's
costs were estimated to remain the same under either option. If the
Naval Inventory Control Point's costs had been the same in the life
cycle business case analysis prepared by the Naval Air Systems Command
for the F/A-18 E/F, the PBL arrangement would be $401 million more
expensive than government support for the 28-year period.
Business Case Analyses Were Often Not Updated for Decision Making After
PBL Implementation:
In 2004, DOD guidance recommended that business case analyses continue
throughout the weapons system's life cycle and be updated at key
decision points both to validate the approach taken and to support
future plans. The services, with the exception of the Air Force, have
also issued policies on conducting such updates. However, even when
business cases were prepared, we found that program offices often did
not update them in accordance with DOD's guidance and service policies,
nor did the program offices validate them against actual support costs
for decision making after PBL implementation. Neither DOD nor the
services has issued guidance that specifies what should occur when
updating or validating a business case analysis.
Army policy states that PBL business case analyses shall be updated
prior to the exercise of an option period when there are significant
changes to the performance period/terms of the contract or evaluation
period, or updated whenever there are major programmatic changes or at
least every 5 years.[Footnote 29] Program offices for four of the Army
PBL arrangements we reviewed had not updated their business case
analyses as called for by Army policy. For example, the Tube-launched
Optically-tracked Wire-guided missile - Improved Target Acquisition
System program office developed a business case analysis in 1998 before
awarding the original PBL contract and has not updated or validated the
original analysis. Further, officials negotiated a follow-on PBL
contract in 2007 after the terms of the original contract were
complete. Although program officials for the Shadow Tactical Unmanned
Aircraft System had planned to complete an update to their 2002
business case analysis by the end of 2007, the effort was delayed and
the update is expected to be completed before the end of 2008. Although
the Javelin program office implemented a PBL arrangement in January
2004, the business case analysis was developed in 2001, and program
officials do not have plans to update the analysis. Additionally,
although program office officials for the Army's Sentinel weapon system
had updated their 2002 business case analysis, Army policy calls for
submission of the business case analysis to both the Office of the
Deputy Assistant Secretary of the Army for Integrated Logistics Support
and Army Materiel Command Headquarters for review and concurrence and
then to the Program Executive Office for approval. In February 2008,
before the new analysis had been reviewed and formally approved,
another PBL contract was awarded.
The Navy's 2007 business case analysis guidance calls for updates every
3 to 5 years or when significant programmatic changes occur.[Footnote
30] Based on this policy, the T-45 program office should complete an
update to its business case analysis for the PBL arrangement for
support of the aircraft's engines by 2008. Although program officials
updated their 2003 analysis in 2006 with actual support costs and
flying hours, they did not expand the analysis to account for the
remaining life cycle of the engines. The analysis projected costs only
through 2008, the original contract period. Program officials did not
plan to further update the business case analysis or prepare another
one because they believed that it was not required.
Neither DOD's nor the services' policies clearly specify what should
occur when a program office updates or validates a business case
analysis. Although some programs are conducting another business case
analysis, as mentioned earlier, program officials for the T-45 did not
plan to conduct another analysis because they had updated their
analysis with actual data. Program officials for the V-22 engine
updated their 1998 analysis in 2004. The update focused on assessing if
several of the ground rules, assumptions, and factors used in the
original study were still valid and providing a preliminary
recommendation on pursuing a follow-on PBL contract, from a cost
standpoint. However, the entire analysis was not updated.
Lack of Specific Criteria and Inadequate Internal Controls Led to
Business Case Weaknesses:
Business case analyses were inconsistently used for PBL decision making
because DOD did not require that the analyses be conducted and updated
or provide specific criteria to guide their development. Further, with
the exception of the Army, the services' have not established effective
internal controls to ensure that the analyses are prepared in
accordance with service policies and guidance. As a result, DOD cannot
ensure that decisions regarding weapon system support options are
consistently guided by comprehensive and sound analytical tools.
DOD Guidance Has Not Provided Specific Criteria or Requirements for
Conducting Sound Business Case Analyses:
DOD guidance has not provided specific criteria for conducting and
updating business case analyses for PBL decision making. Despite DOD's
preexisting economic analysis instruction recommending the analysis of
both quantitative and qualitative factors, in 2001 DOD recommended the
development of a business case analysis prior to implementing a PBL
arrangement but provided little criteria for conducting such an
analysis. In 2003, a Defense Business Board study recommended that DOD
issue standard guidance for the services to take a more consistent
approach to PBL decision making and also require the use of business
case analyses.[Footnote 31] In response, the January 2004 Under
Secretary of Defense (Acquisition, Technology and Logistics) memorandum
was issued containing "guiding principles" for business case analyses.
The memorandum stated that business case analyses "will evaluate all
services or activities needed to meet warfighter performance
requirements using 'best value' assessments." This memorandum also
listed several quantitative and qualitative factors[Footnote 32] for
consideration when developing a business case analysis; however, it did
not indicate how these factors were to be evaluated or their relative
importance in the decision-making process. The memorandum also
recommended that business case analyses be updated or repeated to
validate the approach taken or to support future plans, but did not
provide specific guidance as to when such updates should occur.
According to the memorandum, a DOD PBL business case analysis handbook
was supposed to be forthcoming. Later that year, the DOD/Defense
Acquisition University PBL guide was published. It had two pages
dedicated to the business case analysis concept--providing additional
criteria and also incorporating the guiding business case analysis
principles. However, a handbook specifically for PBL business case
analyses was never issued.
In 2003, when DOD incorporated PBL implementation into DOD Directive
5000.1, which provides mandatory policies for all acquisition programs,
a requirement to conduct and update a business case analysis was not
included. Specifically, the directive only stated that acquisition
managers shall use performance-based strategies for sustaining products
and services whenever feasible and that such PBL strategies shall
optimize total system availability while minimizing cost and logistics
footprint. Also, despite the Defense Business Board's recommendation
later that same year to require the use of business case analyses, DOD
subsequently neither required program managers to prepare the analyses
prior to PBL implementation nor required them to update the analyses
after implementation. In fact, although most of the services have
issued some guidance and requirements for business case analyses, the
current Defense Acquisition Guidebook[Footnote 33] no longer
specifically refers to a business case analysis, but rather recommends
the development of a "support strategy analysis" as part of the PBL
implementation process. According to the guidebook, the support
strategy analysis can be a business case analysis, economic analysis,
decision-tree analysis, or other best-value-type assessment.
Service Policies and Guidance for Conducting Business Case Analyses
Were Slow to Develop, Inconsistent, and Generally Not Enforced Because
of Lack of Effective Internal Controls:
Another reason for the inconsistent use of business case analyses is
that the services' policies and guidance for conducting the analyses
were slow to develop and were generally not enforced because of a lack
of effective internal controls. Moreover, we found inconsistencies
among the services' policies and guidance. In response to DOD's
recommendation that program offices conduct a business case analysis
prior to implementing a PBL arrangement for weapon system support, the
services issued their own policies and guidance. The time frames for
these are summarized in table 4.
Table 4: Service-Issued Business Case Analysis Policies and Guidance:
Service: Army;
2002: Assistant Secretary of the Army Memorandum[A];
2003: [Empty];
2004: U.S. Army Implementation Guide - Performance Based Logistics[B];
2005: Assistant Secretary of the Army Memorandum[C];
2006: Assistant Secretary of the Army Memorandum[D];
2007: Army Regulation 700-127[E];
2008: [Empty].
Service: Navy;
2002: [Empty];
2003: Navy PBL Guidance Document[F];
2004: Secretary of the Navy Instruction 5000.2C[G];
2005: [Empty];
2006: [Empty];
2007: Navy Guide for Developing PBL Business Case Analyses[H];
2008: [Empty].
Service: Air Force;
2002: [Empty];
2003: [Empty];
2004: Air Force Instruction 63-107[I];
2005: Assistant Secretary of the Air Force Memorandum[J];
2006: [Empty];
2007: [Empty];
2008: Air Force Instruction 65-509[K]; Air Force Manual 65-510[L].
Service: Marine Corps;
2002: [Empty];
2003: [Empty];
2004: [Empty];
2005: [Empty];
2006: [Empty];
2007: Marine Corps Order 4081.2[M];
2008: [Empty].
Source: GAO analysis of DOD data.
[A] Assistant Secretary of the Army (Acquisition, Logistics and
Technology), Memorandum, Army Implementation of Performance Based
Logistics (PBL) (Apr. 1, 2002).
[B] Deputy Assistant Secretary of the Army, Integrated Logistics
Support, U.S. Army Implementation Guide - Performance Based Logistics
(PBL) (Washington, D.C.: May 4, 2004).
[C] Assistant Secretary of the Army (Acquisition, Logistics and
Technology), Memorandum, Performance Based Logistics (PBL) Business
Case Analysis (BCA) Policy (Aug. 18, 2005).
[D] Assistant Secretary of the Army (Acquisition, Logistics and
Technology), Memorandum, Performance Based Logistics (PBL) for Army
Working Capital Fund (AWCF) Secondary Items (Apr. 25, 2006).
[E] Army Regulation 700-127, Integrated Logistics Support, September
27, 2007.
[F] Assistant Secretary of the Navy (Research, Development and
Acquisition), Department of the Navy Performance Based Logistics (PBL)
Guidance Document (Jan. 27, 2003).
[G] Navy Instruction 5000.2C, Implementation and Operation of the
Defense Acquisition System and the Joint Capabilities Integration and
Development System, November 19, 2004.
[H] Department of the Navy, Guide for Developing Performance Based
Logistics Business Case Analyses, P07-006 (Nov. 6, 2007).
[I] Air Force Instruction 63-107, Integrated Product Support Planning
and Assessment, November 10, 2004.
[J] Assistant Secretary of the Air Force (Financial Management and
Comptroller), Memorandum, Interim Guidance for Business Case Analysis
(Mar. 18, 2005).
[K] Air Force Instruction 65-509, Business Case Analysis, September 19,
2008.
[L] Air Force Manual 65-510, Business Case Analysis Procedures,
September 22, 2008.
[M] Marine Corps Order 4081.2, Marine Corps Performance Based Logistics
(PBL), January 5, 2007.
[End of table]
Although DOD recommended the use of business case analyses in 2001, the
services' business case analysis policies and guidance have evolved
over time. In some cases, guidance was not issued until years later.
For example, Marine Corps policy did not call for PBL business case
analyses until 2007. Further, although the Air Force included business
case analyses among mandatory procedures in 2004, these procedures were
not specific. The Air Force's instruction states only that "the program
manager is responsible for construction of a business case analysis to
determine the best strategy for meeting the PBL goals." Final Air Force
guidance for business case analyses, including PBL business case
analyses, was not issued until 2008. As another example, the Army's
early business case analysis guidance was general in nature, and more
specific policy memorandums were issued in 2005 and 2006. In 2007,
these policies were included in an Army regulation.
Currently, service policies and guidance vary with respect to which
programs should implement PBL arrangements, which of those programs
shall conduct business case analyses, and how often program managers
should update their business case analyses. Until 2007 the services'
policies and guidance varied significantly. However, the issuance of
Navy guidance and Marine Corps policy in 2007 resulted in more
consistency. Table 5 summarizes the services' business case analysis
policies and guidance.
Table 5: Current Service PBL and Business Case Analysis Policies and
Guidance:
Issue: PBL implementation;
Army: Shall be implemented on all ACAT programs where it is
operationally and economically feasible;
Navy: Preferred for all new programs and fielded ACAT I and II
programs;
Air Force: Shall be implemented for new ACAT I and II programs and
preferred for new ACAT III and fielded programs if practicable;
Marine Corps: Applied to all new acquisitions and existing fielded ACAT
I and II programs.
Issue: Business case analyses;
Army: Shall be performed to support PBL implementation;
Navy: Mandatory basis for selecting a support strategy;
Air Force: Required responsibility of program manager;
Marine Corps: Shall be developed for all ACAT I, II, III, and IV(T)
programs and for all fielded ACAT I and II programs.
Issue: Business case analyses for subsystem-and component-level PBL
arrangements;
Army: PBL may be implemented on systems, subsystems, secondary items,
components, assemblies, subassemblies, as well as processes, as
validated by a business case analysis;
Navy: Serves as a decision support tool for legacy subsystem-and
component-level PBL;
Air Force: Not specified;
Marine Corps: Shall be conducted at the system, subsystem, and
component levels.
Issue: Business case analysis updates;
Army: Shall occur at least every 5 years, or whenever there are major
programmatic changes, or prior to exercising contract options when
there are significant changes to the agreement;
Navy: Guide suggests updates at least every 5 years or when significant
programmatic or support changes occur;
Air Force: Not specified;
Marine Corps: Shall be revalidated at least every 5 years.
Source: GAO analysis of DOD data.
Note: Acquisition categories (ACAT) are generally based on the
acquisition program's location in the acquisition process, dollar
value, and Milestone Decision Authority special interest. DOD
Instruction 5000.2, Operation of the Defense Acquisition System, enc.
2, May 12, 2003.
[End of table]
With the exception of the Army, the services have not established the
internal controls, including a review and approval process, necessary
to ensure that business case analyses are conducted prior to PBL
implementation and updated after implementation. For example, the
Navy's 2003 guidance assigns responsibility for reviewing individual
business cases analyses to the system commands' cost departments.
However, the review only occurs when requested. Although a recently
issued Air Force instruction calls for a formal review of all business
case analyses, including those conducted for PBL arrangements, that
meet certain criteria, it is unclear how many PBL business case
analyses will meet any of the criteria and be subject to this review.
The 2008 Air Force instruction calls for a review of all business case
analyses that will be (1) forwarded outside of the Air Force; (2)
forwarded to senior Air Force officials, such as the Secretary of the
Air Force; and (3) provided for weapon systems that require Defense
Acquisition Board or Air Force Acquisition Board approval.
In contrast, Army policy states that program managers shall report
semiannually on the status of PBL implementation and that business case
analyses for acquisition category I and II programs should be submitted
for review and verification to multiple offices--including Army
headquarters, the Army Materiel Command, and the Office of the Deputy
Assistant Secretary of the Army (Cost and Economics)--and for approval
to the Army Acquisition Executive. In addition, business case analyses
for lower-level programs should be reviewed and approved but will not
verified by the Office of the Deputy Assistant Secretary of the Army
(Cost and Economics), and the approval authority is the program
executive officer or commander of the related life cycle management
command. While the Army's policy first provided for these internal
controls in 2005, Army officials said that no programs have yet passed
the review and approval process completely.
Impact of PBL Arrangements on Reducing Weapon System Support Costs Is
Unclear:
The extent to which PBL arrangements are reducing costs for weapon
system support is unclear and generally remains undocumented even after
several years of PBL implementation. A major difficulty in assessing
the cost impact of PBL arrangements is the lack of detailed and
standardized cost data maintained by the program offices. Various other
factors, such as the lack of systems that are supported by both PBL and
non-PBL support arrangements, the lack of sound program baseline
information, and changing operational and materiel conditions, also
limited our ability to assess the impact of PBL implementation on
support costs. While the overall cost impact was unclear because of a
lack of data and these other factors, the limited evidence on cost
impact that was available showed mixed results. We did find some
evidence that a few PBL arrangements have reduced costs. However, we
also found that characteristics of DOD's PBL support arrangements, such
as short-term contracts and unstable program requirements and funding,
may limit their potential to reduce costs. Further, DOD has not
sufficiently emphasized the potential to reduce costs as a goal for PBL
programs. As a result, DOD cannot be assured that PBL arrangements will
reduce support costs and provide cost-effective support for DOD
systems.
Program Managers Often Lacked Detailed and Standardized Cost Data:
In 2004, a memorandum from the Under Secretary of Defense (Acquisition,
Technology and Logistics) recognized the importance of cost data for
contract management and future cost estimating and price analysis and
stated that PBL contracts shall include cost reporting requirements.
[Footnote 34] However, for the PBL arrangements we reviewed, program
offices often did not have detailed cost data that would provide
insights regarding what the program office was spending for various
aspects of the support program--such as the cost of depot maintenance
by subsystem and major component or the cost of engineering support,
supply support, and transportation. When cost data were available, the
level of detail and format of cost data varied considerably. This
condition significantly affected our ability to determine the impact of
the implementation of PBL on the costs of supporting the systems in our
sample, as many factors influence support costs.
For PBL arrangements using fixed-price contracts or fixed-price
contract line items--DOD's "ideal" type of PBL arrangement--we found
that program offices generally did not receive detailed cost data for
the program and only knew the overall amounts paid for support. Only
two program offices in our sample obtained contractor support cost data
for their fixed-price PBL arrangements, and the format and contents of
the reports were very different. For example, the F/A-18 E/F program
office obtained Boeing's report on fiscal year 2006 costs, including
general/administrative costs and profit, in a detailed reporting
structure approved by the Office of the Secretary of Defense (OSD),
Cost Analysis Improvement Group. According to program officials, an
annual cost reporting requirement was included in the 2005 fixed-price
PBL contract to provide cost visibility into the program and was added
at no additional cost to the government. In contrast, the B-2 program
office receives a monthly funds utilization report that allocates the
amount the Air Force pays the contractor into seven high-level
categories, such as planned depot maintenance.
Although the PBL arrangements that used cost-reimbursable contracts
generally obtained more detailed cost data than those with fixed-price
contracts, the format and level of detail also varied. For example,
under the 1998 PBL contract, the F-117 program office did not receive
cost data in a format that was detailed enough to report in OSD's
standard support cost structure. The program office subsequently
required more detailed cost data reporting from Lockheed Martin in the
2006 PBL contract. As another example, the 2003 C-17 PBL contract has
both fixed-price and cost-reimbursable elements. According to program
officials, Boeing did not report support cost data at the level of
detail necessary to report in OSD's support cost structure under the
contract. According to an Air Force Cost Analysis Agency official, a
cost-reporting requirement was included in the contract's option years
and more detailed cost reporting will begin in fiscal year 2009.
Limited Available Evidence on Cost Impact Indicates Mixed Results:
Although cost data were generally lacking, the limited available
evidence on cost impact showed mixed results. Data we reviewed for the
two systems that were managed by both PBL and non-PBL arrangements
indicate that the PBL arrangements were more costly, but based on other
assessments of available data, there are some indications that PBL
arrangements can reduce costs. However, in seven out of eight programs
we reviewed where follow-on, fixed-price PBL contracts had been
negotiated, expected cost reductions either did not materialize or
could not be determined. Finally, we noted that officials reported
performance levels for some programs that were significantly higher
than required under the PBL arrangement, but it is unknown whether the
required levels could be achieved at a lower cost.
Two Comparisons of PBL Arrangements to Traditional Government Support
Show That PBL Arrangements Cost More:
Of the 29 programs we reviewed, only the F100-PW-220 engine and the KC-
130J/C-130J airframes are maintained by both PBL arrangements and
traditional government support strategies. We found that the Air
Force's traditional support arrangement for the F100-PW-220 engine
costs slightly less than the Navy's PBL arrangement for the same
engine. The Navy uses the F100-PW-220 engines in its F-16A/B aircraft
and sustains the engines through a PBL contract with Pratt & Whitney.
The Air Force uses the same engines in its F-16 and F-15 aircraft and
supports the engines at the Oklahoma City Air Logistics Center. The Air
Force maintains an engine total ownership cost estimate that includes
all costs incurred (depot-level repairables, general services division
(expendable repair parts), depot programmed equipment maintenance,
organizational-level maintenance, intermediate-level maintenance, and
continuous improvement program). To compare the Navy's PBL costs with
the Air Force's engine total ownership costs, we removed the costs
associated with organizational-level maintenance from the Air Force's
costs. As shown in figure 2, converted to costs per flight hour, the
Navy's PBL costs were slightly higher than the Air Force's costs in
constant fiscal year 2008 dollars.[Footnote 35]
Figure 2: Comparison of Navy PBL Costs with Air Force Government/
Traditional Costs:
[Refer to PDF for image]
This figure is a multiple line graph depicting the following data:
Fiscal year: 2004;
Air Force cost per flight hour: $1,767.71;
Navy cost per flight hour: $2,056.73.
Fiscal year: 2005;
Air Force cost per flight hour: $2,048.43;
Navy cost per flight hour: $2,279.99.
Fiscal year: 2006;
Air Force cost per flight hour: $1,879.47;
Navy cost per flight hour: $2,051.36.
Fiscal year: 2007;
Air Force cost per flight hour: $1,583.31;
Navy cost per flight hour: $1,645.31.
Source: GAO analysis of DOD data.
[End of figure]
Although the cost difference appears to be decreasing, the Navy's 5-
year contract ended in 2008 and a new PBL contract has not yet been
negotiated. The engines are currently being supported under a 6-month
extension to the original contract, and the fixed price paid per engine
cycle is significantly higher than that paid during the previous 5
years. According to Navy officials, the decision to contract with Pratt
& Whitney for the support of the Navy's engines was not solely based on
costs but was also based on other factors, such as turnaround time for
engine repairs. However, program officials could not provide the data
on which they based their decision.
Elements of the Air Force's PBL arrangement to support the C-130J
airframe are more expensive than the support for the KC-130J airframe
provided by the Navy. According to Navy officials, an analysis was
prepared in 2005 to compare costs for alternative repair arrangements
to determine whether to continue using the Navy's repair capability or
to transition to contractor-provided repair in 2006. The Navy's
analysis concluded that the support provided by the Naval Surface
Warfare Center, Crane Division, would cost 43 percent less than the
support provided by the contractor. The analysis was based on
anticipated 2006 flight hours, actual government support costs from
2005, and the costs to exercise an option for repair under a
preexisting contract. Additionally, we independently compared overall
costs for inventory management and repair of repairable components
[Footnote 36] and found that the Air Force's PBL costs on a per flight
hour basis for these elements were significantly higher than the Navy's
costs--approximately 131 percent higher in 2006 and 164 percent higher
in 2007. However, according to officials, several factors account for
some of the difference. For example, the Air Force's PBL arrangement
includes 36 percent more consumable and repair parts than the Navy's
support arrangement, maintenance of support equipment, and support for
six locations, while the Navy's arrangement includes support for only
three locations.
A Few Indicators Showed That PBL Arrangements Reduced Support Costs for
Some Programs:
Only a few of programs we reviewed were able to provide some indicators
of reduced weapon system support costs that could be attributed to the
use of a PBL arrangement. As mentioned earlier, some programs did not
have a business case analysis demonstrating how current support costs
compared to other support approaches. Of the nine PBL arrangements that
had been implemented and had a business case analysis that looked at
alternative support options, only four could be compared with PBL
contract costs. Based on this comparison, three of these four PBL
arrangements indicate potential savings from PBL implementation, while
the fourth is more expensive than estimated in the business case
analysis. The remaining analyses could not be compared to actual
program costs after PBL implementation because of programmatic changes
that occurred after the analyses were conducted.
The 2005 business case analysis for the Army's Patriot estimated a 3-
year cost savings of $1.6 million from using a PBL arrangement to
provide 107 high-demand parts.[Footnote 37] According to a program
official, the contract is in its final year and total obligations are
expected to be about $1 million less than estimated in the business
case analysis. Additionally, two business case analyses prepared by the
Naval Inventory Control Point estimated that supply chain management
support costs were reduced by implementing a PBL arrangement. The
business case analyses projected cost savings of about $2.2 million for
the 5-year Close-In Weapon System PBL arrangement awarded in 2006 and
$1.3 million for the 5-year Harrier PBL arrangement awarded in 2007.
[Footnote 38] Based on actual contract costs--and if the contracts are
not modified in the future--the total savings for these programs are
projected to be $5.2 million and $5.8 million, respectively. Although
the F/A-18 E/F business case analysis estimated a 5-year supply chain
management savings of approximately $1.4 million, the actual contract
cost is about $1.6 million more than the estimated amount in the
analysis. Given the difference, the PBL arrangement has not reduced
support costs for the program. As previously noted, two of the PBL
arrangements having evidence of reduced support costs are managed by
the Naval Inventory Control Point. This activity has used PBL
arrangements since fiscal year 2000 and has reported achieving cost
reductions as a result, using the Navy working capital fund to issue
longer-term, multiyear contracts that can extend up to 5 years in
length to support aircraft or ship subsystems or components. According
to agency officials, these longer-term agreements have enabled the
Naval Inventory Control Point to guarantee the contractors a more
stable business base, which provides contractors an incentive to make
investments to reduce costs. Overall, as a result of using PBL
arrangements, Naval Inventory Control Point officials estimate that
they have reduced support costs by approximately $26.7 million and
$63.8 million--or 2.8 and 5.8 percent--in fiscal years 2006 and 2007,
respectively.
Although the V-22 program conducted a business case analysis in 1998 to
estimate alternative costs for supporting the engines and projected
savings of $249.5 million[Footnote 39] over the 53-year life cycle of
the program, the analysis did not take into account the time value of
money and calculate savings based on net present value. For this and
other reasons, we cannot validate that the savings are reasonable. In
addition to DOD's economic analysis instruction, guidance from the
Office of Management and Budget also states that net present value is
the standard criterion for deciding whether a government program can be
justified on economic principles. In 2004, another analysis was
prepared for the V-22 engine program to determine (1) if several
assumptions used in the 1998 analysis were still valid and (2) the
impact of any changes to those assumptions on the cost savings estimate
for the PBL arrangement. The later analysis concluded that differences
in three of the original assumptions increased the projected PBL cost
savings to $305.9 million--an increase of $56.4 million.[Footnote 40]
The updated savings again were not calculated using net present value.
Moreover, although limited actual data were available, the calculations
generally made adjustments using assumptions that generated the maximum
potential savings for the PBL alternative. For example, when adjusting
the 1998 analysis to account for differences in the costs experienced
for excluded repairs (repairs that were not covered by the PBL
arrangement), the total potential PBL cost savings were increased by
$48 million because the average excluded repair cost was lower than
previously estimated. However, even though data showed that excluded
repairs occurred at a higher frequency than projected in the original
analysis, the later analysis did not adjust for the higher frequency of
excluded repairs. Thus, the savings calculation is questionable,
because the analysis noted that the frequency of these repairs could
eliminate all of the estimated cost savings. Finally, the 10-year-old
analysis has not been completely updated to estimate costs based on
actual data.
The remainder of the analyses could not be compared to current PBL
arrangement costs because of programmatic changes that occurred after
the analyses were conducted. For example:
* According to an Air Force C-130J program official involved in the
development of the 2004 business case analysis, the analysis was
conducted while the aircraft was supported by a commercial contract;
therefore, the program office did not have detailed cost data on which
to base the estimate. The estimate was developed, in part, using cost
data from other legacy programs and other assumptions that program
officials said did not turn out to be accurate. Thus, though the
business case analysis helped program officials develop the program's
support strategy, the cost estimates contained within are not useful
for monitoring current program costs.
* The 2002 business case analysis for the Army's Sentinel PBL
arrangement estimated costs for a 26-year period beginning in 2003
using a fleet size ranging from 126 to 198 radars. According to program
officials, since 2003 the fleet size has ranged from 140 to 143 radars
and additional radars are not anticipated. Although a new business case
analysis was prepared, it had not completed the Army's review and
approval process at the time of our review.
PBL Cost Reductions Either Did Not Materialize or Were Not Documented
for Most Follow-on Contracts:
Few of the remaining programs in our sample could document cost
reductions attributable to the use of a PBL arrangement after
negotiating a follow-on fixed-price contract. The PBL concept envisions
that support providers are incentivized to improve reliability to
ensure that performance metrics are met and reduce their costs to
provide support to increase profits--especially under fixed-price
arrangements. To the extent practicable, we examined how contract costs
changed for eight programs in our sample that negotiated follow-on
contracts or priced previously unpriced contract options after
completing fixed-price PBL contracts. According to officials, a variety
of factors affected the support costs negotiated in the PBL contracts
that caused both costs increases and decreases. Only one program had
decreasing support costs that program officials attributed to the use
of a PBL arrangement. One additional program supported under a cost-
plus-award-fee contract also reduced support costs by changing the
metrics included in the contract. However, these two programs did not
have updated business case analyses that analyzed alternative support
options over the remaining life cycle of the program. Finally, only one
program office had developed a methodology for tracking and verifying
reliability improvements made under the PBL arrangement, although this
is necessary for quantifying the related cost savings.
Support costs for the Navy's Consolidated Automated Support System have
decreased over the 8-year PBL arrangement that began in April 2000.
Program officials attribute the cost reductions the program has
experienced to the implementation of a PBL arrangement. Depending on
the level of support chosen, the fixed price charged for the annual
support of a test station decreased from 53 to 20 percent (constant
2008 dollars) from 2000 through 2008. Program officials said that they
closely monitored maintenance data and failure rates in order to
negotiate lower fixed prices where possible. In addition, officials
said that they were able to increase the number of repair and
consumable parts covered under the arrangement over the years.
According to officials, prior to the implementation of the PBL strategy
support costs for the program were even higher, but officials were
unable to locate the contracts.
Although support costs for a few of the other seven programs decreased,
officials said that there were a number of other factors that
influenced costs, such as changes in the scope of work or planned usage
of the systems. For example, according to Tube-launched Optically-
tracked Wire-guided missile - Improved Target Acquisition System
program officials, a variety of factors affected the costs negotiated
in the 2007 contract, and increased fleet size was one factor that
allowed them to negotiate lower rates per system. In addition, when the
first fixed-price PBL arrangement was implemented in 2001 the program
was fairly new with very few systems, so the program office did not
have an extensive amount of historical program data with which to
negotiate. Since 2001, the program office has collected actual data
that it used to negotiate lower rates in the latest contract. However,
according to program officials, the contractor only recently started
making changes to the system to improve reliability. These improvements
were not included in negotiations for the 2007-2011 contract but have
begun to improve failure rates and are expected to reduce costs in
future contracts.
Although the Army's Shadow Tactical Unmanned Aircraft System is not
supported by a firm-fixed-price PBL contract, program officials for the
system said that they were able to reduce support costs by changing the
performance metrics used in the PBL arrangement. The maximum amounts
authorized in the annual cost-reimbursable PBL contract for the support
of this system were reduced by 28 percent from fiscal years 2006
through 2007. According to program officials, a program office review
of PBL processes in early fiscal year 2006 concluded that while the PBL
arrangement was effective in terms of meeting the performance levels
specified in the contract, it was not cost efficient and costs
associated with the vehicle's high accident rate were an area of
particular concern. In response, the program office changed the
performance metrics in the contract to encourage the contractor to
improve reliability and reduce the accident rate, and also to improve
depot maintenance efficiency. As the accident rate improved, the
program office was able to negotiate for lower support costs in the
2007 PBL contract.
Finally, while the 2005 life cycle business case analysis for the F/A-
18 E/F program office estimated that support provided under a PBL
arrangement would be more expensive than government-provided support,
program officials for the Navy's F/A-18 E/F PBL arrangement have
developed a process to track and document support cost reductions
attributed to contractor investments to improve reliability and reduce
costs. Program officials said that both the Navy and Boeing have funded
initiatives to improve F/A-18E/F component reliability,
maintainability, and supportability as part of the Supportability Cost
Reduction Initiatives program. Under the current fixed-price PBL
arrangement, Boeing has invested approximately $11.39 million to fund
initiatives that officials currently estimate will generate cost
reductions of approximately $279 million over the remaining life cycle
of the system. According to program officials, Naval Air Systems
Command cost analysts have validated baseline estimates and will
annually track the results of the initiatives in order to quantify
actual support cost reductions attributed to the investments in the
future.
Costs to Achieve Higher-Than-Required Performance Levels Are Unknown:
According to program officials, eight of the PBL arrangements within
our sample of 29 systems generally achieved a level of performance that
significantly exceeded what is required under the contracts. According
to the DOD/Defense Acquisition University PBL guide, PBL arrangements
should be structured to meet the needs of the warfighter. Therefore, if
actual performance exceeds what is called for in the PBL arrangement,
it also exceeds the level of performance that is needed. According to
program officials, for eight of the PBL arrangements we reviewed, the
contractors significantly exceeded some of the contractual performance
requirements. For example:
* Since 2002, Army officials said that the average annual operational
readiness for the Tube-launched, Optically-tracked, Wire-guided
missile - Improved Target Acquisition System has not been below 99
percent, and the system's operational readiness has averaged 100
percent since 2004. According to a program official, the Army's
readiness standard for this system is 90 percent. Despite the Army's
standard, it continued to include a performance incentive that
encouraged higher levels of performance when negotiating a follow-on
PBL contract in 2007. The performance incentive includes payment of an
award fee that encourages operational readiness rates from 91 to 100
percent, with the highest award fee paid for 100 percent average
operational readiness.
* According to officials, since early 2005, monthly readiness rates for
the Army's Javelin have generally been measured above 98 percent.
However, the PBL contract for support of this system only requires 90
percent mission readiness.
* Although the contractual requirement for parts availability for the
Navy's V-22 engine PBL arrangement has been 90 percent since 1998,
according to program officials, actual parts availability has
consistently averaged 98 to 100 percent.
* Similarly, with availability averaging 98 percent since 2004, Air
Force program officials for the LITENING Advanced Airborne Targeting
and Navigation Pod said that the contractor has consistently exceeded
the contract requirement for 92 percent system availability.
For programs where performance significantly exceeded contractual
requirements, it is unclear how much extra was paid to get the added
performance. Since the government is paying for this excess
performance, then the arrangement, as structured, may not provide the
best value to the government, particularly since there are other DOD
programs that are not funded at levels that would be required to
achieve their desired level of readiness.
Characteristics of DOD's PBL Arrangements May Limit Their Potential to
Reduce Support Costs:
Several characteristics of DOD's PBL arrangements may limit their
potential to reduce costs. First, DOD's PBL contracts are limited to
relatively short time periods, while proponents of the PBL concept
believe that longer-term PBL arrangements are necessary to encourage
support providers to make investments to improve reliability. Second,
in DOD--where changing requirements and priorities can result in
fluctuations in the funding for support of DOD's weapon systems--
creating a stable level of funding is challenging. Third, many PBL
arrangements only transfer responsibility for inventory management to
the contractor and do not transfer inventory ownership, which reduces
incentives for ensuring a correctly sized inventory level. Finally,
many of DOD's PBL arrangements do not contain cost metrics or offer
specific incentives to encourage cost reduction initiatives.
Short-term Contracts:
According to program officials, DOD support contracts, including PBL
contracts, that are funded by operation and maintenance funds[Footnote
41] are generally limited to 1 year, and working-capital-funded
[Footnote 42] contracts are generally limited to 5 years, with
subsequent option years allowed up to a total of 10 years. However,
according to the DOD/Defense Acquisition University PBL guide, longer-
term PBL contracts are preferred because a key aspect of PBL is the
provision of incentives for contractors to reduce costs over time
through increased reliability while still making a profit. Further,
contract length should be sufficient to allow for an adequate return on
any investments made to improve reliability.
Officials from several PBL arrangements cited instances in which
reliability improvements were needed but contractors were hesitant to
make investments while under annual support contracts. For example,
Joint Primary Air Training System program officials said that during
the original 10-year PBL arrangement that began in 1996, the contractor
did not make any investments to improve unreliable components. Although
officials were expecting the fixed-price performance contract to
motivate the contractor to invest in improvements to increase
reliability and maximize profit, they found that the contractor
minimized its own costs during the contract period and passed on the
costs to improve the reliability of components with high failures to
the government when the contract was renegotiated. Our prior work found
that the private sector sometimes used PBL contracts of 10 to 12 years.
[Footnote 43]
Unstable Requirements and Funding:
Stable requirements and funding, like longer-term contracts, could
enable contractors to make reliability improvements and other business
decisions, such as long-term supplier arrangements, that could improve
performance and reduce future support costs because they have
reasonable assurance of future needs. For example, officials
representing one of the PBL arrangements we reviewed credited stable
funding for much of the program's cost savings. The F-117 program
office estimated that its arrangement would have cost over $80 million
more if the Air Force had not agreed to stabilize the program's support
budget and provide the necessary support funding each year of the
contract. However, DOD's requirements and priorities, and related
funding, for weapon system support are not always stable. For example,
according to Army officials, the Tactical Airspace Integration System's
PBL arrangement was affected by a significant reduction of the
program's support budget. The Army subsequently requested that the
Defense Acquisition University study the implications of funding on PBL
arrangements and prepare a case study based on this example. In
addition, for the last several years some of the Army's PBL
arrangements we reviewed did not receive all of their support funds at
the beginning of the fiscal year but rather in increments throughout
the year. Program officials for one Army system said that at one point
during fiscal year 2005, they almost had to discontinue some of the
support provided under the contract because they did not have adequate
support funds. Additional funding was eventually made available after
the program office notified its major command of the situation. Army
program officials said that this funding instability further
exacerbates the impact of having short-term contracts, since all of the
funds are not available to the contractor to make business arrangements
or investments for reliability improvements.
DOD's Ownership of Inventory:
Many of the PBL arrangements we reviewed only transferred
responsibility for inventory management, not ownership, to the
contractor. An analysis by Sang-Hyun Kim, Morris A. Cohen, and Serguei
Netessine of the Wharton School, University of Pennsylvania, suggests
that the efficiency of a PBL arrangement depends heavily on the asset
ownership structure: with higher ownership responsibility, the supplier
is more likely to spend more to increase reliability and less for
maintaining inventory levels.[Footnote 44] According to this study,
under an arrangement in which the contractor owns and manages the
inventory, reliability improvements and inventory costs will both be
evaluated in terms of their ability to meet performance metrics and
minimize costs. If the PBL arrangement only includes inventory
management, higher inventory levels may be used, instead of investments
to improve reliability, to meet performance metrics--particularly those
that measure availability--since inventory holding costs are not
incurred by the contractor. Consequently, under DOD's PBL arrangements,
contractors may choose to make fewer reliability improvements.
Availability of Cost Metrics and Effective Incentives:
Finally, many of DOD's PBL arrangements do not contain cost metrics or
offer specific incentives to encourage reduced costs. According to an
August 2004 memorandum from the Under Secretary of Defense
(Acquisition, Technology and Logistics) regarding performance-based
criteria, PBL should be constructed to purchase performance, which is
defined in terms of operational availability, operational reliability,
cost per unit usage, logistics footprint, and logistics response time.
The guidance recommended that PBL metrics be tailored to reflect the
unique circumstance of the arrangement, but still support desired
outcomes in terms of the five performance criteria. A subsequent 2005
memorandum from the Under Secretary of Defense (Acquisition, Technology
and Logistics) directed the use of these metrics as the standard set of
metrics for evaluating overall total life cycle systems management.
Some of the aviation PBL arrangements we reviewed negotiated their
support on a cost per flight hour basis. For those that did not, cost
per flight hour was generally not included as part of the contract
performance plan, with the exception of the F/A-18 E/F PBL arrangement.
For example, the C-17 program office did not negotiate its contract on
a per flight hour basis and does not monitor cost per flight hour as
part of its PBL arrangement. None of the nonaviation PBL arrangements
we reviewed included cost metrics as part of the PBL arrangement. In
addition, only four of the PBL arrangements we reviewed contained
incentives for reducing or controlling costs. For example, the F-117
and Shadow Tactical Unmanned Aircraft System PBL arrangements each
included a cost-sharing provision where the government and the
contractor would share annual savings if actual costs were below
negotiated costs. Further, officials said that the award plan for the F-
22 PBL arrangement also will consider how actual costs compare to
negotiated costs when calculating the amount of award fee the
contractor earns at the end of the year.
DOD Did Not Emphasize the Potential for PBL Arrangements to Reduce
Costs or Require the Collection and Reporting of Detailed Contractor
Support Cost Data:
Although PBL arrangements were included in a DOD pilot program intended
to demonstrate the ability of various initiatives to reduce support
costs, DOD did not emphasize this goal in its guidance or requirements
as it established the concept as the department's preferred weapon
system support strategy. In general, improved performance was given
greater emphasis, and we found only a few references to cost reduction
in DOD's guidance on implementing PBL arrangements. With respect to
requirements for cost reporting, DOD and the services do not require
that programs using PBL arrangements, or other contractor logistics
support arrangements, collect and report detailed cost data in a
consistent, standardized format.
Since 2001 DOD's guidance regarding PBL has emphasized higher levels of
readiness and stressed rapid implementation. For example, in 2001, when
DOD cited PBL as the preferred weapon system support strategy, PBL was
described as a strategy for achieving a higher level of system
readiness through efficient management and direct accountability. In a
2002 Under Secretary of Defense (Acquisition, Technology and Logistics)
memorandum, the services were instructed to prepare PBL implementation
plans that aggressively pursue the earliest feasible program
implementation end dates.[Footnote 45] A January 2004 Under Secretary
of Defense (Acquisition, Technology and Logistics) memorandum stated
that PBL was the department's near-term strategy to increase weapon
system readiness through integrated logistics chains and public/private
partnerships. The memorandum contained guidance to implement PBL where
economically feasible and provided guiding principles for a best-value
assessment. The following month a Deputy Secretary of Defense
memorandum again directed the services to provide plans for
aggressively implementing PBL arrangements.[Footnote 46]
In contrast to DOD's clearly stated goal to reduce support costs in the
late 1990s, we found few references to the potential for PBL to reduce
support costs since 2001. DOD guidance generally only indirectly refers
to potential PBL cost reductions to "compress the supply chain" and
"reduce non-value added steps." In May 2003, DOD Directive 5000.1, The
Defense Acquisition System, was updated to emphasize that program
managers shall implement PBL strategies "that optimize total system
availability while minimizing cost and logistics footprint." In March
2004, an Under Secretary of Defense (Acquisition, Technology and
Logistics) memorandum reiterated that PBL was the preferred strategy
and provided criteria on which to assess potential for PBL application.
[Footnote 47] One of the criteria stated that the cost per operational
unit of performance (such as a flying hour) should be capable of being
reduced through PBL implementation. Finally, in 2005, the DOD/Defense
Acquisition University PBL guide contained several references to the
potential for PBL to improve reliability and reduce costs.
Program offices often lacked detailed and standardized weapon system
support cost data because DOD has not required them to obtain and
report cost data from the contractors that provide such support,
including those involved in PBL arrangements. According to the OSD
Office of Program Analysis and Evaluation, historical operating and
support costs, organized in a standard format, are necessary for
preparation of life cycle cost estimates for new systems, budget
formulation, analysis of working capital funds, development of business
case analyses, and future contract negotiations. Until 2007, DOD's
guidance for structure of support cost estimates, which is also
suggested as a defined presentation format for historical operating and
support costs, included all contractor support--labor, materials,
overhead, and other assets--in one category, while government-provided
support was reported in greater detail among multiple categories and
lower-level subcategories.[Footnote 48] Therefore, amounts paid for
contractor support were generally reported in the aggregate. In October
2007, DOD changed its guidance to include a more detailed presentation
of contractor support costs in the various categories, similar to the
reporting of government support costs.[Footnote 49] However, neither
DOD nor the services have required program offices to obtain or report
contractor support costs, including PBL arrangements with contractors,
in this format.
OSD and service officials are beginning to recognize the need for
further visibility of the costs of support provided by contractors. In
late 2006, OSD's Office of Program Analysis and Evaluation began a
study regarding the collection of contractor support costs because the
department acknowledged that visibility into these costs in DOD's
systems was generally limited. Many of the programs studied were PBL
arrangements also included in our sample. OSD's study also found that
program offices often did not have detailed cost data and, if cost data
were provided, the data often did not conform to, or could not be
converted to, the standard support cost structure. Based on the study
results, OSD is considering requiring contractors to report their
actual costs for providing logistics support, including profit and
general and administrative expenses, in DOD's standard cost structure.
However, the details of the requirement and which programs will be
subject to such reporting have not been finalized.
Similarly, Air Force officials have also recognized the limitations on
visibility into contractor support costs for weapon systems. The Air
Force is currently considering expanding visibility by requiring that
all contractor-supported programs report actual obligations for
contractor labor and materials (including PBL arrangements) in each of
DOD's cost structure categories for each aircraft mission design
series. According to Air Force Cost Analysis Agency officials, this
requirement is different from the one being considered by OSD in that
the Air Force will have visibility over the Air Force's costs for
contractor support but not the contractor's actual costs.
Use and Characteristics of Availability Contracts by the United
Kingdom's Ministry of Defence:
The United Kingdom's Ministry of Defence also uses performance-based
arrangements to support its weapon systems. Ministry of Defence
officials refer to this initiative as contracting for availability.
Similar to DOD, when using availability contracts the Ministry of
Defence pays industry for aircraft, engines, or components to be
available for military operations, rather than paying for specific
repairs, spares, and technical support. According to officials, the use
of contracting for availability also started as an approach for
reducing costs for weapon system support. Ministry of Defence officials
said that their current contracts for availability generally provide
support for aviation systems, such as helicopters and combat aircraft.
Although there are maritime availability contracts, they said that most
of the ministry's maritime availability contracts support specific
types of equipment rather than entire ships. In general, the
availability contracts used by the ministry are significantly longer
than those used by DOD, and the ministry uses an "open book accounting"
arrangement to gain visibility into the contractors' costs to provide
support. According to officials, the annual budget for the Defence
Equipment and Support organization is approximately £13 billion,
including funds for conflict operations.
Ministry of Defence Budget Pressures Drove Need for Support Cost
Reductions:
In 1999, the United Kingdom's Defence Logistics Operation, one of two
[Footnote 50] entities that merged into the current Defence Equipment
and Support organization, established a goal to reduce costs 20 percent
by 2005/2006. According to Ministry of Defence officials, contracting
for availability began during this period as a way to maintain or
improve performance while assisting in achieving cost reductions. They
believe that if industry is paid for a given level of availability,
there are incentives to reduce support chain costs and make the weapon
system more reliable and processes more efficient. The cost reduction
goal was a key driver in the transformation of the maintenance, repair,
and overhaul activity for Harrier and Tornado fast jet aircraft. A
member of the Tornado Integrated Project Team stated that a number of
factors drove the support strategy change for the Tornado aircraft, but
the primary factor was the need to reduce costs to match budget
reductions; the team identified availability contracting as an
effective way to reduce costs and maintain performance. Officials also
stated that the support strategies for all of the ministry's
helicopters were changed because of increased budget pressures.
In 2007, the United Kingdom's National Audit Office reported that the
Ministry of Defence has experienced significant reductions in the costs
to support its fast jets; the Tornado and Harrier costs have been
reduced from a total of £711 million in 2001through 2002 to £328
million in 2006 through 2007, providing a cumulative saving of some
£1.4 billion over the 6-year period.[Footnote 51] The National Audit
Office reported that the savings were achieved by working with industry
to reform traditional contracts into availability contracts. However,
the report also stated that the ministry did not have sufficient data
to assess the impact of changes in the pattern of frontline operations
and productivity increases from the use of lean techniques on total
costs.[Footnote 52] National Audit Office officials with whom we met
confirmed that while they could validate overall cost reductions, they
could not attribute the entire savings solely to the use of
availability contracts. Other related initiatives, such as the
reorganization and reduction of locations for aircraft repair and
upgrade, the use of lean techniques, and the use of reliability-
centered maintenance, also contributed to the support cost reductions.
Analytical Tools Used for Decision Making:
Ministry of Defence officials said that they do not require the use of
availability contracts or promote their use as the preferred strategy.
According to officials, the support strategy can and should vary from
system to system depending on the circumstances; in some cases, it may
be appropriate for government activities to support some systems in the
traditional manner and for others to use contracting for availability.
To assist with the decision-making process, the Defence Equipment and
Support organization developed a "support options matrix" for use in
reviewing current and future support arrangements. Officials said that
the matrix was developed to assist with analyzing components of support
for cost and performance drivers, illustrating a range of support
options differentiated by the gradual transfer of cost and performance
drivers into industry management and presenting a clear rationale for
each support chain design in terms of the benefit to be derived from
the transfer of specific cost and performance drivers into industry
management. In addition to the matrix, a contractor capability
assessment is also completed to determine the ability of industry to
assume greater management responsibility. Finally, according to
officials, before they enter into a contract for availability, two
additional analyses are conducted. The first is an investment
appraisal, or an "internal value benchmark," which calculates the
lowest cost at which the service could be provided by the government.
The second is a business case analysis, which discusses the different
proposals and justifies the selection of the proposed approach.
Officials noted that the proposed approach does not have to be the
lowest-cost option, but is usually the option that offers the best
value solution overall.
In its 2007 report, the National Audit Office indicated that internal
value benchmarks were not developed consistently and recommended
development of improved guidance and consistent application of a common
methodology for benchmarks against which to assess the value of
proposed availability contracts. National Audit Office officials said
that they found variance in the quality of these cost estimates and a
shortage of qualified people for cost modeling. They also pointed out
that as less and less support is provided by the government, accurate
cost modeling for use when renegotiating contracts will become more
important, and the Ministry of Defence needs to maintain or improve
visibility of support costs for its weapon systems.
Long-term Contracts (19-25 Years) Are Critical to Success:
Defence Equipment and Support officials said that they have found the
long-term nature of availability contracts a key factor in reducing
costs and that annual contracts cannot achieve the same benefits as the
longer-term contracts do. According to officials, the long-term
contracts for Tornado aircraft and helicopter fleets reduced costs
because the contractors were able to stabilize their supply chain and
obtain better prices from the supplier base. The Ministry of Defence
also found that industry preferred long-term contracts. In a discussion
of contracting for availability, the "Defence Industrial Strategy," a
white paper dated December 2005,[Footnote 53] stated that companies are
generally interested in using availability contracts because it
provides the commercial firms with greater returns over a longer
period.
Ministry of Defence officials provided us with the following examples
of their long-term availability contracts:
* The Ministry of Defence has a 10-year contract with AgustaWestland to
support the Sea King helicopter until it is projected to be removed
from service. The Ministry of Defence has priced the contract for the
first 5 years, and thereafter it will establish the price in 5-year
increments.
* The Ministry of Defence has a 23-year contract with VT Group to
support two survey ships owned by the ministry. The contract has price
renegotiation points at the 7-, 15-, and 20-year points.
* The Ministry of Defence has a 19-year contract with BAE to support
the fleet of Tornado aircraft. The ministry awarded the contract in
December 2006 and priced it for the first 10 years.
* The Ministry of Defence has a 25-year contract with AgustaWestland to
support the Merlin helicopter until it is projected to be removed from
service. The price for the initial 5-year period of the contract is
fixed, and the ministry is currently negotiating prices for the next 5-
year period of performance that begins in 2011.
Other Characteristics of Ministry of Defence Availability Contracts:
Ministry of Defence officials said that other factors, such as
inventory ownership, contract incentives, and cost visibility, were
also important when contracting for availability. Officials told us
that they preferred to transfer not only management of inventory but
also inventory ownership under such arrangements. They noted that under
some of their current availability contracts this had not been possible
for a variety of reasons. Nonetheless, in the future they intend to
pursue transfer of inventory ownership as much as possible. Examples of
Ministry of Defence availability contracts where officials said that
inventory is owned by industry, also known as spares inclusive, include
a contract for support of two survey ships.
In addition, according to ministry officials, several of the
availability contracts--including those supporting the Sea King and
Merlin helicopters and Tornado fast jets--had incentives referred to as
gain share or pain share. In these types of arrangements, the
contractor and government share cost savings or cost overruns in
prenegotiated proportions. According to officials, they found that
these types of metrics are useful to influence contractor cost control
measures and provide an incentive for industry to develop changes and
modifications that reduce support costs. Officials familiar with the
Tornado fast jet availability contract explained that their arrangement
included gain sharing and pain sharing on both the variable and fixed-
price portions of the contract.
Finally, officials explained that in many of the Ministry of Defence's
availability contracts, the concept of open book accounting is
employed. Open book accounting is not a defined term but is more of a
general expression describing a level of access to accounting data that
would not normally be available under a conventional contract. In
availability contracts, open book accounting allows government program
officials to review the accounting records of the contractor. This
access is not without limits. Officials said that the level of access
must be agreed to in advance on a case-by-case basis and reflects the
circumstances of the arrangement and the need for access to certain
data to monitor performance or benefits arising from the arrangement.
For example, one contract may only provide for man-hour data because
that is all that needs to be shared given the circumstances. However,
another contract may allow access to direct cost, direct labor hours,
and other rates and factors that are relevant for the work involved.
According to officials, the Ministry of Defence has an open book
accounting agreement with AgustaWestland for the Merlin contract and
the government has full visibility of the accounts pertaining to
Merlin, including overhead costs. The contract must explicitly address
the data access arrangements and not rely on vague and undefined
phrases that could be open to misinterpretation.
The Ministry of Defence's Long-term Availability Contracts May Affect
Budget Flexibility:
According to the 2007 National Audit Office report, long-term
availability contracts may limit flexibility to respond to changes in
resources. In the past, integrated project team leaders in the Ministry
of Defence had some ability to move funding between resource lines to
overcome short-term funding issues. However, this flexibility is
diminishing because of the transition to availability contracts, as
larger portions of the budget are pre-allocated to fund these
contracts. The Mine Warfare Patrol and Hydrographic Team also raised
concerns about loss of budget flexibility. This team is responsible for
providing support for 2 hydrographic ships, 1 patrol ship (HMS Clyde),
3 River class ships, 16 mine hunters, and 38 smaller ships. The budget
for providing support to these ships is approximately £40 million, with
£18 million devoted to the long-term availability contracts for the 2
survey ships, 1 patrol ship, and 3 River class patrol ships. According
to Ministry of Defence officials, these arrangements have for the most
part been beneficial. However, as they are structured, these programs
do not allow for any flexibility. When the Mine Warfare Patrol and
Hydrographic Team recently had to absorb a 20 percent budget cut,
officials said that the mine hunter ships bore the brunt of the cut
because they had the majority of the remaining support budget not
earmarked for an availability contract. The team views the 20 percent
cut to its budget to be, effectively, a 40 percent cut to the mine
hunter ship budget. Defence Equipment and Support organization
officials said that they are looking to add more flexibility to future
availability contracts.
The Ministry of Defence has already incorporated some flexibility in a
few availability contracts. Officials said that the Tornado contract
contains both fixed-price elements for management team, training,
logistics, and information systems and a variable price element for
flying hours. Given this, the contract is fairly flexible and payment
is based on certain flying hour availability bands--with the bands
ranging from 70 to 110 percent availability in 10 percent increments
that are agreed to annually. As another example, officials explained
that the Merlin contract provides flexibility in that the prenegotiated
price is linked to banded flying hours with fixed and variable
elements. Under traditional contracting, they estimate that only 20
percent of the cost would vary with flying hours. Also, within the
basic contract parameters there is a provision for surge delivery for
the Merlin helicopter. Finally, according to officials, the Sea King
helicopter support contract has a similar flexibility because there are
a number of flying hour bands and each band has its own price. In this
manner, the Ministry of Defence can increase or decrease flying hours
without renegotiating the contract. Officials pointed out that one
drawback is that the price charged per flying hour at the lower bands
is higher because the contractor must be able to cover fixed costs with
fewer flight hours to charge. However, they said that the cost per
flying hour is still far less than it would have been under a more
flexible traditional arrangement.
Conclusions:
While the concept of using PBL support arrangements was intended to be
a cost reduction strategy as well as one that would result in improved
performance, DOD's emphasis has been more focused on performance and
less focused on cost. DOD no longer emphasizes reducing costs as a goal
for PBL programs, and DOD's implementation of PBL, in its current form,
does not ensure that its PBL arrangements are cost effective. DOD's
emphasis on the implementation of PBL as the preferred weapon system
support strategy has deemphasized the importance of the development of
consistent, comprehensive, and sound business case analyses to
influence decisions regarding the use of a PBL arrangement. Although
DOD's guidance recommends using business case analyses to guide
decisions about using PBL arrangements for weapon system support, the
DOD guidance does not require these analyses and almost half of the
programs we reviewed either did not perform a business case analysis or
did not retain documentation of their analysis. Further, the quality of
the analyses of those programs that had performed a business case
analysis varied considerably since many were missing elements of what
DOD guidance recommends for sound economic analyses. Additionally, most
of those analyses that should have been updated had not been. Thus, DOD
lacks a sound approach for analyzing whether proposed PBL arrangements
are the most cost-effective strategy for supporting weapon systems.
Without instituting a more consistent, comprehensive, and sound process
on which to base decisions regarding the type of arrangement to be used
in supporting DOD systems, it is unlikely that the department will be
successful in achieving the sizable savings that were envisioned when
the PBL concept was adopted.
Assessing the cost-effectiveness of PBL programs also requires the
availability of better cost data at a level of detail that would
support the improved management of ongoing PBL programs, including
awarding contract fees, assessing performance versus the cost to
achieve it, evaluating historical costs to determine whether the status
quo should be maintained over time, and making support decisions about
future follow-on programs. Such data are usually not available for PBL
programs, limiting the ability of program offices to make program
adjustments or take restructuring actions when appropriate.
Nonetheless, a few program offices have acquired data at this level and
indicate that they obtained them in a cost-effective manner. Improved
access to detailed cost data is another essential element in improving
the quality of data available to DOD decision makers regarding the cost-
effectiveness of PBL arrangements.
Recommendations for Executive Action:
To ensure that PBL arrangements are the most cost-effective option for
weapon system support, we recommend that the Secretary of Defense
direct the Under Secretary of Defense (Acquisition, Technology and
Logistics) to take the following five actions:
* revise DOD's Acquisition Directive to require development of a
business case analysis to support the decision-making process regarding
weapon system support alternatives, including PBL;
* revise PBL business case analysis guidance to more clearly define
what should be included in a business case analysis and to establish
specific criteria and methods for evaluating PBL support arrangements,
including evaluation at the subsystem and component levels;
* revise PBL business case analysis guidance to more clearly define
when business case analyses should be updated during the weapon system
life cycle;
* require that each service revise guidance to implement internal
controls to ensure that program offices prepare and update business
case analyses that are comprehensive and sound; and:
* require program offices to collect and report cost data for PBL
arrangements in a consistent, standardized format with sufficient
detail to support traditional cost analysis and effective program
management.
Agency Comments and Our Evaluation:
In written comments to a draft of this report (see app. II), DOD
generally concurred with our five recommendations, noting that the
department is committed to evaluating PBL strategies using business
case analyses as part of the overall supportability assessment made
during the development stages of weapon system acquisition programs.
Specifically, the department fully concurred with three recommendations
and partially concurred with two.
DOD fully concurred with our first recommendation to revise DOD's
acquisition directive to require the development of a business case
analysis to support the decision-making process regarding weapon system
support alternatives, including PBL. DOD stated that the department
will take steps to address this issue in the next iteration of the DOD
Directive 5000.1 and DOD Instruction 5000.2 acquisition regulations.
According to DOD's response, this new policy will require that the use
of a business case analysis be mandatory and that this analysis serve
as a sound basis for the selected supportability strategy.
In response to our second recommendation to revise PBL business case
analysis guidance to clearly define what should be included in a
business case analysis and to establish specific criteria and methods
for evaluating PBL support arrangements, DOD partially concurred,
stating that it established a Life Cycle Product Support Assessment
Team in September 2008 to study product support policy, guidance, past
performance, and results. As part of the study, existing business case
analysis policy is being reviewed, and the department will evaluate the
team's recommendations on providing specific criteria and methods for
evaluating support arrangements and determine how best to incorporate
these recommendations into mandatory policy. The team's initial
recommendations are expected in April 2009.
DOD fully concurred with our third recommendation to revise PBL
business case analysis guidance to more clearly define when, during the
weapon system life cycle, business case analyses should be updated.
According to DOD's response, the department's Life Cycle Product
Support Assessment Team will evaluate the appropriate timing of initial
business case analyses and follow-on updates to validate the life cycle
support approach for weapon systems, and the team's recommendations
will be evaluated for inclusion into mandatory policy.
DOD fully concurred with our fourth recommendation to require that each
service revise guidance to implement internal controls to ensure that
program offices prepare and update business case analyses that are
comprehensive and sound. As we noted in our report, the Army has
already implemented a PBL business case analysis review and approval
process. DOD stated that the Army's internal controls will be reviewed
by the Life Cycle Product Support Assessment Team, which will make
recommendations for expansion for DOD-wide governance policy as part of
the team's overall recommendations expected in April 2009.
DOD partially concurred with our fifth recommendation to require
program offices to collect and report support cost data for PBL
arrangements in a consistent, standardized format with sufficient
detail to support traditional cost analysis and effective program
management. DOD stated that a provision for tailored cost reporting for
major acquisition programs designed to facilitate future cost
estimating and price analysis has been included in the draft DOD
Instruction 5000.2, which is expected to be approved in the next 30
days. Additionally, the Life Cycle Product Support Assessment Team is
reviewing support cost reporting and cost analysis as a part of its
ongoing study. According to DOD's response, the ultimate goal is
standardized support cost reporting for all life cycle product support
efforts, to include support provided by government activities.
While concurring with our recommendations, DOD's response noted that
the department disagrees with the assertion that the goal of PBL
arrangements is to reduce costs. Rather, the primary goal of PBL
arrangements is to increase readiness and availability while reducing
overall sustainment costs in the long run. Our report recognized that
the current DOD Directive 5000.1 provides that PBL arrangements shall
optimize total system availability. However, our report notes that this
directive also provides that PBL arrangements shall minimize costs and
the logistics footprint. Moreover, our report stated that PBL emerged
from a 1999 DOD study to test logistics reengineering concepts that
placed greater reliance on the private sector for providing weapon
system support to both reduce support costs and improve weapon system
performance. Thus, reducing costs was a central focus of the adoption
of PBL as DOD's preferred support strategy. Based on our analysis in
this report, we continue to believe that the PBL support arrangement
concept was intended to be a cost reduction strategy as well as a
strategy that would result in improved performance.
DOD's response also noted that 22 of the 29 programs we reviewed
produced business case analyses that enabled sound support strategy
determinations. DOD further stated that for 28 of the 29 programs, the
PBL strategies produced performance benefits, readiness benefits, or
both, and 15 of the programs reflect cost-neutral or savings benefits
resulting from the application of the PBL strategies. However, based on
our analysis in this report, we continue to believe that only 20,
rather than 22, of the programs had business case analyses that
evaluated PBL strategies. Further, as we stated in our report, 6 of
these did not retain some or all of the documentation and 13 were
missing elements of DOD's criteria for economic analyses. For example,
we found that for one analysis the less costly option would have
changed if the department had calculated the net present value of the
two options considered. Additionally, because the department did not
document all the potential support options in the business case
analyses, it is not possible to determine if the most cost-effective
options were chosen. Thus we continue to question the extent to which
these analyses enabled sound support strategy determination. Finally,
while we recognize that the PBL arrangements may have produced
performance benefits, readiness benefits, or both, deficiencies in
updated business case analyses and detailed cost data did not support
an assessment of support costs. Therefore, it is unclear how many of
the programs may have actually had cost-neutral or savings benefits
resulting from PBL strategies. We continue to believe that improvements
in collection and reporting of support cost data and the updating of
business case analyses are essential if DOD is to determine the cost-
effectiveness of its PBL arrangements.
We are sending copies of this report to interested congressional
committees and the Secretary of Defense. The report is also available
at no charge on the GAO Web site at [hyperlink, http://www.gao.gov].
If you or your staff have any questions regarding this report, please
contact me at (202) 512-8365 or solisw@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. Key contributors to the report are listed
in appendix III.
Signed by:
William M. Solis:
Director, Defense Capabilities and Management:
[End of section]
Appendix I: Scope and Methodology:
To evaluate (1) the extent to which the Department of Defense (DOD)
used business case analyses to guide decisions regarding performance
based logistics (PBL) arrangements and (2) the impact PBL arrangements
have had on weapon system support costs, we selected a nonprobability
sample of 29 PBL arrangements for weapon system support initiated from
1996 through 2007. The 29 PBL arrangements were selected from lists of
weapon systems supported by PBL arrangements provided by service
officials. With the exception of the Navy's, we found that the lists
provided by the services either were not current or contained
inaccuracies, and the content of the lists changed significantly during
the course of our review, which affected our sample selection. We chose
system-, subsystem-, and component-level PBL arrangements from each of
the services based on length of time since implementation, location of
program office, dollar value, and prior audit findings. The 29 PBL
arrangements we selected constitute a nonprobability sample and the
results are not generalizable to the population of PBL arrangements.
To evaluate the extent to which DOD used business case analyses to
guide decisions regarding PBL arrangements, we interviewed officials
regarding DOD and service requirements, policies, and guidance for
business case analyses since 2001 and reviewed applicable documents. We
also reviewed DOD's 1995 economic analysis instruction, which states
that analytical studies that evaluate the cost and effectiveness of
weapon system support are considered to be "economic analyses," and
determined that the guidance is consistent with Office of Management
and Budget guidance for benefit-cost analyses of federal programs.
[Footnote 54] We interviewed program officials to discuss any business
case analyses prepared to evaluate the 29 PBL arrangements before or
after PBL implementation and examined the analyses using the criteria
contained in DOD's economic analysis guidance.
To evaluate the impact that PBL arrangements have had on weapon system
support costs, we interviewed program officials to discuss the
characteristics of the PBL arrangements, including contract length,
contract type, scope of work, performance measures, performance
incentives or disincentives, and cost data availability. In addition,
we asked program officials to identify support cost reductions that
occurred as a result of PBL implementation. If a program had renewed a
fixed-price PBL arrangement or had finalized contract options that were
not priced, we analyzed the contracts for trends in PBL support costs.
We also compared PBL contract costs to estimated PBL support costs in
business case analyses, where available, to determine how closely the
estimates matched the actual PBL arrangement costs. We also relied on
previously issued GAO reports on DOD's implementation of PBL.
To analyze the use of availability contracts for weapon system support
by the United Kingdom's Ministry of Defence, we interviewed officials
from the Defence Equipment and Support organization regarding policies
or requirements for availability contracts and trends regarding the use
of these arrangements. We also interviewed officials from programs
identified by the Ministry of Defence as using availability contracts
for weapon system support to identify the characteristics of the
specific arrangements and the impact that the use of these contracts
had on support costs. In addition, we interviewed National Audit Office
officials who reviewed the cost and performance of two availability
contracts for support of fast jets. Finally, we reviewed audit reports
and other documents from the Ministry of Defence and National Audit
Office. We obtained these data for informational purposes only and did
not independently verify the statements or data provided by Ministry of
Defence and National Audit Office officials.
Specifically, in performing our work we interviewed officials and
obtained documents related to PBL at the Office of the Under Secretary
of Defense (Acquisition, Technology and Logistics), the Office of the
Assistant Secretary of the Navy (Research, Development and
Acquisition), the Office of the Assistant Secretary of the Air Force
(Acquisition), the Office of the Assistant Secretary of the Air Force
(Installations, Environment and Logistics), the Office of the Assistant
Secretary of the Army (Acquisition, Logistics and Technology), the
Marine Corps Headquarters, the U.S. Army Materiel Command, the U.S.
Army Aviation and Missile Command, the U.S. Army Communications and
Electronics Command, the Naval Sea Systems Command, the Naval Air
Systems Command, the Naval Center for Cost Analysis, the Air Force Cost
Analysis Agency, the Air Force Directorate of Economics and Business
Management, the Air Force Materiel Command, the Air Force Aeronautical
Systems Center, the Oklahoma City Air Logistics Center, the Warner
Robins Air Logistics Center, the Ogden Air Logistics Center, the United
Kingdom Ministry of Defence, and the United Kingdom National Audit
Office. We conducted this performance audit from February 2007 through
December 2008 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objectives.
We believe that the evidence obtained provides a reasonable basis for
our findings and conclusions based on our audit objectives.
[End of section]
Appendix II: Comments from the Department of Defense:
Deputy Under Secretary Of Defense For Logistics And Materiel Readiness:
3500 Defense Pentagon:
Washington, DC 20301-3500:
December 3, 2008:
Mr. William Solis:
Director, Defense Capabilities and Management:
U.S. Government Accountability Office:
441 G Street, N.W.
Washington, DC 20548:
Dear Mr. Solis:
This is the Department of Defense (DoD) response to the GAO draft
report, GAO-09-41, "Defense Logistics: Improved Analysis and Cost Data
Needed to Evaluate the Cost Effectiveness of Performance Based
Logistics," dated November 4, 2008 (GAO Code 350975).
The Department generally concurs with the recommendations presented in
the report on strengthening Business Case Analyses (BCA) guidance for
evaluating supportability strategies. Further, the Department will take
steps to update Acquisition Policy documents such as the 5000 series of
DoD regulations to include guidance to Program Managers on basing
supportability decisions on a BCA. However, the Department disagrees
with the assertion that the goal of performance based logistics (PBL)
arrangements is to reduce costs. Rather, the primary goal of PBL
arrangements is to increase readiness and availability while reducing
overall sustainment costs in the long run.
The Department is committed to evaluating PBL strategies via BCAs as
part of the overall supportability assessments made during the
development stages of weapon system acquisition programs. Moreover, the
Department maintains that of the 29 programs examined by the GAO, 22
produced BCAs that enabled sound support strategy determinations.
Further, for 28 of the 29 programs, the Military Departments assess
that the PBL strategies produced performance and/or readiness benefits
for the operational warfighter, and 15 of the programs reflect cost-
neutral or savings benefits resulting from application of the PBL
strategies. This number should increase as data becomes available on
the other programs.
Detailed DoD comments on the draft GAO recommendations are provided in
the enclosed. The DoD appreciates the opportunity to comment on the
draft report.
Sincerely,
Signed by:
Jack Bell:
Enclosure: As stated:
GAO Draft Report - Dated November 4, 2008:
GAO Code 350975/GAO-09-41:
"Defense Logistics: Improved Analysis and Cost Data Needed to Evaluate
the Cost Effectiveness of Performance Based Logistics"
Department Of Defense Comments To The Recommendations:
Recommendation 1: The GAO recommends that the Secretary of Defense
direct the Under Secretary of Defense (Acquisition, Technology and
Logistics) to revise DoD's acquisition directive to require development
of a business case analysis to support the decision-making process
regarding weapon system support alternatives including performance
based logistics. [See page 55]
DOD Response: Concur. Although this is already stated by Department
policy in the Performance Based Logistics Guide as mentioned in the
draft report, the Department will take steps to include in the next
iteration of the DoDD 5000.1 and DoDI 5000.2 acquisition regulations
policy that will require use of a business case analysis be mandatory
and that it should serve as a sound basis for the selected
supportability strategy.
Recommendation 2: The GAO recommends that the Secretary of Defense
direct the Under Secretary of Defense (Acquisition, Technology and
Logistics) to revise Performance Based Logistics (PBL) business case
analysis guidance to more clearly define what should be included in a
business case analysis and to establish specific criteria and methods
for evaluating PBL support arrangements, including evaluation at the
subsystem and component level. [See page 55]
DOD Response: Partially concur. As stated in the draft report the
Department has issued business case analysis guidance in the
Department's Performance Based Logistics Guide, in past Department
memos, and in Chapter 5, "Life Cycle Logistics," of the Defense
Acquisition Guidebook. However, all of these documents are considered
guidance in nature and are not mandatory. In September 2008, the
Department established a Life Cycle Product Support Assessment Team
composed of DoD, Service, Industry, and Academia representatives to
review existing product support policy, guidance, past performance, and
results. The goal of this study is to provide recommendations for
furthering life cycle product support within the Department, and
outline a plan for increasing the effectiveness of future life cycle
product support strategies for weapons systems. Existing business case
analysis policy is being reviewed by this group, and the Department
will evaluate its recommendations on providing specific criteria and
methods for evaluating support arrangements. The Department expects an
initial draft version of the team's recommendation in April 2009. The
Department will then review the recommendations and determine how best
to structure the team's business case analysis recommendations for
inclusion into mandatory policy.
Recommendation 3: The GAO recommends that the Secretary of Defense
direct the Under Secretary of Defense (Acquisition, Technology and
Logistics) to revise Performance Based Logistics (PBL) business case
analysis guidance to more clearly define when business case analyses
should be updated during the weapon systems' life cycle. [See page 55]
DOD Response: Concur. As stated in our response to Recommendation 2, a
Product Support Assessment Team has been chartered by the Department to
evaluate the specifics of business case analysis to include the
appropriate timing of initial and follow-on updates to validate the
life cycle sustainment approach for weapons systems. The Department
expects an initial draft version of the team's recommendation in April
2009. The Department will then determine how best to include mandatory
policy for updating business case analyses.
Recommendation 4: The GAO recommends that the Secretary of Defense
direct the Under Secretary of Defense (Acquisition, Technology and
Logistics) to require that each Service revise guidance to implement
internal controls to ensure that program offices prepare and update
business case analyses that are comprehensive and sound. [See page 55]
DOD Response: Concur. As noted in the report the Army has already
implemented internal controls, including a review and approval process,
necessary to ensure that businesses case analyses are conducted prior
to performance based logistics implementation. The Department will
ensure the Product Support Assessment Team reviews the Army's processes
and make recommendations for expansion for DoD-wide governance policy
as part of the team's overall recommendations that are expected in
April 2009.
Recommendation 5: The GAO recommends that the Secretary of Defense
direct the Under Secretary of Defense (Acquisition, Technology and
Logistics) to require program offices to collect and report cost data
for Performance Based Logistics (PBL) arrangements in a consistent,
standardized format with sufficient detail to support traditional cost
analysis and effective program management. [See page 55]
DOD Response: Partially concur. As a first step, the Department has
included in the draft DoDI 5000.2 expected for approval within the next
thirty days a requirement for the provision of tailored cost reporting
designed to facilitate future cost estimating and price analysis on
major acquisition programs. Additionally, the Product Support
Assessment Team is reviewing life cycle product support sustainment
cost reporting and cost analysis as specific study areas, with the
ultimate goal of standardized sustainment cost reporting for all life
cycle product support efforts to include organic activities. The
Department expects to see an initial draft version of the group's
recommendation in April 2009. The Department will then review the
recommendations and determine how best to structure them for
standardized reporting.
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
William M. Solis, (202) 512-8365 or solisw@gao.gov:
Acknowledgments:
In addition to the contact named above, Julia Denman, Assistant
Director; Harold Brumm; Matt Dove; Jennifer Echard; Chaneé Gaskin; Tom
Gosling; Jennifer Jebo; Mae Jones; Kevin Keith; Charles Perdue; Janine
Prybyla; and Karen Thornton made major contributions to this report.
[End of section]
Footnotes:
[1] "Weapon system support" includes functions such as materiel
management, distribution, technical data management, maintenance,
training, cataloging, configuration management, engineering support,
obsolescence management, technology refreshment, and repair parts
management.
[2] Department of Defense, Product Support for the 21st Century: A
Program Manager's Guide to Buying Performance (Washington, D.C., Nov.
6, 2001); Under Secretary of Defense (Acquisition, Technology and
Logistics), Memorandum, Performance Based Logistics (PBL) Business Case
Analysis (BCA) (Jan. 23, 2004); Under Secretary of Defense
(Acquisition, Technology and Logistics), Memorandum, Performance Based
Logistics (PBL) and Business Case Analysis (BCA) (Mar. 20, 2004); and
Department of Defense, Defense Acquisition University, Performance
Based Logistics: A Program Manager's Product Support Guide (Fort
Belvoir, Va., March 2005).
[3] GAO, Defense Management: DOD Needs to Demonstrate That Performance-
Based Logistics Contracts Are Achieving Expected Benefits, [hyperlink,
http://www.gao.gov/products/GAO-05-966] (Washington, D.C.: Sept. 9,
2005), and Defense Management: Opportunities to Enhance the
Implementation of Performance-Based Logistics, [hyperlink,
http://www.gao.gov/products/GAO-04-715] (Washington, D.C.: Aug. 16,
2004).
[4] Of the 29 PBL arrangements, 6 were first initiated before 2001 when
DOD guidance recommended conducting a business case analysis; however,
4 of the 6 negotiated subsequent PBL arrangements after 2001. The
remaining 2 programs conducted a business case analysis.
[5] According to Marine Corps officials, no PBL arrangements for ground
systems have been implemented yet. The Naval Air Systems Command
manages PBL arrangements for support of Marine Corps aviation systems.
[6] The results from this nonprobability sample cannot be used to make
inferences about the population because the sample may not reflect all
of characteristics of the population.
[7] DOD Instruction 7041.3, Economic Analysis for Decisionmaking,
November 7, 1995. In this report, we refer to this as DOD's economic
analysis instruction.
[8] Office of Management and Budget, Circular A-94, Guidelines and
Discount Rates for Benefit-Cost Analysis of Federal Programs (Oct. 29,
1992).
[9] National Audit Office, Transforming Logistics Support for Fast Jets
(London: July 17, 2007).
[10] Federal Acquisition Regulation Subpart 2.101.
[11] Federal Acquisition Regulation Subpart 37.102(a), implementing
section 821 of the Floyd D. Spence Defense Authorization Act for Fiscal
Year 2001 (Pub. L. No. 106-398). The Federal Acquisition Regulation
addresses performance-based contracting generally at Subpart 37.6.
[12] GAO, Defense Logistics: Actions Needed to Enhance Success of
Reengineering Initiatives, [hyperlink,
http://www.gao.gov/products/GAO/NSIAD-00-89] (Washington, D.C.: June
23, 2000).
[13] Department of Defense, Quadrennial Defense Review Report
(Washington, D.C., Sept. 30, 2001).
[14] Department of Defense, Product Support for the 21st Century: A
Program Manager's Guide to Buying Performance. Throughout this report,
we refer to this document as DOD's 2001 PBL guide.
[15] DOD Directive 5000.1, The Defense Acquisition System, paras.
E1.1.16 and E1.1.17, May 12, 2003. The logistics footprint refers to
the size or presence of government or contractor personnel, equipment,
inventory, facilities, transportation assets, or real estate needed to
deploy, sustain, and move the weapon systems or components.
[16] Department of Defense, Defense Acquisition University, Performance
Based Logistics: A Program Manager's Product Support Guide (Washington,
D.C., Sept. 30, 2004). This document, which was reissued in March 2005,
supersedes DOD's Product Support for the 21st Century: A Program
Manager's Guide to Buying Performance, dated November 2001. In a
November 2004 memorandum, the Acting Under Secretary of Defense
(Acquisition, Technology and Logistics) described the Defense
Acquisition University's Program Manager's Product Support Guide as
guidance that should be used by DOD program and product support
managers in conjunction with other DOD policy and Office of the
Secretary of Defense memorandums. Throughout this report, we refer to
this guide as the DOD/Defense Acquisition University PBL guide.
[17] The acquisition category determines the level of review, decision
authority, and applicable procedures for an acquisition program.
Acquisition category I programs have an estimated total of expenditure
of more than $365 million for research, development, test, and
evaluation or procurement of more than $2.19 billion. Acquisition
category II programs have an estimated total expenditure for research,
development, test, and evaluation from $140 million to $365 million or
procurement from $660 million to $2.19 billion.
[18] Under Secretary of Defense (Acquisition, Technology and
Logistics), Memorandum (Jan. 23, 2004). Throughout this report we refer
to this as the January 2004 Under Secretary of Defense (Acquisition,
Technology and Logistics) memorandum.
[19] The costs for the two Marine Corps PBL arrangements are not
included in this estimate because the PBL arrangements have not yet
been implemented. Also, the total does not include support costs for
the Army's Common Ground Station PBL arrangement because this
arrangement is with a government activity and does not involve a
contract. According to Common Ground Station officials, support costs
for fiscal years 2003 through 2007 totaled about $44 million.
[20] According to Office of Management and Budget Circular A-94, a
program is cost effective if, on the basis of a life cycle cost
analysis of competing alternatives, it is determined to have the lowest
costs expressed in present value terms for a given level of benefits.
[21] U.S. Army Audit Agency, Army's Process and Controls for
Effectively Implementing Performance Based Logistics, A-2007-0072-ALM
(Alexandria, Va., Feb. 9, 2007).
[22] Department of Defense, Office of the Inspector General, Logistics:
Implementation of Performance-Based Logistics for the Javelin Weapon
System, D-2005-037 (Arlington, Va., Mar. 7, 2005).
[23] Department of Defense, Office of the Inspector General, Logistics:
Implementation of Performance-Based Logistics for the Joint
Surveillance Target Attack Radar System, D-2006-105 (Arlington, Va.,
Aug. 9, 2006).
[24] The Federal Acquisition Regulation defines fixed price types of
contracts as providing for a price that is not subject to any
adjustment on the basis of the contractor's cost experience in
performing the contract. FAR 16.202-1. A cost-plus-award-fee contract
is a cost-reimbursement contract that provides for a fee consisting of
(1) a base amount (which may be zero) fixed at inception of the
contract and (2) an award amount, based upon a judgmental evaluation by
the government, sufficient to provide motivation for excellence in
contract performance. FAR 16.305.
[25] The B-2 program office received its operations and maintenance
funding--the majority of the funds used for B-2 support--in separate
elements of expense/investment codes, which had to be used for specific
purposes. In the business case analysis, program officials assumed that
operations and maintenance funds could be used more effectively if the
funds were consolidated and not separated into different elements of
expense/investment codes.
[26] Department of Defense, Office of the Inspector General,
Acquisition: Procurement Procedures Used for C-17 Globemaster III
Sustainment Partnership Total System Support, D-2006-101 (Arlington,
Va., July 21, 2006).
[27] According to Office of Management and Budget Circular A-94, net
present value is the standard criterion for deciding whether a
government program can be justified on economic principles.
[28] DOD's economic analysis instruction recommends an evaluation of
the costs and benefits over the life cycle of the program. Further,
according to the Department of the Navy Performance Based Logistics
Guidance Document (Jan. 27, 2003), the program office is responsible
for completion of a business case analysis, and total life cycle costs
should be used in the analysis.
[29] Assistant Secretary of the Army (Acquisition, Logistics and
Technology), Memorandum, Performance-Based Logistics Business Case
Analysis Policy (Aug. 18, 2005), and Army Regulation 700-127,
Integrated Logistics Support (Sept. 27, 2007).
[30] Department of the Navy, Guide for Developing Performance Based
Logistics Business Case Analyses, P07-006 (Nov. 6, 2007). The foreword
to this guide states that programs need to update their business case
analysis data every 3 to 5 years, and the body of the document states
that updates should occur when there are significant programmatic or
support changes.
[31] The Defense Business Board (also known as the Defense Practice
Implementation Board) was established by the Secretary of Defense in
2001 and consists of private-sector executives with experience in
business management who provide independent advice and recommendations
on strategies for the implementation of best business practices.
[32] The factors listed are cost per unit of output (such as mile,
hour, etc.), performance measures, asset ownership, size of footprint,
reliability growth, life cycle costs, diminished manufacturing sources
management, obsolescence mitigation, technology insertion, risk
management, ability to synchronize with the Defense Transportation
System, existing infrastructure, and common consumable support.
[33] The Defense Acquisition Guidebook provides an interactive, online
reference to policy and discretionary best practice. The guidebook is
available at [hyperlink, https://akss.dau.mil/dag/] (accessed Oct. 27,
2008).
[34] Under Secretary of Defense (Acquisition, Technology and
Logistics), Memorandum, Performance Based Logistics: Purchasing Using
Performance Based Criteria (Aug. 16, 2004).
[35] Constant dollars measure the value of purchased goods and services
at price levels that are the same as those in the base or reference
year. Constant dollars do no contain any adjustments for inflationary
changes that have occurred or are forecast to occur outside the base
year.
[36] While we could not identify comparable costs for all elements of
support, we did obtain repair costs for repairable components for both
the Air Force's PBL arrangement and the Navy's government support
arrangement.
[37] The business case analysis did not calculate net present value to
take into account the time value of money. If calculated, the projected
savings in net present value terms would have been about $684,000.
[38] The business case analyses did not calculate net present value to
take into account the time value of money. If calculated, the projected
savings in net present value terms would have been about $2.3 million
and $4.1 million, respectively.
[39] The savings were calculated in constant fiscal year 1996 dollars.
[40] The savings were calculated in constant fiscal year 1996 dollars.
[41] Operations and maintenance funds are typically available for 1
year as provided for by the annual appropriations act. See, for
example, Pub. L. No. 110-116, 121 Stat. 1295, 1298 (Nov. 3, 2007).
[42] Working capital funds are revolving funds and subject to the
authorities stated in section 2208 of Title 10 of the U.S. Code.
[43] [hyperlink, http://www.gao.gov/products/GAO-04-715].
[44] Sang-Hyun Kim, Morris A. Cohen, and Serguei Netessine,
"Reliability or Inventory? Contracting Strategies for After-Sales
Product Support" (working paper for the Wharton School, University of
Pennsylvania, Philadelphia, Pa., March 2007) [hyperlink,
http://opimweb.wharton.upenn.edu/ideas/workingpapers.cfm?year=2007]
(accessed Oct. 21, 2008).
[45] Under Secretary of Defense (Acquisition, Technology and
Logistics), Memorandum, Performance Based Logistics (Feb. 13, 2002).
[46] Deputy Secretary of Defense, Memorandum, Implementation of the
Defense Business Practice Implementation Board (DBB) Recommendation to
the Senior Executive Council (SEC) on Continued Progress on Performance
Based Logistics (Feb. 4, 2004).
[47] Under Secretary of Defense (Acquisition, Technology and
Logistics), Memorandum, Performance Based Logistics (PBL) and Business
Case Analysis (BCA) (Mar. 20, 2004).
[48] Department of Defense, Office of the Secretary of Defense, Cost
Analysis Improvement Group, Operating and Support Cost Estimating Guide
(Washington, D.C., May 1992).
[49] Department of Defense, Office of the Secretary of Defense, Cost
Analysis Improvement Group, Operating and Support Cost Estimating Guide
(Washington, D.C., October 2007).
[50] The other organization was the Defence Procurement Agency.
[51] National Audit Office, Transforming Logistics Support for Fast
Jets.
[52] Lean techniques aim to identify and eliminate any activity that
does not add value to the end user and make the remaining activity flow
in the most efficient sequence possible.
[53] Secretary of State for Defence, "Defence Industrial Strategy"
(Defence White Paper presented to Parliament, December 2005).
[54] Office of Management and Budget Circular A-94, Guidelines and
Discount Rates for Benefit-Cost Analysis of Federal Programs (Oct. 29,
1992).
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
